search

marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
657 stars 179 forks source link

meryl failed to configure #1406

Closed Freemacl closed 5 years ago

Freemacl commented 5 years ago

command: canu -p ecoli -d ecoli-oxford genomeSize=4.8m -nanopore-raw oxford.fasta

- Canu 1.8
--
-- CITATIONS
--
-- Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
-- Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
-- Genome Res. 2017 May;27(5):722-736.
-- http://doi.org/10.1101/gr.215087.116
-- 
-- Koren S, Rhie A, Walenz BP, Dilthey AT, Bickhart DM, Kingan SB, Hiendleder S, Williams JL, Smith TPL, Phillippy AM.
-- De novo assembly of haplotype-resolved genomes with trio binning.
-- Nat Biotechnol. 2018
-- https//doi.org/10.1038/nbt.4277
-- 
-- Read and contig alignments during correction, consensus and GFA building use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_212' (from 'java') with -d64 support.
--
-- WARNING:
-- WARNING:  Failed to run gnuplot using command 'gnuplot'.
-- WARNING:  Plots will be disabled.
-- WARNING:
--
-- Detected 16 CPUs and 63 gigabytes of memory.
-- No grid engine detected, grid disabled.
--
--                            (tag)Concurrency
--                     (tag)Threads          |
--            (tag)Memory         |          |
--        (tag)         |         |          |     total usage     algorithm
--        -------  ------  --------   --------  -----------------  -----------------------------
-- Local: meryl     12 GB    4 CPUs x   4 jobs    48 GB   16 CPUs  (k-mer counting)
-- Local: hap        8 GB    4 CPUs x   4 jobs    32 GB   16 CPUs  (read-to-haplotype assignment)
-- Local: cormhap    6 GB   16 CPUs x   1 job      6 GB   16 CPUs  (overlap detection with mhap)
-- Local: obtovl     4 GB    8 CPUs x   2 jobs     8 GB   16 CPUs  (overlap detection)
-- Local: utgovl     4 GB    8 CPUs x   2 jobs     8 GB   16 CPUs  (overlap detection)
-- Local: ovb        4 GB    1 CPU  x  15 jobs    60 GB   15 CPUs  (overlap store bucketizer)
-- Local: ovs        8 GB    1 CPU  x   7 jobs    56 GB    7 CPUs  (overlap store sorting)
-- Local: red        8 GB    4 CPUs x   4 jobs    32 GB   16 CPUs  (read error detection)
-- Local: oea        4 GB    1 CPU  x  15 jobs    60 GB   15 CPUs  (overlap error adjustment)
-- Local: bat       16 GB    4 CPUs x   1 job     16 GB    4 CPUs  (contig construction with bogart)
-- Local: gfa        8 GB    4 CPUs x   1 job      8 GB    4 CPUs  (GFA alignment and processing)
--
-- Found Nanopore uncorrected reads in the input files.
--
-- Generating assembly 'ecoli' in '/mnt/dba76339-c0dd-47cb-9094-ea4d81432bbf/Claire/Plasmidome/canu/ecoli-oxford'
--
-- Parameters:
--
--  genomeSize        4800000
--
--  Overlap Generation Limits:
--    corOvlErrorRate 0.3200 ( 32.00%)
--    obtOvlErrorRate 0.1200 ( 12.00%)
--    utgOvlErrorRate 0.1200 ( 12.00%)
--
--  Overlap Processing Limits:
--    corErrorRate    0.5000 ( 50.00%)
--    obtErrorRate    0.1200 ( 12.00%)
--    utgErrorRate    0.1200 ( 12.00%)
--    cnsErrorRate    0.2000 ( 20.00%)
--
--
-- BEGIN CORRECTION
--
----------------------------------------
-- Starting command on Wed Jul  3 12:32:03 2019 with 1991.635 GB free disk space

    cd .
    /home/claire/bin/sqStoreCreate \
      -o ./ecoli.seqStore.BUILDING \
      -minlength 1000 \
      ./ecoli.seqStore.ssi \
    > ./ecoli.seqStore.BUILDING.err 2>&1

-- Finished on Wed Jul  3 12:32:04 2019 (one second) with 1991.591 GB free disk space
----------------------------------------
--
-- WARNING: gnuplot failed.
--
----------------------------------------
--
-- In sequence store './ecoli.seqStore':
--   Found 20365 reads.
--   Found 140042151 bases (29.17 times coverage).
--
--   Read length histogram (one '*' equals 41.48 reads):
--     1000   1999    706 *****************
--     2000   2999   1682 ****************************************
--     3000   3999   1624 ***************************************
--     4000   4999   1543 *************************************
--     5000   5999   1905 *********************************************
--     6000   6999   2691 ****************************************************************
--     7000   7999   2904 **********************************************************************
--     8000   8999   2609 **************************************************************
--     9000   9999   1946 **********************************************
--    10000  10999   1280 ******************************
--    11000  11999    733 *****************
--    12000  12999    397 *********
--    13000  13999    181 ****
--    14000  14999    109 **
--    15000  15999     38 
--    16000  16999      9 
--    17000  17999      4 
--    18000  18999      2 
--    19000  19999      0 
--    20000  20999      0 
--    21000  21999      0 
--    22000  22999      1 
--    23000  23999      0 
--    24000  24999      0 
--    25000  25999      1 
----------------------------------------
-- Starting command on Wed Jul  3 12:32:04 2019 with 1991.591 GB free disk space

    cd correction/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Wed Jul  3 12:32:04 2019 (furiously fast) with 1991.591 GB free disk space
----------------------------------------

ERROR:
ERROR:  Failed with exit code 126.  (rc=32256)
ERROR:

CRASH:
CRASH: Canu 1.8
CRASH: Please panic, this is abnormal.
ABORT:
CRASH:   meryl failed to configure.
CRASH:
CRASH: Failed at /home/claire/bin/../lib/site_perl/canu/Meryl.pm line 391.
CRASH:  canu::Meryl::merylConfigure("ecoli", "cor") called at /home/claire/bin/canu line 779
CRASH: 
CRASH: No log file supplied.
CRASH:

-For what it's worth, "please panic" is my all time favourite crash report -also worth noting the ecoli.ms16.config.01.out isn't produced

skoren commented 5 years ago

Sorry took so long to reply, we seem to have missed your issue.

How was Canu installed? Is there a meryl-configure.err or .sh file in your 0-mercounts folder? If so can you post them here?

Freemacl commented 5 years ago

I tried installing canu two ways:

  1. as a conda environment
  2. By downloading the latest release And unfortunately both resulted in this same error

When I execute the canu command, I don't think meryl-configure.sh is being run as it generates no output file in the 0-mercounts directory. When I run meryl-configure.sh by hand it outputs ecoli.ms16.config.01.out:

Counting 133  million canonical 16-mers from 1 input file:
    canu-seqStore: ../../ecoli.seqStore

SIMPLE MODE
-----------
  16-mers
    -> 4294967296 entries for counts up to 65535.
    -> 64 Gbits memory used

  140042151 input bases
    -> expected max count of 560168, needing 5 extra bits.
    -> 20 Gbits memory used
  10 GB memory needed

COMPLEX MODE
------------

prefix     # of   struct   kmers/    segs/     data    total
  bits   prefix   memory   prefix   prefix   memory   memory
------  -------  -------  -------  -------  -------  -------
     1     2  P    64 kB    66 MM  4143  S   517 MB   517 MB
     2     4  P    63 kB    33 MM  2005  S   501 MB   501 MB
     3     8  P    61 kB    16 MM   969  S   484 MB   484 MB
     4    16  P    60 kB  8547 kM   468  S   467 MB   467 MB
     5    32  P    59 kB  4273 kM   226  S   451 MB   451 MB
     6    64  P    60 kB  2136 kM   109  S   435 MB   435 MB
     7   128  P    65 kB  1068 kM    53  S   423 MB   423 MB
     8   256  P    76 kB   534 kM    26  S   415 MB   415 MB
     9   512  P   100 kB   267 kM    13  S   415 MB   415 MB
    10  1024  P   144 kB   133 kM     6  S   383 MB   383 MB  Best Value!
    11  2048  P   240 kB    66 kM     3  S   383 MB   384 MB
    12  4096  P   448 kB    33 kM     2  S   511 MB   512 MB
    13  8192  P   832 kB    16 kM     1  S   511 MB   512 MB
    14    16 kP  1664 kB  8548  M     1  S  1023 MB  1025 MB
    15    32 kP  3328 kB  4274  M     1  S  2047 MB  2050 MB
FINAL CONFIGURATION
-------------------
Configured complex mode for 0.375 GB memory per batch, and up to 1 batch.
Bye.
brianwalenz commented 5 years ago

Exit code 126 means a command is found but is not executable. For some reason, canu isn't able to run meryl-configure.sh.

How are you running meryl-configure.sh by hand?

What does ls -l meryl-configure.sh report?

When canu tries to run it, is there a meryl-configure.err generated?

Freemacl commented 5 years ago

To run meryl-configure.sh by hand I just entered "sh meryl-configure.sh" (after it gives me the meryl failed to configure error). That program prints:

Found perl:
   /opt/miniconda3/envs/canu/bin/perl
   This is perl 5, version 26, subversion 2 (v5.26.2) built for x86_64-linux-thread-multi

Found java:
   /opt/miniconda3/envs/canu/bin/java
   openjdk version "11.0.1" 2018-10-16 LTS

Found canu:
   /home/claire/bin/canu
   Canu 1.8

and after that command, that's when it outputs the ecoli.ms16.config.01.out file printed above.

ls -l meryl-configure.sh prints -rwxr-xr-x 1 claire claire 823 Jul 22 11:15 meryl-configure.sh

no meryl-configure.err file unfortunately.

brianwalenz commented 5 years ago

Can you run, by hand still, ./meryl-configure.sh?

Freemacl commented 5 years ago

Ah! when i type just ./meryl-configure.sh it tells me "Permission denied", even though I am the owner and it is currently "-rwxrwxrwx" ... any ideas?

Freemacl commented 5 years ago

Figured it out! On my system, canu is installed on the home directory, which is an executable filesystem, so no issues there. However, my output was being directed to a mounted HDD with "user" specified in the options of the fstab file, which automatically prevents the execution of binaries from the filesystem. Since meryl-configure.sh is generated in the output directory, it was failing every time.

Now it's giving me a fun new error with Mhap, so if I can't get that to cooperate, I will open a new issue.

Thank you for your help!

Sudosundu commented 4 years ago

@Freemacl Hi, I am trying to execute canu on an external SSD. I got the same error. CRASH: `

CRASH: Canu 1.9 CRASH: Please panic, this is abnormal. CRASH: CRASH: meryl failed to configure. CRASH: CRASH: Failed at /home/mgu/Documents/Tools/canu-1.9/Linux-amd64/bin/../lib/site_perl/canu/Meryl.pm line 414. CRASH: canu::Meryl::merylConfigure("BC09-q7-all6", "cor") called at /home/mgu/Documents/Tools/canu-1.9/Linux-amd64/bin/canu line 859 CRASH: CRASH: Last 50 lines of the relevant log file (correction/0-mercounts/meryl-configure.err): CRASH: CRASH: sh: 1: ./meryl-configure.sh: Permission denied

` How to overcome this?

brianwalenz commented 4 years ago

I'm going to guess that the drive is mounted with options that prevent running programs from it. For example https://superuser.com/questions/728127/what-does-noexec-flag-mean-when-mounting-directories-on-rhel