search

marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
654 stars 179 forks source link

fail to do canu job , help!!! #1163

Closed zhoudka closed 5 years ago

zhoudka commented 5 years ago 
Linux-amd64/bin/canu obtovlMemory=1G obtovlThreads=2 useGrid=false -p ecoli -d ecoli-auto genomeSize=4.8m -pacbio-raw ecoli_p6_25x.filtered.fastq
-- Canu 1.8
--
-- CITATIONS
--
-- Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
-- Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
-- Genome Res. 2017 May;27(5):722-736.
-- http://doi.org/10.1101/gr.215087.116
-- 
-- Koren S, Rhie A, Walenz BP, Dilthey AT, Bickhart DM, Kingan SB, Hiendleder S, Williams JL, Smith TPL, Phillippy AM.
-- De novo assembly of haplotype-resolved genomes with trio binning.
-- Nat Biotechnol. 2018
-- https//doi.org/10.1038/nbt.4277
-- 
-- Read and contig alignments during correction, consensus and GFA building use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_192' (from 'java') with -d64 support.
-- Detected gnuplot version '5.2 patchlevel 2   ' (from 'gnuplot') and image format 'png'.
-- Detected 2 CPUs and 12 gigabytes of memory.
-- No grid engine detected, grid disabled.
--
--                            (tag)Concurrency
--                     (tag)Threads          |
--            (tag)Memory         |          |
--        (tag)         |         |          |     total usage     algorithm
--        -------  ------  --------   --------  -----------------  -----------------------------
-- Local: meryl     12 GB    2 CPUs x   1 job     12 GB    2 CPUs  (k-mer counting)
-- Local: hap        8 GB    2 CPUs x   1 job      8 GB    2 CPUs  (read-to-haplotype assignment)
-- Local: cormhap    6 GB    2 CPUs x   1 job      6 GB    2 CPUs  (overlap detection with mhap)
-- Local: obtovl     1 GB    2 CPUs x   1 job      1 GB    2 CPUs  (overlap detection)
-- Local: utgovl     4 GB    2 CPUs x   1 job      4 GB    2 CPUs  (overlap detection)
-- Local: ovb        4 GB    1 CPU  x   2 jobs     8 GB    2 CPUs  (overlap store bucketizer)
-- Local: ovs        8 GB    1 CPU  x   1 job      8 GB    1 CPU   (overlap store sorting)
-- Local: red        8 GB    2 CPUs x   1 job      8 GB    2 CPUs  (read error detection)
-- Local: oea        4 GB    1 CPU  x   2 jobs     8 GB    2 CPUs  (overlap error adjustment)
-- Local: bat       12 GB    2 CPUs x   1 job     12 GB    2 CPUs  (contig construction with bogart)
-- Local: gfa        8 GB    2 CPUs x   1 job      8 GB    2 CPUs  (GFA alignment and processing)
--
-- In 'ecoli.seqStore', found PacBio reads:
--   Raw:        12528
--   Corrected:  0
--   Trimmed:    0
--
-- Generating assembly 'ecoli' in '/home/dakai/work/canu-1.8/ecoli-auto'
--
-- Parameters:
--
--  genomeSize        4800000
--
--  Overlap Generation Limits:
--    corOvlErrorRate 0.2400 ( 24.00%)
--    obtOvlErrorRate 0.0450 (  4.50%)
--    utgOvlErrorRate 0.0450 (  4.50%)
--
--  Overlap Processing Limits:
--    corErrorRate    0.3000 ( 30.00%)
--    obtErrorRate    0.0450 (  4.50%)
--    utgErrorRate    0.0450 (  4.50%)
--    cnsErrorRate    0.0750 (  7.50%)
--
--
-- BEGIN CORRECTION
--
--  segments   memory batches
--  -------- -------- -------

ABORT:
ABORT: Canu 1.8
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.
ABORT:
ABORT:   failed to parse meryl configure output 'correction/0-mercounts/ecoli.ms16.config.01.out'.
ABORT:
ABORT: Disk space available:  62.001 GB
ABORT:
ABORT: Last 50 lines of the relevant log file (correction/0-mercounts/ecoli.ms16.config.01.out):
ABORT:
ABORT:       equal-to N           return kmers that occur exactly N times in the input.  accepts exactly one input.
ABORT:       not-equal-to N       return kmers that do not occur exactly N times in the input.  accepts exactly one input.
ABORT:   
ABORT:       increase X           add X to the count of each kmer.
ABORT:       decrease X           subtract X from the count of each kmer.
ABORT:       multiply X           multiply the count of each kmer by X.
ABORT:       divide X             divide the count of each kmer by X.
ABORT:       modulo X             set the count of each kmer to the remainder of the count divided by X.
ABORT:   
ABORT:       union                return kmers that occur in any input, set the count to the number of inputs with this kmer.
ABORT:       union-min            return kmers that occur in any input, set the count to the minimum count
ABORT:       union-max            return kmers that occur in any input, set the count to the maximum count
ABORT:       union-sum            return kmers that occur in any input, set the count to the sum of the counts
ABORT:   
ABORT:       intersect            return kmers that occur in all inputs, set the count to the count in the first input.
ABORT:       intersect-min        return kmers that occur in all inputs, set the count to the minimum count.
ABORT:       intersect-max        return kmers that occur in all inputs, set the count to the maximum count.
ABORT:       intersect-sum        return kmers that occur in all inputs, set the count to the sum of the counts.
ABORT:   
ABORT:       difference           return kmers that occur in the first input, but none of the other inputs
ABORT:       symmetric-difference return kmers that occur in exactly one input
ABORT:   
ABORT:     MODIFIERS:
ABORT:   
ABORT:       output O             write kmers generated by the present command to an output  meryl database O
ABORT:                            mandatory for count operations.
ABORT:   
ABORT:     EXAMPLES:
ABORT:   
ABORT:     Example:  Report 22-mers present in at least one of input1.fasta and input2.fasta.
ABORT:               Kmers from each input are saved in meryl databases 'input1' and 'input2',
ABORT:               but the kmers in the union are only reported to the screen.
ABORT:   
ABORT:               meryl print \
ABORT:                       union \
ABORT:                         [count k=22 input1.fasta output input1] \
ABORT:                         [count k=22 input2.fasta output input2]
ABORT:   
ABORT:     Example:  Find the highest count of each kmer present in both files, save the kmers to
ABORT:               database 'maxCount'.
ABORT:   
ABORT:               meryl intersect-max input1 input2 output maxCount
ABORT:   
ABORT:     Example:  Find unique kmers common to both files.  Brackets are necessary
ABORT:               on the first 'equal-to' command to prevent the second 'equal-to' from
ABORT:               being used as an input to the first 'equal-to'.
ABORT:   
ABORT:               meryl intersect [equal-to 1 input1] equal-to 1 input2
ABORT:   
ABORT:   Requested memory 'memory=8' (GB) is more than physical memory 7.79 GB.
ABORT:
brianwalenz commented 5 years ago

How odd. Canu is saying -- Detected 2 CPUs and 12 gigabytes of memory., but the meryl component is only seeing 8 GB (7.79 GB as it reports). Do you have 12 GB RAM or 8 GB RAM?

Anyway, set maxMemory=6 to limit all pieces of canu to 6 GB RAM.

cmaggior commented 5 years ago

I have encountered this same issue. Setting maxMemory=6 still gives the error: Requested memory 'memory=8' (GB) is more than physical memory 7.79 GB. Can you please advise?

skoren commented 5 years ago

You probably have old settings from the previous run still left over. You should start a new run in an empty folder with the maxMemory=6 option. If it still doesn't run post the full log of your run.

cmaggior commented 5 years ago

That did the trick. Thank you!