search

marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
657 stars 179 forks source link

Bogart failed, retry #1829

Closed shaobinzhong closed 4 years ago

shaobinzhong commented 4 years ago

Hello Canu team,

I used Canu v1.8 to assembly PacBio reads generated for a fungus (36 Mb in size). The total reads are 133 Gb. I wanted to determine the minimum number of reads for good assembly, so I partitioned the total reads into a number of files (a, b, c, and so on) each having about 6-7 Gb of reads. Then I run canu with a, a+b, a+b+c, a+b+c+d, ......I got the assemblies with reads from one file, two files or three files combined. However, when I run the same program with reads combined from four or more files, I got the error "Bogart failed, retry". The log is attached here:

-- Canu snapshot v1.8 +296 changes (r9506 56d8be4dd6c05d2b50822a920c582c2eb7f6edd3)
--
-- CITATIONS
--
-- Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
-- Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
-- Genome Res. 2017 May;27(5):722-736.
-- http://doi.org/10.1101/gr.215087.116
-- 
-- Koren S, Rhie A, Walenz BP, Dilthey AT, Bickhart DM, Kingan SB, Hiendleder S, Williams JL, Smith TPL, Phillippy AM.
-- De novo assembly of haplotype-resolved genomes with trio binning.
-- Nat Biotechnol. 2018
-- https//doi.org/10.1038/nbt.4277
-- 
-- Read and contig alignments during correction, consensus and GFA building use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '14.0.1' (from '/gpfs1/projects/shaobin.zhong/jdk-14.0.1/bin/java') without -d64 support.
-- Detected gnuplot version '4.6 patchlevel 2   ' (from 'gnuplot') and image format 'png'.
-- Detected 32 CPUs and 1008 gigabytes of memory.
-- Detected PBSPro '18.1.4' with 'pbsnodes' binary in /cm/shared/apps/pbspro-ce/18.1.4/bin/pbsnodes.
-- Grid engine disabled per useGrid=false option.
--
--                            (tag)Concurrency
--                     (tag)Threads          |
--            (tag)Memory         |          |
--        (tag)         |         |          |     total usage     algorithm
--        -------  ------  --------   --------  -----------------  -----------------------------
-- Local: meryl     12 GB    4 CPUs x   8 jobs    96 GB   32 CPUs  (k-mer counting)
-- Local: hap        8 GB    4 CPUs x   8 jobs    64 GB   32 CPUs  (read-to-haplotype assignment)
-- Local: cormhap    6 GB   16 CPUs x   2 jobs    12 GB   32 CPUs  (overlap detection with mhap)
-- Local: obtovl     4 GB    8 CPUs x   4 jobs    16 GB   32 CPUs  (overlap detection)
-- Local: utgovl     4 GB    8 CPUs x   4 jobs    16 GB   32 CPUs  (overlap detection)
-- Local: cor        8 GB    4 CPUs x   8 jobs    64 GB   32 CPUs  (read correction)
-- Local: ovb        4 GB    1 CPU  x  32 jobs   128 GB   32 CPUs  (overlap store bucketizer)
-- Local: ovs        8 GB    1 CPU  x  32 jobs   256 GB   32 CPUs  (overlap store sorting)
-- Local: red        8 GB    4 CPUs x   8 jobs    64 GB   32 CPUs  (read error detection)
-- Local: oea        4 GB    1 CPU  x  32 jobs   128 GB   32 CPUs  (overlap error adjustment)
-- Local: bat       16 GB    4 CPUs x   1 job     16 GB    4 CPUs  (contig construction with bogart)
-- Local: cns      --- GB    4 CPUs x --- jobs   --- GB  --- CPUs  (consensus)
-- Local: gfa       16 GB    4 CPUs x   1 job     16 GB    4 CPUs  (GFA alignment and processing)
--
-- In 'ND85F.seqStore', found PacBio reads:
--   Raw:        2319925
--   Corrected:  802200
--   Trimmed:    770356
--
-- Generating assembly 'ND85F' in '/gpfs1/scratch/shaobin.zhong/Cs_ND85F_PacBio/ND85F_PacBio_Illumina_Canu_Pilon/ND85-aa-ac-ae-ag--pacbio'
--
-- Parameters:
--
--  genomeSize        36000000
--
--  Overlap Generation Limits:
--    corOvlErrorRate 0.2400 ( 24.00%)
--    obtOvlErrorRate 0.0450 (  4.50%)
--    utgOvlErrorRate 0.0450 (  4.50%)
--
--  Overlap Processing Limits:
--    corErrorRate    0.3000 ( 30.00%)
--    obtErrorRate    0.0450 (  4.50%)
--    utgErrorRate    0.0450 (  4.50%)
--    cnsErrorRate    0.0750 (  7.50%)
--
--
-- BEGIN ASSEMBLY
--
-- Finished stage 'unitig', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'bat' concurrent execution on Sun Oct 25 17:48:09 2020 with 230884.289 GB free disk space (1 processes; 1 concurrently)

    cd unitigging/4-unitigger
    ./unitigger.sh 1 > ./unitigger.000001.out 2>&1

-- Finished on Sun Oct 25 17:48:10 2020 (one second) with 230884.289 GB free disk space
----------------------------------------
--
-- Bogart failed, retry
--
--
-- Running jobs.  Second attempt out of 2.
----------------------------------------
-- Starting 'bat' concurrent execution on Sun Oct 25 17:48:10 2020 with 230884.289 GB free disk space (1 processes; 1 concurrently)

    cd unitigging/4-unitigger
    ./unitigger.sh 1 > ./unitigger.000001.out 2>&1

-- Finished on Sun Oct 25 17:48:11 2020 (one second) with 230884.289 GB free disk space
----------------------------------------
--
-- Bogart failed, tried 2 times, giving up.
--

ABORT:
ABORT: Canu snapshot v1.8 +296 changes (r9506 56d8be4dd6c05d2b50822a920c582c2eb7f6edd3)
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.
ABORT:
ABORT: Disk space available:  230884.289 GB
ABORT:
ABORT: Last 50 lines of the relevant log file (unitigging/4-unitigger/unitigger.err):
ABORT:
ABORT:   Lengths:
ABORT:     Minimum read          0 bases
ABORT:     Minimum overlap       500 bases
ABORT:   
ABORT:   Overlap Error Rates:
ABORT:     Graph                 0.045 (4.500%)
ABORT:     Max                   0.045 (4.500%)
ABORT:   
ABORT:   Deviations:
ABORT:     Graph                 6.000
ABORT:     Bubble                6.000
ABORT:     Repeat                3.000
ABORT:   
ABORT:   Edge Confusion:
ABORT:     Absolute              2100
ABORT:     Percent               200.0000
ABORT:   
ABORT:   Unitig Construction:
ABORT:     Minimum intersection  500 bases
ABORT:     Maxiumum placements   2 positions
ABORT:   
ABORT:   Debugging Enabled:
ABORT:     (none)
ABORT:   
ABORT:   ==> LOADING AND FILTERING OVERLAPS.
ABORT:   
ABORT:   sqStore_loadMetadata()-- Using 'corrected-trimmed' 0x10 reads.
ABORT:   ReadInfo()-- Using 2288081 reads, no minimum read length used.
ABORT:   
ABORT:   OverlapCache()-- limited to 16384MB memory (user supplied).
ABORT:   
ABORT:   OverlapCache()--      17MB for read data.
ABORT:   OverlapCache()--      88MB for best edges.
ABORT:   OverlapCache()--     230MB for tigs.
ABORT:   OverlapCache()--      61MB for tigs - read layouts.
ABORT:   OverlapCache()--      88MB for tigs - error profiles.
ABORT:   OverlapCache()--    4096MB for tigs - error profile overlaps.
ABORT:   OverlapCache()--       0MB for other processes.
ABORT:   OverlapCache()-- ---------
ABORT:   OverlapCache()--    4626MB for data structures (sum of above).
ABORT:   OverlapCache()-- ---------
ABORT:   OverlapCache()--      44MB for overlap store structure.
ABORT:   OverlapCache()--   11712MB for overlap data.
ABORT:   OverlapCache()-- ---------
ABORT:   OverlapCache()--   16384MB allowed.
ABORT:   OverlapCache()--
ABORT:   OverlapCache()-- Retain at least 403 overlaps/read, based on 201.93x coverage.
ABORT:   OverlapCache()-- Initial guess at 330 overlaps/read.
ABORT:   OverlapCache()--
ABORT:   OverlapCache()-- Not enough memory to load the minimum number of overlaps; increase -M.
ABORT:

Please let me know what should I do to solve the problem?

Thanks.

Shaobin

skoren commented 4 years ago

This is documented in issue #1093 and others. You have quite deep coverage, over 200x, so there isn't enough memory to load all the overlaps. The solution ins #1093 should work (batMemory=32 or batMemory=64). Canu 1.8 is also quite old and newer versions would downsample your input data automatically to avoid this error.