search

marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
658 stars 179 forks source link

ABORT: partitioning failed; increase redMemory. #2205

Closed rmormando closed 1 year ago

rmormando commented 1 year ago

System: Linux AWS server (EC2 instance)

Version: canu 2.2

Command: ./canu -p 011_full -d 01_canu genomeSize=12m -trimmed -stopOnLowCoverage=0.001 -nanopore /home/ec2-user/01_full.fastq

Error Message:

-- canu 2.2
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
--   Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
--   Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
--   Genome Res. 2017 May;27(5):722-736.
--   http://doi.org/10.1101/gr.215087.116
--
-- Read and contig alignments during correction and consensus use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
--
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
--
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
--
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
--
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
--
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_362' (from 'java') with -d64 support.
-- Detected gnuplot version '4.6 patchlevel 2   ' (from 'gnuplot') and image format 'png'.
--
-- Detected 16 CPUs and 62 gigabytes of memory on the local machine.
--
-- Local machine mode enabled; grid support not detected or not allowed.
--
--                                (tag)Concurrency
--                         (tag)Threads          |
--                (tag)Memory         |          |
--        (tag)             |         |          |       total usage      algorithm
--        -------  ----------  --------   --------  --------------------  -----------------------------
-- Local: meryl     12.000 GB    4 CPUs x   4 jobs    48.000 GB  16 CPUs  (k-mer counting)
-- Local: hap        8.000 GB    4 CPUs x   4 jobs    32.000 GB  16 CPUs  (read-to-haplotype assignment)
-- Local: cormhap    6.000 GB   16 CPUs x   1 job      6.000 GB  16 CPUs  (overlap detection with mhap)
-- Local: obtovl     4.000 GB    8 CPUs x   2 jobs     8.000 GB  16 CPUs  (overlap detection)
-- Local: utgovl     4.000 GB    8 CPUs x   2 jobs     8.000 GB  16 CPUs  (overlap detection)
-- Local: cor        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (read correction)
-- Local: ovb        4.000 GB    1 CPU  x  15 jobs    60.000 GB  15 CPUs  (overlap store bucketizer)
-- Local: ovs        8.000 GB    1 CPU  x   7 jobs    56.000 GB   7 CPUs  (overlap store sorting)
-- Local: red       15.000 GB    4 CPUs x   4 jobs    60.000 GB  16 CPUs  (read error detection)
-- Local: oea        8.000 GB    1 CPU  x   7 jobs    56.000 GB   7 CPUs  (overlap error adjustment)
-- Local: bat       16.000 GB    4 CPUs x   1 job     16.000 GB   4 CPUs  (contig construction with bogart)
-- Local: cns        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (consensus)
--
-- Found Nanopore reads in '01_full.seqStore':
--   Libraries:
--     Nanopore:              1
--   Reads:
--     Corrected:             372275
--     Corrected and Trimmed: 372275
--
--
-- Generating assembly '01_full' in '/home/ec2-user/canu-2.2/build/bin/01_canu':
--   genomeSize:
--     12000000
--
--   Overlap Generation Limits:
--     corOvlErrorRate 0.3200 ( 32.00%)
--     obtOvlErrorRate 0.1200 ( 12.00%)
--     utgOvlErrorRate 0.1200 ( 12.00%)
--
--   Overlap Processing Limits:
--     corErrorRate    0.3000 ( 30.00%)
--     obtErrorRate    0.1200 ( 12.00%)
--     utgErrorRate    0.1200 ( 12.00%)
--     cnsErrorRate    0.2000 ( 20.00%)
--
--   Stages to run:
--     assemble corrected and trimmed reads.
--
--
-- Correction skipped; not enabled.
--
-- Trimming skipped; not enabled.
--
-- BEGIN ASSEMBLY
--
-- Loading read lengths.
-- Loading number of overlaps per read.
--
-- Configure RED for 15gb memory.
--                   Batches of at most (unlimited) reads.
--                                      500000000 bases.
--                   Expecting evidence of at most 536870912 bases per iteration.
--
--           Total                                               Reads                 Olaps Evidence
--    Job   Memory      Read Range         Reads        Bases   Memory        Olaps   Memory   Memory  (Memory in MB)
--   ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------
--      1  3117.45         1-52               52       372275    45.45          574     0.01  1024.00
--   ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------
--                                                     372275                   574

ABORT:
ABORT: canu 2.2
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.
ABORT:
ABORT:   partitioning failed; increase redMemory.
ABORT:

I have tried increasing the redMemory and the maxMemory and decreasing the stopOnLowCoverage but no matter what I do it gives me this error message. Can someone help me fix it?

skoren commented 1 year ago

This looks the same as #2035 or #2179 which unfortunately was fixed post the 2.2 release. You can either update the code to use the latest if you can compile canu locally from GitHub (or edit the code).

On a related note, it seems you have extremely low coverage for your genome so there isn't really much here to assemble. Do you really only have 400k of data for a 12mb genome? You also would need to correct ONT data before assembling it but again, with this little coverage there isn't going to be anything to correct or assemble.

rmormando commented 1 year ago

Thank you! That worked for me.

It's not actually 12mb - its a small region of a genome where a gene candidate is present so there is "low coverage" in that region but not actually. It's complicated is the short answer lol.

Thank you for your help!