brianwalenz
commented
3 years ago I'm not sure why all the jobs landed on one node. They should have spread across the cluster (and caused even more havoc to the file server). Are you running canu on the head node, or submitting it as a job to the grid? In the latter case, it will (probably) end up running on just a single node and ignoring the grid entirely. Post one of the log files from canu-scripts.
A bit of what's going on here. The 'mhap' processes are computing half of the overlaps, writing (compressed) .ovb files. The other half of the overlaps are just the same as the first half, but 'flipped': if mhap computed "read A aligned to read B" the flip would be "read B aligned to read A" - it's the same alignment, but upside down. This flip is done when overlaps are copied into the ovlStore. The complication is that we need to group all overlaps for a single read together and so "building the ovlStore" is equivalent to "sort 2*45 TB of data" (in your case). This sort is done in two passes, roughly "flip overlaps and write to buckets" (each bucket will hold overlaps for some set of reads) then "sort each bucket". Based on the directory sizes, you're failing in the first pass.
The solution, as you correctly guessed, is to increase the minimum overlap length. In the assembly directory, there is a error_correct.report that contains a read length histogram. Here's a (truncated) example from human:
-- In sequence store './asm.seqStore':
-- Found 11308761 reads.
-- Found 228039710962 bases (73.56 times coverage).
--
-- G=161318970111 sum of || length num
-- NG length index lengths || range seqs
-- ----- ------------ --------- ------------ || ------------------- -------
-- 00010 84952 155984 16131904946 || 1000-15446 8292104|---------------------------------------------------------------
-- 00020 67093 371392 32263812356 || 15447-29893 1304285|----------
-- 00030 54953 637754 48395707775 || 29894-44340 724114|------
-- 00040 45008 962310 64527623427 || 44341-58787 447442|----
-- 00050 36058 1362378 80659502478 || 58788-73234 262662|--
-- 00060 27456 1873889 96791390485 || 73235-87681 142416|--
-- 00070 19119 2575304 112923279581 || 87682-102128 73271|-
-- 00080 11707 3651832 129055187228 || 102129-116575 34932|-
-- 00090 5859 5586443 145187073272 || 116576-131022 15700|-
-- 00100 1000 11308760 161318970111 || 131023-145469 6512|-
-- 001.000x 11308761 161318970111 || 145470-159916 2916|-
-- || 159917-174363 1224|-
-- || 174364-188810 540|-
-- || 188811-203257 286|-
-- || 203258-217704 131|-
-- || 217705-232151 94|-
-- || 232152-246598 45|-For this assembly, I did two things: increase the minimum read length from the default 1,000 to 10,000 and increase the minimum overlap from the default 500 to 8,000. You can use the NG table on the left as a guide for the minimum read length to use, then pick an overlap length a tad below there. Note that 'index' is the number of reads at this length or longer, and by throwing out around 19x of coverage (length 10,000) I threw out around 70% of the reads which should also get rid of (at least) 70% of the overlaps.
Now that the theory is out of the way....how to move forward?
1) Delete everything, restart from scratch after changing minReadLength and minOverlapLength. This is the simplest and 'cleanest', but most expensive option.
2) Filter short overlaps from the existing .ovb files. This is probably the correct way to go, but, surprisingly, I don't see an easy way to do it and I'll have to write a little bit of code. Give me a day or so to get this implemented and tested. The process will likely be to run a command on each of the .ovb files in 1-overlapper/results/.
The existing error_correct.ovlStore.BUILDING (and config files) can be removed.
(Don't forget to post logging from canu-scripts/ and *.report)
AGI-chandler
skoren
conchoecia
Hello,
I am running canu 2.1.1 on a 2.7 Gb highly heterozygous plant genome, starting from ONT reads (N50 about 50 kb, 75x coverage, or 37x per allele). The command used is
canu -d Sikem_ONT_EC -p error_correct genomeSize=5.3g corMaxEvidenceErate=0.15 corMhapFilterThreshold=0.0000000002 corMhapOptions="--threshold 0.80 --num-hashes 512 --num-min-matches 3 --ordered-sketch-size 1000 --ordered-kmer-size 14 --min-olap-length 2000 --repeat-idf-scale 50" mhapMemory=60g mhapBlockSize=500 ovlMerDistinct=0.975 useGrid=true gridEngineResourceOption="-pe smp THREADS -l mem_free=MEMORY" gridOptions="-V -q all.q -S /bin/bash" -nanopore reads.fq.gz(we added the parameters that suppress repeat hits)The goal is to at least error correct the ONT reads in an allele-specific way (thanks to the high sequencing coverage) - this is why we put the size of the diploid genome. After about 3 weeks running on our cluster, it stopped due to no space left on the device. This is the size of the correction folder:
and the *.ovlStore.BUILDING/1-bucketize.successs file is not there yet.
I wonder if we can change some parameters (also given the large read N50) to have a smaller footprint and complete the correction step. Maybe
--min-olap-length 2000to 5000? What else would you tweak? Would it be possible to resume not from the very beginning?Also, all the jobs are now submitted to one node and this is what our sys admin saw: "It spawned a bunch of 1 CPU processes and really tanked the cluster. I think I may need to kill the canu run and/or reboot n009. " This is the usage for this job:
How can we kill those jobs? Thanks, Dario