Closed damientully closed 6 years ago
letunglam1991
commented
6 years ago you can open job 0001.red to see error. i have same issue. I open error file and see: "ead_Olaps()-- Loading 3518262 overlaps. Failed to open '../bc1109.ovlStore/0001<001>' for reading: Too many open files".
wait answer from experts.
brianwalenz
commented
6 years ago Fixed. See 6bb19fcb1e79ce255fb7aaeed659247ae4dca9ba and 098679d3a315131e04658b0351f9031b52a85d7d.
If you update from github, you'll probably need to start the assembly over. File formats have (again) changed.
damientully
commented
6 years ago Hi Brian,
That fixed seemed to of worked. Thank you. However for one of the samples I appear to be getting a different error relating to bogart failed that appears to be a memory issue?
Restoring modules from user's modules
-- Canu snapshot v1.7 +184 changes (r8876 098679d3a315131e04658b0351f9031b52a85d7d)
--
-- CITATIONS
--
-- Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
-- Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
-- Genome Res. 2017 May;27(5):722-736.
-- http://doi.org/10.1101/gr.215087.116
--
-- Read and contig alignments during correction, consensus and GFA building use:
-- Šošic M, Šikic M.
-- Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
-- Bioinformatics. 2017 May 1;33(9):1394-1395.
-- http://doi.org/10.1093/bioinformatics/btw753
--
-- Overlaps are generated using:
-- Berlin K, et al.
-- Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
-- Nat Biotechnol. 2015 Jun;33(6):623-30.
-- http://doi.org/10.1038/nbt.3238
--
-- Myers EW, et al.
-- A Whole-Genome Assembly of Drosophila.
-- Science. 2000 Mar 24;287(5461):2196-204.
-- http://doi.org/10.1126/science.287.5461.2196
--
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
-- Chin CS, et al.
-- Phased diploid genome assembly with single-molecule real-time sequencing.
-- Nat Methods. 2016 Dec;13(12):1050-1054.
-- http://doi.org/10.1038/nmeth.4035
--
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
-- Chin CS, et al.
-- Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
-- Nat Methods. 2013 Jun;10(6):563-9
-- http://doi.org/10.1038/nmeth.2474
--
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_112' (from '/n/app/java/jdk-1.8u112/bin/java').
-- Detected gnuplot version '4.6 patchlevel 2' (from 'gnuplot') and image format 'png'.
-- Detected 32 CPUs and 252 gigabytes of memory.
-- Detected Slurm with 'sinfo' binary in /usr/bin/sinfo.
-- Grid engine disabled per useGrid=false option.
--
-- (tag)Concurrency
-- (tag)Threads |
-- (tag)Memory | |
-- (tag) | | | total usage algorithm
-- ------- ------ -------- -------- ----------------- -----------------------------
-- Local: meryl 8 GB 4 CPUs x 1 job 8 GB 4 CPUs (k-mer counting)
-- Local: cormhap 6 GB 16 CPUs x 2 jobs 12 GB 32 CPUs (overlap detection with mhap)
-- Local: obtovl 4 GB 8 CPUs x 4 jobs 16 GB 32 CPUs (overlap detection)
-- Local: utgovl 4 GB 8 CPUs x 4 jobs 16 GB 32 CPUs (overlap detection)
-- Local: ovb 4 GB 1 CPU x 32 jobs 128 GB 32 CPUs (overlap store bucketizer)
-- Local: ovs 8 GB 1 CPU x 31 jobs 248 GB 31 CPUs (overlap store sorting)
-- Local: red 8 GB 4 CPUs x 8 jobs 64 GB 32 CPUs (read error detection)
-- Local: oea 4 GB 1 CPU x 32 jobs 128 GB 32 CPUs (overlap error adjustment)
-- Local: bat 16 GB 4 CPUs x 1 job 16 GB 4 CPUs (contig construction with bogart)
-- Local: gfa 8 GB 4 CPUs x 1 job 8 GB 4 CPUs (GFA alignment and processing)
--
-- Found Nanopore uncorrected reads in the input files.
--
-- Generating assembly 'BC11' in '/n/scratch2/dct7/Run11_Canu/BC11_565'
--
-- Parameters:
--
-- genomeSize 4000
--
-- Overlap Generation Limits:
-- corOvlErrorRate 0.3200 ( 32.00%)
-- obtOvlErrorRate 0.1440 ( 14.40%)
-- utgOvlErrorRate 0.1440 ( 14.40%)
--
-- Overlap Processing Limits:
-- corErrorRate 0.5000 ( 50.00%)
-- obtErrorRate 0.1440 ( 14.40%)
-- utgErrorRate 0.1440 ( 14.40%)
-- cnsErrorRate 0.1920 ( 19.20%)
--
--
-- BEGIN CORRECTION
--
----------------------------------------
-- Starting command on Sat May 5 18:29:22 2018 with 336768.516 GB free disk space
cd .
/home/dct7/canu/Linux-amd64/bin/sqStoreCreate \
-o ./BC11.seqStore.BUILDING \
-minlength 1000 \
./BC11.seqStore.ssi \
> ./BC11.seqStore.BUILDING.err 2>&1
-- Finished on Sat May 5 18:29:23 2018 (one second) with 336768.516 GB free disk space
----------------------------------------
--
-- In sequence store './BC11.seqStore':
-- Found 37154 reads.
-- Found 108335973 bases (27083.99 times coverage).
--
-- Read length histogram (one '*' equals 224.62 reads):
-- 1000 1099 1262 *****
-- 1100 1199 884 ***
-- 1200 1299 771 ***
-- 1300 1399 734 ***
-- 1400 1499 716 ***
-- 1500 1599 667 **
-- 1600 1699 576 **
-- 1700 1799 451 **
-- 1800 1899 374 *
-- 1900 1999 355 *
-- 2000 2099 661 **
-- 2100 2199 1946 ********
-- 2200 2299 526 **
-- 2300 2399 269 *
-- 2400 2499 223
-- 2500 2599 257 *
-- 2600 2699 250 *
-- 2700 2799 226 *
-- 2800 2899 266 *
-- 2900 2999 293 *
-- 3000 3099 405 *
-- 3100 3199 689 ***
-- 3200 3299 1404 ******
-- 3300 3399 4855 *********************
-- 3400 3499 15724 **********************************************************************
-- 3500 3599 1929 ********
-- 3600 3699 77
-- 3700 3799 34
-- 3800 3899 43
-- 3900 3999 61
-- 4000 4099 12
-- 4100 4199 12
-- 4200 4299 8
-- 4300 4399 12
-- 4400 4499 14
-- 4500 4599 20
-- 4600 4699 5
-- 4700 4799 7
-- 4800 4899 8
-- 4900 4999 8
-- 5000 5099 1
-- 5100 5199 9
-- 5200 5299 3
-- 5300 5399 3
-- 5400 5499 6
-- 5500 5599 8
-- 5600 5699 10
-- 5700 5799 3
-- 5800 5899 5
-- 5900 5999 4
-- 6000 6099 2
-- 6100 6199 6
-- 6200 6299 1
-- 6300 6399 4
-- 6400 6499 1
-- 6500 6599 10
-- 6600 6699 13
-- 6700 6799 22
-- 6800 6899 6
-- 6900 6999 0
-- 7000 7099 0
-- 7100 7199 0
-- 7200 7299 1
-- 7300 7399 0
-- 7400 7499 1
-- 7500 7599 0
-- 7600 7699 0
-- 7700 7799 0
-- 7800 7899 0
-- 7900 7999 0
-- 8000 8099 0
-- 8100 8199 0
-- 8200 8299 0
-- 8300 8399 0
-- 8400 8499 0
-- 8500 8599 0
-- 8600 8699 0
-- 8700 8799 0
-- 8800 8899 0
-- 8900 8999 0
-- 9000 9099 0
-- 9100 9199 0
-- 9200 9299 0
-- 9300 9399 0
-- 9400 9499 0
-- 9500 9599 0
-- 9600 9699 0
-- 9700 9799 0
-- 9800 9899 1
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Sat May 5 18:29:23 2018 with 336768.378 GB free disk space (1 processes; 1 concurrently)
cd correction/0-mercounts
./meryl.sh 1 > ./meryl.000001.out 2>&1
-- Finished on Sat May 5 18:29:31 2018 (8 seconds) with 336767.72 GB free disk space
----------------------------------------
-- Meryl finished successfully.
--
-- 16-mers Fraction
-- Occurrences NumMers Unique Total
-- 1- 1 11167613 *******************************************************************--> 0.7349 0.1036
-- 2- 2 1613146 ********************************************************************** 0.8411 0.1336
-- 3- 4 1012779 ******************************************* 0.8841 0.1518
-- 5- 7 512646 ********************** 0.9229 0.1758
-- 8- 11 277495 ************ 0.9475 0.1995
-- 12- 16 166100 ******* 0.9625 0.2212
-- 17- 22 106537 **** 0.9721 0.2411
-- 23- 29 71621 *** 0.9785 0.2592
-- 30- 37 51060 ** 0.9829 0.2757
-- 38- 46 37126 * 0.9860 0.2911
-- 47- 56 29072 * 0.9884 0.3052
-- 57- 67 22773 0.9902 0.3188
-- 68- 79 17694 0.9917 0.3317
-- 80- 92 14451 0.9928 0.3436
-- 93- 106 11747 0.9938 0.3549
-- 107- 121 9881 0.9945 0.3656
-- 122- 137 8691 0.9952 0.3761
-- 138- 154 6985 0.9957 0.3863
-- 155- 172 5830 0.9962 0.3957
-- 173- 191 4949 0.9966 0.4045
-- 192- 211 4301 0.9969 0.4128
-- 212- 232 3822 0.9972 0.4208
-- 233- 254 3399 0.9974 0.4286
-- 255- 277 2941 0.9976 0.4362
-- 278- 301 2617 0.9978 0.4435
-- 302- 326 2444 0.9980 0.4504
-- 327- 352 2170 0.9981 0.4576
-- 353- 379 1915 0.9983 0.4644
-- 380- 407 1693 0.9984 0.4708
-- 408- 436 1437 0.9985 0.4770
-- 437- 466 1218 0.9986 0.4826
-- 467- 497 1236 0.9987 0.4877
-- 498- 529 1138 0.9988 0.4932
-- 530- 562 1035 0.9989 0.4986
-- 563- 596 880 0.9989 0.5038
-- 597- 631 809 0.9990 0.5086
-- 632- 667 756 0.9990 0.5132
-- 668- 704 670 0.9991 0.5177
-- 705- 742 585 0.9991 0.5220
-- 743- 781 548 0.9992 0.5259
-- 782- 821 555 0.9992 0.5297
--
-- 27280 (max occurrences)
-- 96611050 (total mers, non-unique)
-- 4028327 (distinct mers, non-unique)
-- 11167613 (unique mers)
-- For mhap overlapping, the threshold 0.000005000000000 is too low, resetting to 0.000628208757795 to capture 5.000000 * 27083 coverage.
-- For mhap overlapping, set repeat k-mer threshold to 135415.
--
-- Found 107778663 16-mers; 15195940 distinct and 11167613 unique. Largest count 27280.
--
-- OVERLAPPER (mhap) (correction)
--
-- Set corMhapSensitivity=low based on read coverage of 27083.
--
-- PARAMETERS: hashes=256, minMatches=3, threshold=0.85
--
-- Given 6 GB, can fit 18000 reads per block.
-- For 4 blocks, set stride to 2 blocks.
-- Logging partitioning to 'correction/1-overlapper/partitioning.log'.
-- Configured 3 mhap precompute jobs.
-- Configured 3 mhap overlap jobs.
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Sat May 5 18:29:32 2018 with 336767.72 GB free disk space (3 processes; 2 concurrently)
cd correction/1-overlapper
./precompute.sh 1 > ./precompute.000001.out 2>&1
./precompute.sh 2 > ./precompute.000002.out 2>&1
./precompute.sh 3 > ./precompute.000003.out 2>&1
-- Finished on Sat May 5 18:30:53 2018 (81 seconds) with 336775.846 GB free disk space
----------------------------------------
-- All 3 mhap precompute jobs finished successfully.
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Sat May 5 18:30:53 2018 with 336775.846 GB free disk space (3 processes; 2 concurrently)
cd correction/1-overlapper
./mhap.sh 1 > ./mhap.000001.out 2>&1
./mhap.sh 2 > ./mhap.000002.out 2>&1
./mhap.sh 3 > ./mhap.000003.out 2>&1
-- Finished on Sat May 5 19:39:11 2018 (4098 seconds) with 336655.617 GB free disk space
----------------------------------------
-- Found 3 mhap overlap output files.
----------------------------------------
-- Starting command on Sat May 5 19:39:11 2018 with 336655.617 GB free disk space
cd correction
/home/dct7/canu/Linux-amd64/bin/ovStoreConfig \
-S ../BC11.seqStore \
-M 4-8 \
-L ./1-overlapper/ovljob.files \
-create ./BC11.ovlStore.config \
> ./BC11.ovlStore.config.txt \
2> ./BC11.ovlStore.config.err
-- Finished on Sat May 5 19:39:11 2018 (like a bat out of hell) with 336655.617 GB free disk space
----------------------------------------
--
-- Creating overlap store correction/BC11.ovlStore using:
-- 3 buckets
-- 4 slices
-- using at most 8 GB memory each
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'ovB' concurrent execution on Sat May 5 19:39:11 2018 with 336655.617 GB free disk space (3 processes; 32 concurrently)
cd correction/BC11.ovlStore.BUILDING
./scripts/1-bucketize.sh 1 > ./logs/1-bucketize.000001.out 2>&1
./scripts/1-bucketize.sh 2 > ./logs/1-bucketize.000002.out 2>&1
./scripts/1-bucketize.sh 3 > ./logs/1-bucketize.000003.out 2>&1
-- Finished on Sat May 5 19:42:16 2018 (185 seconds) with 336666.835 GB free disk space
----------------------------------------
-- Overlap store bucketizer finished.
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'ovS' concurrent execution on Sat May 5 19:42:16 2018 with 336666.835 GB free disk space (4 processes; 31 concurrently)
cd correction/BC11.ovlStore.BUILDING
./scripts/2-sort.sh 1 > ./logs/2-sort.000001.out 2>&1
./scripts/2-sort.sh 2 > ./logs/2-sort.000002.out 2>&1
./scripts/2-sort.sh 3 > ./logs/2-sort.000003.out 2>&1
./scripts/2-sort.sh 4 > ./logs/2-sort.000004.out 2>&1
-- Finished on Sat May 5 19:44:41 2018 (145 seconds) with 336658.011 GB free disk space
----------------------------------------
-- Overlap store sorter finished.
----------------------------------------
-- Starting command on Sat May 5 19:44:41 2018 with 336658.011 GB free disk space
cd correction
/home/dct7/canu/Linux-amd64/bin/ovStoreIndexer \
-O ./BC11.ovlStore.BUILDING \
-S ../BC11.seqStore \
-C ./BC11.ovlStore.config \
-delete \
> ./BC11.ovlStore.BUILDING.index.err 2>&1
-- Finished on Sat May 5 19:44:41 2018 (like a bat out of hell) with 336658.011 GB free disk space
----------------------------------------
FROM: 'correction/BC11.ovlStore.BUILDING'
TO: 'correction/BC11.ovlStore'
-- Checking store.
----------------------------------------
-- Starting command on Sat May 5 19:44:41 2018 with 336658.011 GB free disk space
cd correction
/home/dct7/canu/Linux-amd64/bin/ovStoreDump \
-S ../BC11.seqStore \
-O ./BC11.ovlStore \
-counts \
> ./BC11.ovlStore/counts.dat 2> ./BC11.ovlStore/counts.err
-- Finished on Sat May 5 19:44:41 2018 (lickety-split) with 336658.011 GB free disk space
----------------------------------------
--
-- Overlap store 'correction/BC11.ovlStore' successfully constructed.
-- Found 1182701906 overlaps for 37154 reads; 0 reads have no overlaps.
--
--
-- Purged 9.974 GB in 12 overlap output files.
-- Set corMinCoverage=4 based on read coverage of 27083.
-- Global filter scores will be estimated.
-- Computing correction layouts.
----------------------------------------
-- Starting command on Sat May 5 19:44:41 2018 with 336658.011 GB free disk space
cd correction
/home/dct7/canu/Linux-amd64/bin/generateCorrectionLayouts \
-S ../BC11.seqStore \
-O ./BC11.ovlStore \
-C ./BC11.corStore.WORKING \
-eC 80 \
> ./BC11.corStore.err 2>&1
-- Finished on Sat May 5 19:45:52 2018 (71 seconds) with 336683.103 GB free disk space
----------------------------------------
-- Computing correction layouts.
----------------------------------------
-- Starting command on Sat May 5 19:45:52 2018 with 336683.103 GB free disk space
cd correction/2-correction
/home/dct7/canu/Linux-amd64/bin/filterCorrectionLayouts \
-S ../../BC11.seqStore \
-C ../BC11.corStore \
-R ./BC11.readsToCorrect.WORKING \
-cc 4 \
-cl 1000 \
-g 4000 \
-c 40 \
> ./BC11.readsToCorrect.err 2>&1
-- Finished on Sat May 5 19:45:53 2018 (one second) with 336683.103 GB free disk space
----------------------------------------
-- original original
-- raw reads raw reads
-- category w/overlaps w/o/overlaps
-- -------------------- ------------- -------------
-- Number of Reads 25063 12091
-- Number of Bases 74401765 33934208
-- Coverage 18600.441 8483.552
-- Median 3422 3341
-- Mean 2968 2806
-- N50 3441 3384
-- Minimum 1000 1000
-- Maximum 9883 4970
--
-- --------corrected--------- ----------rescued----------
-- evidence expected expected
-- category reads raw corrected raw corrected
-- -------------------- ------------- ------------- ------------- ------------- -------------
-- Number of Reads 717 24 24 15973 15973
-- Number of Bases 2089487 162848 162653 48398574 44281471
-- Coverage 522.372 40.712 40.663 12099.644 11070.368
-- Median 3103 6768 6756 3437 3365
-- Mean 2914 6785 6777 3030 2772
-- N50 3381 6769 6767 3449 3425
-- Minimum 1000 6714 6713 1012 1001
-- Maximum 9883 7218 7217 6170 5864
--
-- --------uncorrected--------
-- expected
-- category raw corrected
-- -------------------- ------------- -------------
-- Number of Reads 21157 21157
-- Number of Bases 59774551 828571
-- Coverage 14943.638 207.143
-- Median 3360 0
-- Mean 2825 39
-- N50 3401 3075
-- Minimum 1000 0
-- Maximum 9883 9865
--
-- Maximum Memory 626939888
--
-- (tag)Concurrency
-- (tag)Threads |
-- (tag)Memory | |
-- (tag) | | | total usage algorithm
-- ------- ------ -------- -------- ----------------- -----------------------------
-- Local: cor 2 GB 4 CPUs x 8 jobs 16 GB 32 CPUs (read correction)
--
-- Using overlaps no worse than 0.5 fraction error for correcting reads (from corErrorRate parameter).
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'cor' concurrent execution on Sat May 5 19:45:53 2018 with 336683.103 GB free disk space (8 processes; 8 concurrently)
cd correction/2-correction
./correctReads.sh 1 > ./correctReads.000001.out 2>&1
./correctReads.sh 2 > ./correctReads.000002.out 2>&1
./correctReads.sh 3 > ./correctReads.000003.out 2>&1
./correctReads.sh 4 > ./correctReads.000004.out 2>&1
./correctReads.sh 5 > ./correctReads.000005.out 2>&1
./correctReads.sh 6 > ./correctReads.000006.out 2>&1
./correctReads.sh 7 > ./correctReads.000007.out 2>&1
./correctReads.sh 8 > ./correctReads.000008.out 2>&1
-- Finished on Sat May 5 19:48:31 2018 (158 seconds) with 336679.285 GB free disk space
----------------------------------------
-- Found 8 read correction output files.
-- Found 8 read correction output files.
--
-- Loading corrected reads into corStore and seqStore.
----------------------------------------
-- Starting command on Sat May 5 19:48:31 2018 with 336679.285 GB free disk space
cd correction
/home/dct7/canu/Linux-amd64/bin/loadCorrectedReads \
-S ../BC11.seqStore \
-C ./BC11.corStore \
-L ./2-correction/corjob.files \
> ./BC11.loadCorrectedReads.log \
2> ./BC11.loadCorrectedReads.err
-- Finished on Sat May 5 19:48:32 2018 (one second) with 336679.285 GB free disk space
----------------------------------------
--
-- In sequence store './BC11.seqStore':
-- Found 15993 reads.
-- Found 43868121 bases (10967.03 times coverage).
--
-- Read length histogram (one '*' equals 90.55 reads):
-- 300 399 1
-- 400 499 0
-- 500 599 1
-- 600 699 6
-- 700 799 13
-- 800 899 32
-- 900 999 97 *
-- 1000 1099 825 *********
-- 1100 1199 411 ****
-- 1200 1299 409 ****
-- 1300 1399 371 ****
-- 1400 1499 440 ****
-- 1500 1599 373 ****
-- 1600 1699 279 ***
-- 1700 1799 245 **
-- 1800 1899 192 **
-- 1900 1999 196 **
-- 2000 2099 457 *****
-- 2100 2199 1080 ***********
-- 2200 2299 144 *
-- 2300 2399 142 *
-- 2400 2499 132 *
-- 2500 2599 135 *
-- 2600 2699 120 *
-- 2700 2799 131 *
-- 2800 2899 134 *
-- 2900 2999 120 *
-- 3000 3099 164 *
-- 3100 3199 214 **
-- 3200 3299 434 ****
-- 3300 3399 2160 ***********************
-- 3400 3499 6339 **********************************************************************
-- 3500 3599 49
-- 3600 3699 9
-- 3700 3799 13
-- 3800 3899 24
-- 3900 3999 17
-- 4000 4099 4
-- 4100 4199 5
-- 4200 4299 5
-- 4300 4399 7
-- 4400 4499 8
-- 4500 4599 11
-- 4600 4699 4
-- 4700 4799 4
-- 4800 4899 1
-- 4900 4999 2
-- 5000 5099 0
-- 5100 5199 1
-- 5200 5299 2
-- 5300 5399 1
-- 5400 5499 2
-- 5500 5599 2
-- 5600 5699 1
-- 5700 5799 0
-- 5800 5899 0
-- 5900 5999 0
-- 6000 6099 0
-- 6100 6199 0
-- 6200 6299 0
-- 6300 6399 0
-- 6400 6499 0
-- 6500 6599 0
-- 6600 6699 0
-- 6700 6799 21
-- 6800 6899 3
--
-- Purging correctReads output after loading into stores.
-- Purged 8 .cns outputs.
-- Purged 16 .out job log outputs.
--
-- Purging overlaps used for correction.
--
--
-- BEGIN TRIMMING
--
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Sat May 5 19:48:32 2018 with 336679.285 GB free disk space (1 processes; 1 concurrently)
cd trimming/0-mercounts
./meryl.sh 1 > ./meryl.000001.out 2>&1
-- Finished on Sat May 5 19:48:34 2018 (2 seconds) with 336680.166 GB free disk space
----------------------------------------
-- Meryl finished successfully.
--
-- 22-mers Fraction
-- Occurrences NumMers Unique Total
-- 1- 1 1164966 *******************************************************************--> 0.6963 0.0268
-- 2- 2 175737 ********************************************************************** 0.8013 0.0348
-- 3- 4 120357 *********************************************** 0.8469 0.0401
-- 5- 7 68436 *************************** 0.8909 0.0475
-- 8- 11 41006 **************** 0.9220 0.0557
-- 12- 16 24220 ********* 0.9427 0.0639
-- 17- 22 15896 ****** 0.9551 0.0709
-- 23- 29 11867 **** 0.9638 0.0777
-- 30- 37 8666 *** 0.9705 0.0846
-- 38- 46 5808 ** 0.9753 0.0910
-- 47- 56 4571 * 0.9787 0.0964
-- 57- 67 3623 * 0.9813 0.1017
-- 68- 79 2882 * 0.9834 0.1068
-- 80- 92 2300 0.9851 0.1116
-- 93- 106 1888 0.9864 0.1161
-- 107- 121 1557 0.9875 0.1203
-- 122- 137 1557 0.9885 0.1244
-- 138- 154 1312 0.9894 0.1290
-- 155- 172 1147 0.9902 0.1334
-- 173- 191 1109 0.9908 0.1377
-- 192- 211 763 0.9915 0.1423
-- 212- 232 690 0.9920 0.1458
-- 233- 254 556 0.9924 0.1493
-- 255- 277 500 0.9927 0.1523
-- 278- 301 452 0.9930 0.1553
-- 302- 326 499 0.9933 0.1584
-- 327- 352 313 0.9936 0.1620
-- 353- 379 308 0.9937 0.1644
-- 380- 407 356 0.9939 0.1670
-- 408- 436 286 0.9941 0.1703
-- 437- 466 286 0.9943 0.1730
-- 467- 497 300 0.9945 0.1759
-- 498- 529 263 0.9947 0.1793
-- 530- 562 288 0.9948 0.1823
-- 563- 596 246 0.9950 0.1860
-- 597- 631 209 0.9951 0.1892
-- 632- 667 181 0.9953 0.1922
-- 668- 704 176 0.9954 0.1948
-- 705- 742 147 0.9955 0.1976
-- 743- 781 160 0.9956 0.2001
-- 782- 821 160 0.9956 0.2028
--
-- 18591 (max occurrences)
-- 42367302 (total mers, non-unique)
-- 508202 (distinct mers, non-unique)
-- 1164966 (unique mers)
----------------------------------------
-- Starting command on Sat May 5 19:48:35 2018 with 336680.166 GB free disk space
cd trimming/0-mercounts
/home/dct7/canu/Linux-amd64/bin/meryl \
-Dt \
-n 12462 \
-s ./BC11.ms22 \
> ./BC11.ms22.frequentMers.fasta \
2> ./BC11.ms22.frequentMers.fasta.err
-- Finished on Sat May 5 19:48:35 2018 (in the blink of an eye) with 336680.166 GB free disk space
----------------------------------------
-- Reset obtOvlMerThreshold from auto to 12462.
--
-- Found 43532268 22-mers; 1673168 distinct and 1164966 unique. Largest count 18591.
--
-- OVERLAPPER (normal) (trimming) erate=0.144
--
----------------------------------------
-- Starting command on Sat May 5 19:48:35 2018 with 336680.166 GB free disk space
cd trimming/1-overlapper
/home/dct7/canu/Linux-amd64/bin/overlapInCorePartition \
-S ../../BC11.seqStore \
-hl 85312000 \
-rl 1000000000 \
-ol 500 \
-o ./BC11.partition \
> ./BC11.partition.err 2>&1
-- Finished on Sat May 5 19:48:35 2018 (like a bat out of hell) with 336680.166 GB free disk space
----------------------------------------
--
-- Configured 1 overlapInCore jobs.
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'obtovl' concurrent execution on Sat May 5 19:48:35 2018 with 336680.166 GB free disk space (1 processes; 4 concurrently)
cd trimming/1-overlapper
./overlap.sh 1 > ./overlap.000001.out 2>&1
-- Finished on Sun May 6 08:03:18 2018 (44083 seconds) with 335610.382 GB free disk space
----------------------------------------
-- Found 1 overlapInCore output files.
--
-- overlapInCore compute 'trimming/1-overlapper':
-- kmer hits
-- with no overlap 89078902 89078902 +- 0
-- with an overlap 121397108 121397108 +- 0
--
-- overlaps 121388258 121388258 +- 0
-- contained 0 0 +- 0
-- dovetail 0 0 +- 0
--
-- overlaps rejected
-- multiple per pair 0 0 +- 0
-- bad short window 0 0 +- 0
-- bad long window 0 0 +- 0
----------------------------------------
-- Starting command on Sun May 6 08:03:18 2018 with 335610.382 GB free disk space
cd trimming
/home/dct7/canu/Linux-amd64/bin/ovStoreConfig \
-S ../BC11.seqStore \
-M 8 \
-L ./1-overlapper/ovljob.files \
-create ./BC11.ovlStore.config \
> ./BC11.ovlStore.config.txt \
2> ./BC11.ovlStore.config.err
-- Finished on Sun May 6 08:03:18 2018 (lickety-split) with 335610.382 GB free disk space
----------------------------------------
--
-- Creating overlap store trimming/BC11.ovlStore using:
-- 1 bucket
-- 1 slice
-- using at most 7 GB memory each
----------------------------------------
-- Starting command on Sun May 6 08:03:18 2018 with 335610.382 GB free disk space
cd trimming
/home/dct7/canu/Linux-amd64/bin/ovStoreBuild \
-O ./BC11.ovlStore.BUILDING \
-S ../BC11.seqStore \
-C ./BC11.ovlStore.config \
> ./BC11.ovlStore.err 2>&1
-- Finished on Sun May 6 08:04:33 2018 (75 seconds) with 335599.595 GB free disk space
----------------------------------------
-- Checking store.
----------------------------------------
-- Starting command on Sun May 6 08:04:33 2018 with 335599.595 GB free disk space
cd trimming
/home/dct7/canu/Linux-amd64/bin/ovStoreDump \
-S ../BC11.seqStore \
-O ./BC11.ovlStore \
-counts \
> ./BC11.ovlStore/counts.dat 2> ./BC11.ovlStore/counts.err
-- Finished on Sun May 6 08:04:33 2018 (like a bat out of hell) with 335599.595 GB free disk space
----------------------------------------
--
-- Overlap store 'trimming/BC11.ovlStore' successfully constructed.
-- Found 242794216 overlaps for 15990 reads; 21164 reads have no overlaps.
--
--
-- Purged 1.911 GB in 3 overlap output files.
----------------------------------------
-- Starting command on Sun May 6 08:04:33 2018 with 335599.595 GB free disk space
cd trimming/3-overlapbasedtrimming
/home/dct7/canu/Linux-amd64/bin/trimReads \
-S ../../BC11.seqStore \
-O ../BC11.ovlStore \
-Co ./BC11.1.trimReads.clear \
-e 0.144 \
-minlength 1000 \
-ol 1 \
-oc 1 \
-o ./BC11.1.trimReads \
> ./BC11.1.trimReads.err 2>&1
-- Finished on Sun May 6 08:05:22 2018 (49 seconds) with 335596.633 GB free disk space
----------------------------------------
-- PARAMETERS:
-- ----------
-- 1000 (reads trimmed below this many bases are deleted)
-- 0.1440 (use overlaps at or below this fraction error)
-- 1 (break region if overlap is less than this long, for 'largest covered' algorithm)
-- 1 (break region if overlap coverage is less than this many read, for 'largest covered' algorithm)
--
-- INPUT READS:
-- -----------
-- 37154 reads 43868121 bases (reads processed)
-- 0 reads 0 bases (reads not processed, previously deleted)
-- 0 reads 0 bases (reads not processed, in a library where trimming isn't allowed)
--
-- OUTPUT READS:
-- ------------
-- 2389 reads 6141070 bases (trimmed reads output)
-- 13420 reads 37409827 bases (reads with no change, kept as is)
-- 21164 reads 2175 bases (reads with no overlaps, deleted)
-- 181 reads 170989 bases (reads with short trimmed length, deleted)
--
-- TRIMMING DETAILS:
-- ----------------
-- 1159 reads 76336 bases (bases trimmed from the 5' end of a read)
-- 1317 reads 67724 bases (bases trimmed from the 3' end of a read)
----------------------------------------
-- Starting command on Sun May 6 08:05:22 2018 with 335596.633 GB free disk space
cd trimming/3-overlapbasedtrimming
/home/dct7/canu/Linux-amd64/bin/splitReads \
-S ../../BC11.seqStore \
-O ../BC11.ovlStore \
-Ci ./BC11.1.trimReads.clear \
-Co ./BC11.2.splitReads.clear \
-e 0.144 \
-minlength 1000 \
-o ./BC11.2.splitReads \
> ./BC11.2.splitReads.err 2>&1
-- Finished on Sun May 6 08:06:57 2018 (95 seconds) with 335584.579 GB free disk space
----------------------------------------
-- PARAMETERS:
-- ----------
-- 1000 (reads trimmed below this many bases are deleted)
-- 0.1440 (use overlaps at or below this fraction error)
-- INPUT READS:
-- -----------
-- 15809 reads 43694957 bases (reads processed)
-- 21345 reads 173164 bases (reads not processed, previously deleted)
-- 0 reads 0 bases (reads not processed, in a library where trimming isn't allowed)
--
-- PROCESSED:
-- --------
-- 0 reads 0 bases (no overlaps)
-- 0 reads 0 bases (no coverage after adjusting for trimming done already)
-- 0 reads 0 bases (processed for chimera)
-- 0 reads 0 bases (processed for spur)
-- 15809 reads 43694957 bases (processed for subreads)
--
-- READS WITH SIGNALS:
-- ------------------
-- 0 reads 0 signals (number of 5' spur signal)
-- 0 reads 0 signals (number of 3' spur signal)
-- 0 reads 0 signals (number of chimera signal)
-- 2 reads 2 signals (number of subread signal)
--
-- SIGNALS:
-- -------
-- 0 reads 0 bases (size of 5' spur signal)
-- 0 reads 0 bases (size of 3' spur signal)
-- 0 reads 0 bases (size of chimera signal)
-- 2 reads 1323 bases (size of subread signal)
--
-- TRIMMING:
-- --------
-- 2 reads 2397 bases (trimmed from the 5' end of the read)
-- 0 reads 0 bases (trimmed from the 3' end of the read)
----------------------------------------
-- Starting command on Sun May 6 08:06:57 2018 with 335584.579 GB free disk space
cd trimming/3-overlapbasedtrimming
/home/dct7/canu/Linux-amd64/bin/loadTrimmedReads \
-S ../../BC11.seqStore \
-c ./BC11.2.splitReads.clear \
> ./BC11.loadtrimmedReads.err 2>&1
-- Finished on Sun May 6 08:06:57 2018 (fuirously fast) with 335584.579 GB free disk space
----------------------------------------
--
-- In sequence store './BC11.seqStore':
-- Found 15808 reads.
-- Found 43547680 bases (10886.92 times coverage).
--
-- Read length histogram (one '*' equals 89.71 reads):
-- 1000 1099 823 *********
-- 1100 1199 415 ****
-- 1200 1299 399 ****
-- 1300 1399 383 ****
-- 1400 1499 453 *****
-- 1500 1599 375 ****
-- 1600 1699 285 ***
-- 1700 1799 238 **
-- 1800 1899 185 **
-- 1900 1999 200 **
-- 2000 2099 452 *****
-- 2100 2199 1072 ***********
-- 2200 2299 151 *
-- 2300 2399 147 *
-- 2400 2499 128 *
-- 2500 2599 139 *
-- 2600 2699 125 *
-- 2700 2799 127 *
-- 2800 2899 144 *
-- 2900 2999 125 *
-- 3000 3099 170 *
-- 3100 3199 206 **
-- 3200 3299 429 ****
-- 3300 3399 2188 ************************
-- 3400 3499 6280 **********************************************************************
-- 3500 3599 51
-- 3600 3699 5
-- 3700 3799 11
-- 3800 3899 22
-- 3900 3999 16
-- 4000 4099 2
-- 4100 4199 1
-- 4200 4299 3
-- 4300 4399 5
-- 4400 4499 8
-- 4500 4599 11
-- 4600 4699 1
-- 4700 4799 1
-- 4800 4899 0
-- 4900 4999 2
-- 5000 5099 0
-- 5100 5199 0
-- 5200 5299 2
-- 5300 5399 1
-- 5400 5499 1
-- 5500 5599 1
-- 5600 5699 1
-- 5700 5799 0
-- 5800 5899 0
-- 5900 5999 0
-- 6000 6099 0
-- 6100 6199 0
-- 6200 6299 0
-- 6300 6399 0
-- 6400 6499 0
-- 6500 6599 0
-- 6600 6699 0
-- 6700 6799 21
-- 6800 6899 3
--
-- Overlaps used for trimming saved.
--
--
-- BEGIN ASSEMBLY
--
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Sun May 6 08:06:57 2018 with 335584.579 GB free disk space (1 processes; 1 concurrently)
cd unitigging/0-mercounts
./meryl.sh 1 > ./meryl.000001.out 2>&1
-- Finished on Sun May 6 08:06:59 2018 (2 seconds) with 335584.133 GB free disk space
----------------------------------------
-- Meryl finished successfully.
--
-- 22-mers Fraction
-- Occurrences NumMers Unique Total
-- 1- 1 1091214 *******************************************************************--> 0.6855 0.0253
-- 2- 2 172168 ********************************************************************** 0.7937 0.0332
-- 3- 4 118686 ************************************************ 0.8409 0.0384
-- 5- 7 67733 *************************** 0.8867 0.0459
-- 8- 11 40344 **************** 0.9190 0.0540
-- 12- 16 23876 ********* 0.9403 0.0621
-- 17- 22 15706 ****** 0.9531 0.0690
-- 23- 29 11807 **** 0.9623 0.0758
-- 30- 37 8511 *** 0.9692 0.0827
-- 38- 46 5753 ** 0.9743 0.0891
-- 47- 56 4523 * 0.9777 0.0945
-- 57- 67 3602 * 0.9805 0.0998
-- 68- 79 2835 * 0.9827 0.1049
-- 80- 92 2286 0.9844 0.1096
-- 93- 106 1879 0.9858 0.1141
-- 107- 121 1577 0.9870 0.1183
-- 122- 137 1519 0.9880 0.1225
-- 138- 154 1326 0.9889 0.1270
-- 155- 172 1123 0.9897 0.1315
-- 173- 191 1095 0.9904 0.1357
-- 192- 211 769 0.9911 0.1403
-- 212- 232 689 0.9916 0.1439
-- 233- 254 530 0.9920 0.1473
-- 255- 277 486 0.9923 0.1503
-- 278- 301 476 0.9926 0.1533
-- 302- 326 487 0.9929 0.1565
-- 327- 352 297 0.9932 0.1600
-- 353- 379 315 0.9934 0.1623
-- 380- 407 363 0.9936 0.1649
-- 408- 436 273 0.9938 0.1682
-- 437- 466 291 0.9940 0.1709
-- 467- 497 302 0.9942 0.1740
-- 498- 529 254 0.9944 0.1774
-- 530- 562 291 0.9946 0.1804
-- 563- 596 246 0.9947 0.1840
-- 597- 631 203 0.9949 0.1873
-- 632- 667 181 0.9950 0.1902
-- 668- 704 177 0.9951 0.1929
-- 705- 742 146 0.9952 0.1957
-- 743- 781 159 0.9953 0.1982
-- 782- 821 165 0.9954 0.2009
--
-- 18349 (max occurrences)
-- 42124498 (total mers, non-unique)
-- 500557 (distinct mers, non-unique)
-- 1091214 (unique mers)
----------------------------------------
-- Starting command on Sun May 6 08:06:59 2018 with 335584.133 GB free disk space
cd unitigging/0-mercounts
/home/dct7/canu/Linux-amd64/bin/meryl \
-Dt \
-n 12385 \
-s ./BC11.ms22 \
> ./BC11.ms22.frequentMers.fasta \
2> ./BC11.ms22.frequentMers.fasta.err
-- Finished on Sun May 6 08:06:59 2018 (in the blink of an eye) with 335584.133 GB free disk space
----------------------------------------
-- Reset utgOvlMerThreshold from auto to 12385.
--
-- Found 43215712 22-mers; 1591771 distinct and 1091214 unique. Largest count 18349.
--
-- OVERLAPPER (normal) (assembly) erate=0.144
--
----------------------------------------
-- Starting command on Sun May 6 08:06:59 2018 with 335584.133 GB free disk space
cd unitigging/1-overlapper
/home/dct7/canu/Linux-amd64/bin/overlapInCorePartition \
-S ../../BC11.seqStore \
-hl 85312000 \
-rl 1000000000 \
-ol 500 \
-o ./BC11.partition \
> ./BC11.partition.err 2>&1
-- Finished on Sun May 6 08:06:59 2018 (fast as lightning) with 335584.133 GB free disk space
----------------------------------------
--
-- Configured 1 overlapInCore jobs.
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'utgovl' concurrent execution on Sun May 6 08:06:59 2018 with 335584.133 GB free disk space (1 processes; 4 concurrently)
cd unitigging/1-overlapper
./overlap.sh 1 > ./overlap.000001.out 2>&1
-- Finished on Mon May 7 01:47:11 2018 (63612 seconds) with 354954.779 GB free disk space
----------------------------------------
-- Found 1 overlapInCore output files.
--
-- overlapInCore compute 'unitigging/1-overlapper':
-- kmer hits
-- with no overlap 110479702 110479702 +- 0
-- with an overlap 94235889 94235889 +- 0
--
-- overlaps 94235889 94235889 +- 0
-- contained 39960749 39960749 +- 0
-- dovetail 54275140 54275140 +- 0
--
-- overlaps rejected
-- multiple per pair 0 0 +- 0
-- bad short window 0 0 +- 0
-- bad long window 0 0 +- 0
----------------------------------------
-- Starting command on Mon May 7 01:47:12 2018 with 354954.779 GB free disk space
cd unitigging
/home/dct7/canu/Linux-amd64/bin/ovStoreConfig \
-S ../BC11.seqStore \
-M 7 \
-L ./1-overlapper/ovljob.files \
-create ./BC11.ovlStore.config \
> ./BC11.ovlStore.config.txt \
2> ./BC11.ovlStore.config.err
-- Finished on Mon May 7 01:47:12 2018 (fuirously fast) with 354954.779 GB free disk space
----------------------------------------
--
-- Creating overlap store unitigging/BC11.ovlStore using:
-- 1 bucket
-- 1 slice
-- using at most 5 GB memory each
----------------------------------------
-- Starting command on Mon May 7 01:47:12 2018 with 354954.779 GB free disk space
cd unitigging
/home/dct7/canu/Linux-amd64/bin/ovStoreBuild \
-O ./BC11.ovlStore.BUILDING \
-S ../BC11.seqStore \
-C ./BC11.ovlStore.config \
> ./BC11.ovlStore.err 2>&1
-- Finished on Mon May 7 01:48:15 2018 (63 seconds) with 354949.658 GB free disk space
----------------------------------------
-- Checking store.
----------------------------------------
-- Starting command on Mon May 7 01:48:15 2018 with 354949.658 GB free disk space
cd unitigging
/home/dct7/canu/Linux-amd64/bin/ovStoreDump \
-S ../BC11.seqStore \
-O ./BC11.ovlStore \
-counts \
> ./BC11.ovlStore/counts.dat 2> ./BC11.ovlStore/counts.err
-- Finished on Mon May 7 01:48:16 2018 (one second) with 354949.658 GB free disk space
----------------------------------------
--
-- Overlap store 'unitigging/BC11.ovlStore' successfully constructed.
-- Found 188471778 overlaps for 15808 reads; 21346 reads have no overlaps.
--
--
-- Purged 1.246 GB in 3 overlap output files.
----------------------------------------
-- Starting command on Mon May 7 01:48:16 2018 with 354949.658 GB free disk space
cd unitigging
/home/dct7/canu/Linux-amd64/bin/ovStoreStats \
-C 10886 \
-S ../BC11.seqStore \
-O ./BC11.ovlStore \
-o ./BC11.ovlStore \
> ./BC11.ovlStore.summary.err 2>&1
-- Finished on Mon May 7 01:48:58 2018 (42 seconds) with 354951.011 GB free disk space
----------------------------------------
--
-- Overlap store 'unitigging/BC11.ovlStore' contains:
--
-- category reads % read length feature size or coverage analysis
-- ---------------- ------- ------- ---------------------- ------------------------ --------------------
-- middle-missing 0 0.00 0.00 +- 0.00 0.00 +- 0.00 (bad trimming)
-- middle-hump 1 0.01 4458.00 +- 0.00 0.00 +- 0.00 (bad trimming)
-- no-5-prime 2 0.01 3661.00 +- 1159.66 449.50 +- 635.69 (bad trimming)
-- no-3-prime 4 0.03 2333.50 +- 1078.21 147.75 +- 216.09 (bad trimming)
--
-- low-coverage 1727 10.92 2039.02 +- 410.65 1951.20 +- 515.24 (easy to assemble, potential for lower quality consensus)
-- unique 7111 44.98 2606.38 +- 951.99 11273.95 +- 952.29 (easy to assemble, perfect, yay)
-- repeat-cont 0 0.00 0.00 +- 0.00 0.00 +- 0.00 (potential for consensus errors, no impact on assembly)
-- repeat-dove 0 0.00 0.00 +- 0.00 0.00 +- 0.00 (hard to assemble, likely won't assemble correctly or even at all)
--
-- span-repeat 44 0.28 3505.23 +- 1343.01 2773.34 +- 1258.21 (read spans a large repeat, usually easy to assemble)
-- uniq-repeat-cont 6907 43.69 3077.35 +- 754.24 (should be uniquely placed, low potential for consensus errors, no impact on assembly)
-- uniq-repeat-dove 12 0.08 5145.50 +- 944.02 (will end contigs, potential to misassemble)
-- uniq-anchor 0 0.00 0.00 +- 0.00 0.00 +- 0.00 (repeat read, with unique section, probable bad read)
--
-- Loading read lengths.
-- Loading number of overlaps per read.
--
-- Configure RED for 8gb memory.
-- Batches of at most (unlimited) reads.
-- 500000000 bases.
-- Expecting evidence of at most 536870912 bases per iteration.
--
-- Total Reads Olaps Evidence
-- Job Memory Read Range Reads Bases Memory Olaps Memory Memory (Memory in MB)
-- ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------
-- 1 5727.75 1-37154 15808 43547680 498.86 188471778 2156.89 1024.00
-- ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------
-- 43547680 188471778
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'red' concurrent execution on Mon May 7 01:48:59 2018 with 354951.011 GB free disk space (1 processes; 8 concurrently)
cd unitigging/3-overlapErrorAdjustment
./red.sh 1 > ./red.000001.out 2>&1
-- Finished on Mon May 7 04:59:00 2018 (11401 seconds) with 354876.501 GB free disk space
----------------------------------------
-- Found 1 read error detection output files.
--
-- Loading read lengths.
-- Loading number of overlaps per read.
--
-- Configure OEA for 4gb memory.
-- Batches of at most (unlimited) reads.
-- 300000000 bases.
--
-- Total Reads Olaps Adjusts
-- Job Memory Read Range Reads Bases Memory Olaps Memory Memory (Memory in MB)
-- ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------
-- 1 4096.23 1-10862 4704 12962153 20.51 55927364 1706.77 320.96
-- 2 4096.34 10863-23113 4690 12736452 20.29 55938032 1707.09 320.96
-- 3 4096.40 23114-33279 4689 13038102 20.58 55930512 1706.86 320.96
-- 4 3012.57 33280-37154 1725 4810973 12.64 20675870 630.98 320.96
-- ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------
-- 43547680 188471778
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'oea' concurrent execution on Mon May 7 04:59:00 2018 with 354876.501 GB free disk space (4 processes; 32 concurrently)
cd unitigging/3-overlapErrorAdjustment
./oea.sh 1 > ./oea.000001.out 2>&1
./oea.sh 2 > ./oea.000002.out 2>&1
./oea.sh 3 > ./oea.000003.out 2>&1
./oea.sh 4 > ./oea.000004.out 2>&1
-- Finished on Mon May 7 08:27:28 2018 (12508 seconds) with 354749.002 GB free disk space
----------------------------------------
-- Found 4 overlap error adjustment output files.
----------------------------------------
-- Starting command on Mon May 7 08:27:28 2018 with 354749.002 GB free disk space
cd unitigging/3-overlapErrorAdjustment
/home/dct7/canu/Linux-amd64/bin/loadErates \
-S ../../BC11.seqStore \
-O ../BC11.ovlStore \
-L ./oea.files \
> ./oea.apply.err 2>&1
-- Finished on Mon May 7 08:27:29 2018 (one second) with 354749.002 GB free disk space
----------------------------------------
-- No report available.
--
-- Running jobs. First attempt out of 2.
----------------------------------------
-- Starting 'bat' concurrent execution on Mon May 7 08:27:29 2018 with 354749.002 GB free disk space (1 processes; 1 concurrently)
cd unitigging/4-unitigger
./unitigger.sh 1 > ./unitigger.000001.out 2>&1
-- Finished on Mon May 7 08:27:29 2018 (fast as lightning) with 354748.974 GB free disk space
----------------------------------------
--
-- Bogart failed, retry
--
--
-- Running jobs. Second attempt out of 2.
----------------------------------------
-- Starting 'bat' concurrent execution on Mon May 7 08:27:29 2018 with 354748.974 GB free disk space (1 processes; 1 concurrently)
cd unitigging/4-unitigger
./unitigger.sh 1 > ./unitigger.000001.out 2>&1
-- Finished on Mon May 7 08:27:29 2018 (like a bat out of hell) with 354748.974 GB free disk space
----------------------------------------
--
-- Bogart failed, tried 2 times, giving up.
--
ABORT:
ABORT: Canu snapshot v1.7 +184 changes (r8876 098679d3a315131e04658b0351f9031b52a85d7d)
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting. If that doesn't work, ask for help.
ABORT:
ABORT: Disk space available: 354748.974 GB
ABORT:
ABORT: Last 50 lines of the relevant log file (unitigging/4-unitigger/unitigger.err):
ABORT:
ABORT:
ABORT: Lengths:
ABORT: Minimum read 0 bases
ABORT: Minimum overlap 500 bases
ABORT:
ABORT: Overlap Error Rates:
ABORT: Graph 0.144 (14.400%)
ABORT: Max 0.144 (14.400%)
ABORT:
ABORT: Deviations:
ABORT: Graph 6.000
ABORT: Bubble 6.000
ABORT: Repeat 3.000
ABORT:
ABORT: Edge Confusion:
ABORT: Absolute 2100
ABORT: Percent 200.0000
ABORT:
ABORT: Unitig Construction:
ABORT: Minimum intersection 500 bases
ABORT: Maxiumum placements 2 positions
ABORT:
ABORT: Debugging Enabled:
ABORT: (none)
ABORT:
ABORT: ==> LOADING AND FILTERING OVERLAPS.
ABORT:
ABORT: ReadInfo()-- Using 37154 reads, no minimum read length used.
ABORT:
ABORT: OverlapCache()-- limited to 16384MB memory (user supplied).
ABORT:
ABORT: OverlapCache()-- 0MB for read data.
ABORT: OverlapCache()-- 1MB for best edges.
ABORT: OverlapCache()-- 3MB for tigs.
ABORT: OverlapCache()-- 0MB for tigs - read layouts.
ABORT: OverlapCache()-- 1MB for tigs - error profiles.
ABORT: OverlapCache()-- 4096MB for tigs - error profile overlaps.
ABORT: OverlapCache()-- 0MB for other processes.
ABORT: OverlapCache()-- ---------
ABORT: OverlapCache()-- 4104MB for data structures (sum of above).
ABORT: OverlapCache()-- ---------
ABORT: OverlapCache()-- 0MB for overlap store structure.
ABORT: OverlapCache()-- 12278MB for overlap data.
ABORT: OverlapCache()-- ---------
ABORT: OverlapCache()-- 16384MB allowed.
ABORT: OverlapCache()--
ABORT: OverlapCache()-- Retain at least 21773 overlaps/read, based on 10886.92x coverage.
ABORT: OverlapCache()-- Initial guess at 21658 overlaps/read.
ABORT: OverlapCache()--
ABORT: OverlapCache()-- Not enough memory to load the minimum number of overlaps; increase -M.
ABORT:
skoren
commented
6 years ago This is related to your high coverage, this looks like an amplicon (also see #876 and #894)? You start with >5000 fold coverage and end up with > 2000 fold corrected reads. Try running with batMemory=200, you should just be able to remove the unitigging/4-unitigger folder and re-start the same command. Alternatively, randomly downsample your dataset to 200-fold coverage.
Hi,
I am using the following command:
./canu -useGrid=0 -p BC05 -d /n/scratch2/dct7/Run11_Canu/BC05_AC347 genomeSize=9000 contigFilter="2 1000 1.0 1.0 2" -nanopore-raw /n/scratch2/dct7/Run11_Canu/BC05.fastqand get the following errorAny help would be greatly appreciated.
Thanks, Damien