search

marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
660 stars 179 forks source link

The reads coverage is too low to continue #2255

Closed yaoxkkkkk closed 1 year ago

yaoxkkkkk commented 1 year ago

I am assembling a genome using Canu 2.2 with the following commands on a slurm cluster:

canu useGrid=false \
    -p tf -d tf-pacbio \
    genomeSize=164.4m \
    -corrected \
    corMhapSensitivity=normal \
    -pacbio SRR23272336_1_corrected.fastq

The input Pacbio fasta file is corrected by LoRDEC. The error of low coverage came up for 3 times and I don't know what to do, so I come for some help. The log is below:

-- canu 2.2
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
--   Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
--   Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
--   Genome Res. 2017 May;27(5):722-736.
--   http://doi.org/10.1101/gr.215087.116
-- 
-- Read and contig alignments during correction and consensus use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_312' (from 'java') with -d64 support.
-- Detected gnuplot version '5.2 patchlevel 4   ' (from 'gnuplot') and image format 'png'.
--
-- Detected 40 CPUs and 320000 gigabytes of memory on the local machine.
--
-- Detected Slurm with 'sinfo' binary in /usr/bin/sinfo.
--          Slurm disabled by useGrid=false
--
-- Local machine mode enabled; grid support not detected or not allowed.
--
--                                (tag)Concurrency
--                         (tag)Threads          |
--                (tag)Memory         |          |
--        (tag)             |         |          |       total usage      algorithm
--        -------  ----------  --------   --------  --------------------  -----------------------------
-- Local: meryl     24.000 GB    8 CPUs x   5 jobs   120.000 GB  40 CPUs  (k-mer counting)
-- Local: hap       12.000 GB   20 CPUs x   2 jobs    24.000 GB  40 CPUs  (read-to-haplotype assignment)
-- Local: cormhap   13.000 GB   10 CPUs x   4 jobs    52.000 GB  40 CPUs  (overlap detection with mhap)
-- Local: obtovl     8.000 GB    8 CPUs x   5 jobs    40.000 GB  40 CPUs  (overlap detection)
-- Local: utgovl     8.000 GB    8 CPUs x   5 jobs    40.000 GB  40 CPUs  (overlap detection)
-- Local: cor        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (read correction)
-- Local: ovb        4.000 GB    1 CPU  x  40 jobs   160.000 GB  40 CPUs  (overlap store bucketizer)
-- Local: ovs        8.000 GB    1 CPU  x  40 jobs   320.000 GB  40 CPUs  (overlap store sorting)
-- Local: red       16.000 GB    5 CPUs x   8 jobs   128.000 GB  40 CPUs  (read error detection)
-- Local: oea        8.000 GB    1 CPU  x  40 jobs   320.000 GB  40 CPUs  (overlap error adjustment)
-- Local: bat       64.000 GB    8 CPUs x   1 job     64.000 GB   8 CPUs  (contig construction with bogart)
-- Local: cns        -.--- GB    8 CPUs x   - jobs     -.--- GB   - CPUs  (consensus)
--
-- Found untrimmed corrected PacBio CLR reads in the input files.
--
-- Generating assembly 'tf' in '/dssg/home/acct-jiang.lu/jiang.lu/Tf/Genome_data/tf-pacbio':
--   genomeSize:
--     164400000
--
--   Overlap Generation Limits:
--     corOvlErrorRate 0.2400 ( 24.00%)
--     obtOvlErrorRate 0.0450 (  4.50%)
--     utgOvlErrorRate 0.0450 (  4.50%)
--
--   Overlap Processing Limits:
--     corErrorRate    0.2500 ( 25.00%)
--     obtErrorRate    0.0450 (  4.50%)
--     utgErrorRate    0.0450 (  4.50%)
--     cnsErrorRate    0.0750 (  7.50%)
--
--   Stages to run:
--     trim corrected reads.
--     assemble corrected and trimmed reads.
--
--
-- Correction skipped; not enabled.
--
-- BEGIN TRIMMING
----------------------------------------
-- Starting command on Sat Sep  2 09:25:56 2023 with 795304.031 GB free disk space

    cd .
    ./tf.seqStore.sh \
    > ./tf.seqStore.err 2>&1

-- Finished on Sat Sep  2 09:29:53 2023 (237 seconds) with 795310.062 GB free disk space
----------------------------------------
--
-- In sequence store './tf.seqStore':
--   Found 1411073 reads.
--   Found 23257214780 bases (141.46 times coverage).
--    Histogram of corrected reads:
--    
--    G=23257214780                      sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010        52551     36126   2325761120  ||       1000-7395       261891|------------------------------
--    00020        40690     86828   4651448764  ||       7396-13791      555449|---------------------------------------------------------------
--    00030        32763    150761   6977187856  ||      13792-20187      236518|---------------------------
--    00040        26583    229701   9302910962  ||      20188-26583      127515|---------------
--    00050        21435    327248  11628627051  ||      26584-32979       81150|----------
--    00060        17024    449097  13954330013  ||      32980-39375       53400|-------
--    00070        13654    602575  16280055328  ||      39376-45771       34875|----
--    00080        11536    788591  18605780643  ||      45772-52167       23081|---
--    00090         9370   1010032  20931494352  ||      52168-58563       15068|--
--    00100         1000   1411072  23257214780  ||      58564-64959        9295|--
--    001.000x             1411073  23257214780  ||      64960-71355        5487|-
--                                               ||      71356-77751        3205|-
--                                               ||      77752-84147        1748|-
--                                               ||      84148-90543        1077|-
--                                               ||      90544-96939         603|-
--                                               ||      96940-103335        322|-
--                                               ||     103336-109731        176|-
--                                               ||     109732-116127         97|-
--                                               ||     116128-122523         51|-
--                                               ||     122524-128919         25|-
--                                               ||     128920-135315         14|-
--                                               ||     135316-141711          8|-
--                                               ||     141712-148107          1|-
--                                               ||     148108-154503          0|
--                                               ||     154504-160899          3|-
--                                               ||     160900-167295          2|-
--                                               ||     167296-173691          2|-
--                                               ||     173692-180087          1|-
--                                               ||     180088-186483          4|-
--                                               ||     186484-192879          1|-
--                                               ||     192880-199275          0|
--                                               ||     199276-205671          1|-
--                                               ||     205672-212067          0|
--                                               ||     212068-218463          0|
--                                               ||     218464-224859          1|-
--                                               ||     224860-231255          1|-
--                                               ||     231256-237651          0|
--                                               ||     237652-244047          0|
--                                               ||     244048-250443          0|
--                                               ||     250444-256839          0|
--                                               ||     256840-263235          0|
--                                               ||     263236-269631          0|
--                                               ||     269632-276027          0|
--                                               ||     276028-282423          0|
--                                               ||     282424-288819          0|
--                                               ||     288820-295215          0|
--                                               ||     295216-301611          0|
--                                               ||     301612-308007          0|
--                                               ||     308008-314403          0|
--                                               ||     314404-320799          1|-
--    
----------------------------------------
-- Starting command on Sat Sep  2 09:29:55 2023 with 795310.046 GB free disk space

    cd trimming/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Sat Sep  2 09:29:56 2023 (one second) with 795310.046 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--        01 18.93 GB       5
--        02 18.93 GB       3
--        04 18.93 GB       2
--        06 12.97 GB       2
--        08  9.72 GB       2
--        12  6.74 GB       2
--        16  5.18 GB       2
--        20  4.12 GB       2
--        24  3.64 GB       2
--        32  2.66 GB       2
--        40  2.31 GB       2
--        48  1.93 GB       2
--        56  1.66 GB       2
--        64  1.45 GB       2
--        96  0.97 GB       2
--
--  For 1411073 reads with 23257214780 bases, limit to 232 batches.
--  Will count kmers using 04 jobs, each using 20 GB and 8 threads.
--
-- Finished stage 'merylConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Sat Sep  2 09:29:56 2023 with 795310.046 GB free disk space (4 processes; 5 concurrently)

    cd trimming/0-mercounts
    ./meryl-count.sh 1 > ./meryl-count.000001.out 2>&1
    ./meryl-count.sh 2 > ./meryl-count.000002.out 2>&1
    ./meryl-count.sh 3 > ./meryl-count.000003.out 2>&1
    ./meryl-count.sh 4 > ./meryl-count.000004.out 2>&1

-- Finished on Sat Sep  2 09:35:58 2023 (362 seconds) with 795092.5 GB free disk space
----------------------------------------
-- Found 4 Kmer counting (meryl) outputs.
-- Finished stage 'obt-merylCountCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Sat Sep  2 09:35:58 2023 with 795092.5 GB free disk space (1 processes; 5 concurrently)

    cd trimming/0-mercounts
    ./meryl-process.sh 1 > ./meryl-process.000001.out 2>&1

-- Finished on Sat Sep  2 09:39:19 2023 (201 seconds) with 795085.312 GB free disk space
----------------------------------------
-- Meryl finished successfully.  Kmer frequency histogram:
--
--  22-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2 517605005 ********************************************************************** 0.6110 0.2442
--       3-     4 189541639 *************************                                              0.7648 0.3363
--       5-     7  74239672 **********                                                             0.8758 0.4333
--       8-    11  33347172 ****                                                                   0.9367 0.5158
--      12-    16  13855307 *                                                                      0.9667 0.5768
--      17-    22   6253690                                                                        0.9799 0.6153
--      23-    29   3499104                                                                        0.9863 0.6412
--      30-    37   2212741                                                                        0.9901 0.6612
--      38-    46   1508957                                                                        0.9925 0.6779
--      47-    56   1075063                                                                        0.9942 0.6923
--      57-    67    794240                                                                        0.9954 0.7050
--      68-    79    589726                                                                        0.9963 0.7164
--      80-    92    438966                                                                        0.9970 0.7263
--      93-   106    324082                                                                        0.9975 0.7350
--     107-   121    246019                                                                        0.9979 0.7425
--     122-   137    193833                                                                        0.9982 0.7490
--     138-   154    158007                                                                        0.9984 0.7549
--     155-   172    133431                                                                        0.9986 0.7602
--     173-   191    112777                                                                        0.9987 0.7654
--     192-   211     96628                                                                        0.9989 0.7702
--     212-   232     83559                                                                        0.9990 0.7747
--     233-   254     74396                                                                        0.9991 0.7791
--     255-   277     64709                                                                        0.9992 0.7833
--     278-   301     55895                                                                        0.9992 0.7874
--     302-   326     48954                                                                        0.9993 0.7912
--     327-   352     43043                                                                        0.9994 0.7948
--     353-   379     38522                                                                        0.9994 0.7982
--     380-   407     32938                                                                        0.9995 0.8015
--     408-   436     28999                                                                        0.9995 0.8045
--     437-   466     26135                                                                        0.9995 0.8074
--     467-   497     23166                                                                        0.9996 0.8102
--     498-   529     21463                                                                        0.9996 0.8128
--     530-   562     20051                                                                        0.9996 0.8154
--     563-   596     18774                                                                        0.9996 0.8180
--     597-   631     17406                                                                        0.9997 0.8206
--     632-   667     15885                                                                        0.9997 0.8231
--     668-   704     15027                                                                        0.9997 0.8255
--     705-   742     14050                                                                        0.9997 0.8279
--     743-   781     12736                                                                        0.9997 0.8303
--     782-   821     12058                                                                        0.9997 0.8326
--
--           0 (max occurrences)
--  4239688921 (total mers, non-unique)
--   847096842 (distinct mers, non-unique)
--           0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'trimming/0-mercounts/tf.ms22'.
--
-- OVERLAPPER (normal) (trimming) erate=0.045
--
----------------------------------------
-- Starting command on Sat Sep  2 09:39:19 2023 with 795089.546 GB free disk space

    cd trimming/1-overlapper
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/overlapInCorePartition \
     -S  ../../tf.seqStore \
     -hl 160000000 \
     -rl 5000000000 \
     -ol 500 \
     -o  ./tf.partition \
    > ./tf.partition.err 2>&1

-- Finished on Sat Sep  2 09:39:19 2023 (like a bat out of hell) with 795089.515 GB free disk space
----------------------------------------
--
-- Configured 420 overlapInCore jobs.
-- Finished stage 'obt-overlapConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'obtovl' concurrent execution on Sat Sep  2 09:39:19 2023 with 795089.468 GB free disk space (420 processes; 5 concurrently)

    cd trimming/1-overlapper
    ./overlap.sh 1 > ./overlap.000001.out 2>&1
    ./overlap.sh 2 > ./overlap.000002.out 2>&1
    ./overlap.sh 3 > ./overlap.000003.out 2>&1
    ./overlap.sh 4 > ./overlap.000004.out 2>&1
    ./overlap.sh 5 > ./overlap.000005.out 2>&1
    ./overlap.sh 6 > ./overlap.000006.out 2>&1
    ./overlap.sh 7 > ./overlap.000007.out 2>&1
    ./overlap.sh 8 > ./overlap.000008.out 2>&1
    ./overlap.sh 9 > ./overlap.000009.out 2>&1
    ./overlap.sh 10 > ./overlap.000010.out 2>&1
    ./overlap.sh 11 > ./overlap.000011.out 2>&1
    ./overlap.sh 12 > ./overlap.000012.out 2>&1
    ./overlap.sh 13 > ./overlap.000013.out 2>&1
    ./overlap.sh 14 > ./overlap.000014.out 2>&1
    ./overlap.sh 15 > ./overlap.000015.out 2>&1
    ./overlap.sh 16 > ./overlap.000016.out 2>&1
    ./overlap.sh 17 > ./overlap.000017.out 2>&1
    ./overlap.sh 18 > ./overlap.000018.out 2>&1
    ./overlap.sh 19 > ./overlap.000019.out 2>&1
    ./overlap.sh 20 > ./overlap.000020.out 2>&1
    ./overlap.sh 21 > ./overlap.000021.out 2>&1
    ./overlap.sh 22 > ./overlap.000022.out 2>&1
    ./overlap.sh 23 > ./overlap.000023.out 2>&1
    ./overlap.sh 24 > ./overlap.000024.out 2>&1
    ./overlap.sh 25 > ./overlap.000025.out 2>&1
    ./overlap.sh 26 > ./overlap.000026.out 2>&1
    ./overlap.sh 27 > ./overlap.000027.out 2>&1
    ./overlap.sh 28 > ./overlap.000028.out 2>&1
    ./overlap.sh 29 > ./overlap.000029.out 2>&1
    ./overlap.sh 30 > ./overlap.000030.out 2>&1
    ./overlap.sh 31 > ./overlap.000031.out 2>&1
    ./overlap.sh 32 > ./overlap.000032.out 2>&1
    ./overlap.sh 33 > ./overlap.000033.out 2>&1
    ./overlap.sh 34 > ./overlap.000034.out 2>&1
    ./overlap.sh 35 > ./overlap.000035.out 2>&1
    ./overlap.sh 36 > ./overlap.000036.out 2>&1
    ./overlap.sh 37 > ./overlap.000037.out 2>&1
    ./overlap.sh 38 > ./overlap.000038.out 2>&1
    ./overlap.sh 39 > ./overlap.000039.out 2>&1
    ./overlap.sh 40 > ./overlap.000040.out 2>&1
    ./overlap.sh 41 > ./overlap.000041.out 2>&1
    ./overlap.sh 42 > ./overlap.000042.out 2>&1
    ./overlap.sh 43 > ./overlap.000043.out 2>&1
    ./overlap.sh 44 > ./overlap.000044.out 2>&1
    ./overlap.sh 45 > ./overlap.000045.out 2>&1
    ./overlap.sh 46 > ./overlap.000046.out 2>&1
    ./overlap.sh 47 > ./overlap.000047.out 2>&1
    ./overlap.sh 48 > ./overlap.000048.out 2>&1
    ./overlap.sh 49 > ./overlap.000049.out 2>&1
    ./overlap.sh 50 > ./overlap.000050.out 2>&1
    ./overlap.sh 51 > ./overlap.000051.out 2>&1
    ./overlap.sh 52 > ./overlap.000052.out 2>&1
    ./overlap.sh 53 > ./overlap.000053.out 2>&1
    ./overlap.sh 54 > ./overlap.000054.out 2>&1
    ./overlap.sh 55 > ./overlap.000055.out 2>&1
    ./overlap.sh 56 > ./overlap.000056.out 2>&1
    ./overlap.sh 57 > ./overlap.000057.out 2>&1
    ./overlap.sh 58 > ./overlap.000058.out 2>&1
    ./overlap.sh 59 > ./overlap.000059.out 2>&1
    ./overlap.sh 60 > ./overlap.000060.out 2>&1
    ./overlap.sh 61 > ./overlap.000061.out 2>&1
    ./overlap.sh 62 > ./overlap.000062.out 2>&1
    ./overlap.sh 63 > ./overlap.000063.out 2>&1
    ./overlap.sh 64 > ./overlap.000064.out 2>&1
    ./overlap.sh 65 > ./overlap.000065.out 2>&1
    ./overlap.sh 66 > ./overlap.000066.out 2>&1
    ./overlap.sh 67 > ./overlap.000067.out 2>&1
    ./overlap.sh 68 > ./overlap.000068.out 2>&1
    ./overlap.sh 69 > ./overlap.000069.out 2>&1
    ./overlap.sh 70 > ./overlap.000070.out 2>&1
    ./overlap.sh 71 > ./overlap.000071.out 2>&1
    ./overlap.sh 72 > ./overlap.000072.out 2>&1
    ./overlap.sh 73 > ./overlap.000073.out 2>&1
    ./overlap.sh 74 > ./overlap.000074.out 2>&1
    ./overlap.sh 75 > ./overlap.000075.out 2>&1
    ./overlap.sh 76 > ./overlap.000076.out 2>&1
    ./overlap.sh 77 > ./overlap.000077.out 2>&1
    ./overlap.sh 78 > ./overlap.000078.out 2>&1
    ./overlap.sh 79 > ./overlap.000079.out 2>&1
    ./overlap.sh 80 > ./overlap.000080.out 2>&1
    ./overlap.sh 81 > ./overlap.000081.out 2>&1
    ./overlap.sh 82 > ./overlap.000082.out 2>&1
    ./overlap.sh 83 > ./overlap.000083.out 2>&1
    ./overlap.sh 84 > ./overlap.000084.out 2>&1
    ./overlap.sh 85 > ./overlap.000085.out 2>&1
    ./overlap.sh 86 > ./overlap.000086.out 2>&1
    ./overlap.sh 87 > ./overlap.000087.out 2>&1
    ./overlap.sh 88 > ./overlap.000088.out 2>&1
    ./overlap.sh 89 > ./overlap.000089.out 2>&1
    ./overlap.sh 90 > ./overlap.000090.out 2>&1
    ./overlap.sh 91 > ./overlap.000091.out 2>&1
    ./overlap.sh 92 > ./overlap.000092.out 2>&1
    ./overlap.sh 93 > ./overlap.000093.out 2>&1
    ./overlap.sh 94 > ./overlap.000094.out 2>&1
    ./overlap.sh 95 > ./overlap.000095.out 2>&1
    ./overlap.sh 96 > ./overlap.000096.out 2>&1
    ./overlap.sh 97 > ./overlap.000097.out 2>&1
    ./overlap.sh 98 > ./overlap.000098.out 2>&1
    ./overlap.sh 99 > ./overlap.000099.out 2>&1
    ./overlap.sh 100 > ./overlap.000100.out 2>&1
    ./overlap.sh 101 > ./overlap.000101.out 2>&1
    ./overlap.sh 102 > ./overlap.000102.out 2>&1
    ./overlap.sh 103 > ./overlap.000103.out 2>&1
    ./overlap.sh 104 > ./overlap.000104.out 2>&1
    ./overlap.sh 105 > ./overlap.000105.out 2>&1
    ./overlap.sh 106 > ./overlap.000106.out 2>&1
    ./overlap.sh 107 > ./overlap.000107.out 2>&1
    ./overlap.sh 108 > ./overlap.000108.out 2>&1
    ./overlap.sh 109 > ./overlap.000109.out 2>&1
    ./overlap.sh 110 > ./overlap.000110.out 2>&1
    ./overlap.sh 111 > ./overlap.000111.out 2>&1
    ./overlap.sh 112 > ./overlap.000112.out 2>&1
    ./overlap.sh 113 > ./overlap.000113.out 2>&1
    ./overlap.sh 114 > ./overlap.000114.out 2>&1
    ./overlap.sh 115 > ./overlap.000115.out 2>&1
    ./overlap.sh 116 > ./overlap.000116.out 2>&1
    ./overlap.sh 117 > ./overlap.000117.out 2>&1
    ./overlap.sh 118 > ./overlap.000118.out 2>&1
    ./overlap.sh 119 > ./overlap.000119.out 2>&1
    ./overlap.sh 120 > ./overlap.000120.out 2>&1
    ./overlap.sh 121 > ./overlap.000121.out 2>&1
    ./overlap.sh 122 > ./overlap.000122.out 2>&1
    ./overlap.sh 123 > ./overlap.000123.out 2>&1
    ./overlap.sh 124 > ./overlap.000124.out 2>&1
    ./overlap.sh 125 > ./overlap.000125.out 2>&1
    ./overlap.sh 126 > ./overlap.000126.out 2>&1
    ./overlap.sh 127 > ./overlap.000127.out 2>&1
    ./overlap.sh 128 > ./overlap.000128.out 2>&1
    ./overlap.sh 129 > ./overlap.000129.out 2>&1
    ./overlap.sh 130 > ./overlap.000130.out 2>&1
    ./overlap.sh 131 > ./overlap.000131.out 2>&1
    ./overlap.sh 132 > ./overlap.000132.out 2>&1
    ./overlap.sh 133 > ./overlap.000133.out 2>&1
    ./overlap.sh 134 > ./overlap.000134.out 2>&1
    ./overlap.sh 135 > ./overlap.000135.out 2>&1
    ./overlap.sh 136 > ./overlap.000136.out 2>&1
    ./overlap.sh 137 > ./overlap.000137.out 2>&1
    ./overlap.sh 138 > ./overlap.000138.out 2>&1
    ./overlap.sh 139 > ./overlap.000139.out 2>&1
    ./overlap.sh 140 > ./overlap.000140.out 2>&1
    ./overlap.sh 141 > ./overlap.000141.out 2>&1
    ./overlap.sh 142 > ./overlap.000142.out 2>&1
    ./overlap.sh 143 > ./overlap.000143.out 2>&1
    ./overlap.sh 144 > ./overlap.000144.out 2>&1
    ./overlap.sh 145 > ./overlap.000145.out 2>&1
    ./overlap.sh 146 > ./overlap.000146.out 2>&1
    ./overlap.sh 147 > ./overlap.000147.out 2>&1
    ./overlap.sh 148 > ./overlap.000148.out 2>&1
    ./overlap.sh 149 > ./overlap.000149.out 2>&1
    ./overlap.sh 150 > ./overlap.000150.out 2>&1
    ./overlap.sh 151 > ./overlap.000151.out 2>&1
    ./overlap.sh 152 > ./overlap.000152.out 2>&1
    ./overlap.sh 153 > ./overlap.000153.out 2>&1
    ./overlap.sh 154 > ./overlap.000154.out 2>&1
    ./overlap.sh 155 > ./overlap.000155.out 2>&1
    ./overlap.sh 156 > ./overlap.000156.out 2>&1
    ./overlap.sh 157 > ./overlap.000157.out 2>&1
    ./overlap.sh 158 > ./overlap.000158.out 2>&1
    ./overlap.sh 159 > ./overlap.000159.out 2>&1
    ./overlap.sh 160 > ./overlap.000160.out 2>&1
    ./overlap.sh 161 > ./overlap.000161.out 2>&1
    ./overlap.sh 162 > ./overlap.000162.out 2>&1
    ./overlap.sh 163 > ./overlap.000163.out 2>&1
    ./overlap.sh 164 > ./overlap.000164.out 2>&1
    ./overlap.sh 165 > ./overlap.000165.out 2>&1
    ./overlap.sh 166 > ./overlap.000166.out 2>&1
    ./overlap.sh 167 > ./overlap.000167.out 2>&1
    ./overlap.sh 168 > ./overlap.000168.out 2>&1
    ./overlap.sh 169 > ./overlap.000169.out 2>&1
    ./overlap.sh 170 > ./overlap.000170.out 2>&1
    ./overlap.sh 171 > ./overlap.000171.out 2>&1
    ./overlap.sh 172 > ./overlap.000172.out 2>&1
    ./overlap.sh 173 > ./overlap.000173.out 2>&1
    ./overlap.sh 174 > ./overlap.000174.out 2>&1
    ./overlap.sh 175 > ./overlap.000175.out 2>&1
    ./overlap.sh 176 > ./overlap.000176.out 2>&1
    ./overlap.sh 177 > ./overlap.000177.out 2>&1
    ./overlap.sh 178 > ./overlap.000178.out 2>&1
    ./overlap.sh 179 > ./overlap.000179.out 2>&1
    ./overlap.sh 180 > ./overlap.000180.out 2>&1
    ./overlap.sh 181 > ./overlap.000181.out 2>&1
    ./overlap.sh 182 > ./overlap.000182.out 2>&1
    ./overlap.sh 183 > ./overlap.000183.out 2>&1
    ./overlap.sh 184 > ./overlap.000184.out 2>&1
    ./overlap.sh 185 > ./overlap.000185.out 2>&1
    ./overlap.sh 186 > ./overlap.000186.out 2>&1
    ./overlap.sh 187 > ./overlap.000187.out 2>&1
    ./overlap.sh 188 > ./overlap.000188.out 2>&1
    ./overlap.sh 189 > ./overlap.000189.out 2>&1
    ./overlap.sh 190 > ./overlap.000190.out 2>&1
    ./overlap.sh 191 > ./overlap.000191.out 2>&1
    ./overlap.sh 192 > ./overlap.000192.out 2>&1
    ./overlap.sh 193 > ./overlap.000193.out 2>&1
    ./overlap.sh 194 > ./overlap.000194.out 2>&1
    ./overlap.sh 195 > ./overlap.000195.out 2>&1
    ./overlap.sh 196 > ./overlap.000196.out 2>&1
    ./overlap.sh 197 > ./overlap.000197.out 2>&1
    ./overlap.sh 198 > ./overlap.000198.out 2>&1
    ./overlap.sh 199 > ./overlap.000199.out 2>&1
    ./overlap.sh 200 > ./overlap.000200.out 2>&1
    ./overlap.sh 201 > ./overlap.000201.out 2>&1
    ./overlap.sh 202 > ./overlap.000202.out 2>&1
    ./overlap.sh 203 > ./overlap.000203.out 2>&1
    ./overlap.sh 204 > ./overlap.000204.out 2>&1
    ./overlap.sh 205 > ./overlap.000205.out 2>&1
    ./overlap.sh 206 > ./overlap.000206.out 2>&1
    ./overlap.sh 207 > ./overlap.000207.out 2>&1
    ./overlap.sh 208 > ./overlap.000208.out 2>&1
    ./overlap.sh 209 > ./overlap.000209.out 2>&1
    ./overlap.sh 210 > ./overlap.000210.out 2>&1
    ./overlap.sh 211 > ./overlap.000211.out 2>&1
    ./overlap.sh 212 > ./overlap.000212.out 2>&1
    ./overlap.sh 213 > ./overlap.000213.out 2>&1
    ./overlap.sh 214 > ./overlap.000214.out 2>&1
    ./overlap.sh 215 > ./overlap.000215.out 2>&1
    ./overlap.sh 216 > ./overlap.000216.out 2>&1
    ./overlap.sh 217 > ./overlap.000217.out 2>&1
    ./overlap.sh 218 > ./overlap.000218.out 2>&1
    ./overlap.sh 219 > ./overlap.000219.out 2>&1
    ./overlap.sh 220 > ./overlap.000220.out 2>&1
    ./overlap.sh 221 > ./overlap.000221.out 2>&1
    ./overlap.sh 222 > ./overlap.000222.out 2>&1
    ./overlap.sh 223 > ./overlap.000223.out 2>&1
    ./overlap.sh 224 > ./overlap.000224.out 2>&1
    ./overlap.sh 225 > ./overlap.000225.out 2>&1
    ./overlap.sh 226 > ./overlap.000226.out 2>&1
    ./overlap.sh 227 > ./overlap.000227.out 2>&1
    ./overlap.sh 228 > ./overlap.000228.out 2>&1
    ./overlap.sh 229 > ./overlap.000229.out 2>&1
    ./overlap.sh 230 > ./overlap.000230.out 2>&1
    ./overlap.sh 231 > ./overlap.000231.out 2>&1
    ./overlap.sh 232 > ./overlap.000232.out 2>&1
    ./overlap.sh 233 > ./overlap.000233.out 2>&1
    ./overlap.sh 234 > ./overlap.000234.out 2>&1
    ./overlap.sh 235 > ./overlap.000235.out 2>&1
    ./overlap.sh 236 > ./overlap.000236.out 2>&1
    ./overlap.sh 237 > ./overlap.000237.out 2>&1
    ./overlap.sh 238 > ./overlap.000238.out 2>&1
    ./overlap.sh 239 > ./overlap.000239.out 2>&1
    ./overlap.sh 240 > ./overlap.000240.out 2>&1
    ./overlap.sh 241 > ./overlap.000241.out 2>&1
    ./overlap.sh 242 > ./overlap.000242.out 2>&1
    ./overlap.sh 243 > ./overlap.000243.out 2>&1
    ./overlap.sh 244 > ./overlap.000244.out 2>&1
    ./overlap.sh 245 > ./overlap.000245.out 2>&1
    ./overlap.sh 246 > ./overlap.000246.out 2>&1
    ./overlap.sh 247 > ./overlap.000247.out 2>&1
    ./overlap.sh 248 > ./overlap.000248.out 2>&1
    ./overlap.sh 249 > ./overlap.000249.out 2>&1
    ./overlap.sh 250 > ./overlap.000250.out 2>&1
    ./overlap.sh 251 > ./overlap.000251.out 2>&1
    ./overlap.sh 252 > ./overlap.000252.out 2>&1
    ./overlap.sh 253 > ./overlap.000253.out 2>&1
    ./overlap.sh 254 > ./overlap.000254.out 2>&1
    ./overlap.sh 255 > ./overlap.000255.out 2>&1
    ./overlap.sh 256 > ./overlap.000256.out 2>&1
    ./overlap.sh 257 > ./overlap.000257.out 2>&1
    ./overlap.sh 258 > ./overlap.000258.out 2>&1
    ./overlap.sh 259 > ./overlap.000259.out 2>&1
    ./overlap.sh 260 > ./overlap.000260.out 2>&1
    ./overlap.sh 261 > ./overlap.000261.out 2>&1
    ./overlap.sh 262 > ./overlap.000262.out 2>&1
    ./overlap.sh 263 > ./overlap.000263.out 2>&1
    ./overlap.sh 264 > ./overlap.000264.out 2>&1
    ./overlap.sh 265 > ./overlap.000265.out 2>&1
    ./overlap.sh 266 > ./overlap.000266.out 2>&1
    ./overlap.sh 267 > ./overlap.000267.out 2>&1
    ./overlap.sh 268 > ./overlap.000268.out 2>&1
    ./overlap.sh 269 > ./overlap.000269.out 2>&1
    ./overlap.sh 270 > ./overlap.000270.out 2>&1
    ./overlap.sh 271 > ./overlap.000271.out 2>&1
    ./overlap.sh 272 > ./overlap.000272.out 2>&1
    ./overlap.sh 273 > ./overlap.000273.out 2>&1
    ./overlap.sh 274 > ./overlap.000274.out 2>&1
    ./overlap.sh 275 > ./overlap.000275.out 2>&1
    ./overlap.sh 276 > ./overlap.000276.out 2>&1
    ./overlap.sh 277 > ./overlap.000277.out 2>&1
    ./overlap.sh 278 > ./overlap.000278.out 2>&1
    ./overlap.sh 279 > ./overlap.000279.out 2>&1
    ./overlap.sh 280 > ./overlap.000280.out 2>&1
    ./overlap.sh 281 > ./overlap.000281.out 2>&1
    ./overlap.sh 282 > ./overlap.000282.out 2>&1
    ./overlap.sh 283 > ./overlap.000283.out 2>&1
    ./overlap.sh 284 > ./overlap.000284.out 2>&1
    ./overlap.sh 285 > ./overlap.000285.out 2>&1
    ./overlap.sh 286 > ./overlap.000286.out 2>&1
    ./overlap.sh 287 > ./overlap.000287.out 2>&1
    ./overlap.sh 288 > ./overlap.000288.out 2>&1
    ./overlap.sh 289 > ./overlap.000289.out 2>&1
    ./overlap.sh 290 > ./overlap.000290.out 2>&1
    ./overlap.sh 291 > ./overlap.000291.out 2>&1
    ./overlap.sh 292 > ./overlap.000292.out 2>&1
    ./overlap.sh 293 > ./overlap.000293.out 2>&1
    ./overlap.sh 294 > ./overlap.000294.out 2>&1
    ./overlap.sh 295 > ./overlap.000295.out 2>&1
    ./overlap.sh 296 > ./overlap.000296.out 2>&1
    ./overlap.sh 297 > ./overlap.000297.out 2>&1
    ./overlap.sh 298 > ./overlap.000298.out 2>&1
    ./overlap.sh 299 > ./overlap.000299.out 2>&1
    ./overlap.sh 300 > ./overlap.000300.out 2>&1
    ./overlap.sh 301 > ./overlap.000301.out 2>&1
    ./overlap.sh 302 > ./overlap.000302.out 2>&1
    ./overlap.sh 303 > ./overlap.000303.out 2>&1
    ./overlap.sh 304 > ./overlap.000304.out 2>&1
    ./overlap.sh 305 > ./overlap.000305.out 2>&1
    ./overlap.sh 306 > ./overlap.000306.out 2>&1
    ./overlap.sh 307 > ./overlap.000307.out 2>&1
    ./overlap.sh 308 > ./overlap.000308.out 2>&1
    ./overlap.sh 309 > ./overlap.000309.out 2>&1
    ./overlap.sh 310 > ./overlap.000310.out 2>&1
    ./overlap.sh 311 > ./overlap.000311.out 2>&1
    ./overlap.sh 312 > ./overlap.000312.out 2>&1
    ./overlap.sh 313 > ./overlap.000313.out 2>&1
    ./overlap.sh 314 > ./overlap.000314.out 2>&1
    ./overlap.sh 315 > ./overlap.000315.out 2>&1
    ./overlap.sh 316 > ./overlap.000316.out 2>&1
    ./overlap.sh 317 > ./overlap.000317.out 2>&1
    ./overlap.sh 318 > ./overlap.000318.out 2>&1
    ./overlap.sh 319 > ./overlap.000319.out 2>&1
    ./overlap.sh 320 > ./overlap.000320.out 2>&1
    ./overlap.sh 321 > ./overlap.000321.out 2>&1
    ./overlap.sh 322 > ./overlap.000322.out 2>&1
    ./overlap.sh 323 > ./overlap.000323.out 2>&1
    ./overlap.sh 324 > ./overlap.000324.out 2>&1
    ./overlap.sh 325 > ./overlap.000325.out 2>&1
    ./overlap.sh 326 > ./overlap.000326.out 2>&1
    ./overlap.sh 327 > ./overlap.000327.out 2>&1
    ./overlap.sh 328 > ./overlap.000328.out 2>&1
    ./overlap.sh 329 > ./overlap.000329.out 2>&1
    ./overlap.sh 330 > ./overlap.000330.out 2>&1
    ./overlap.sh 331 > ./overlap.000331.out 2>&1
    ./overlap.sh 332 > ./overlap.000332.out 2>&1
    ./overlap.sh 333 > ./overlap.000333.out 2>&1
    ./overlap.sh 334 > ./overlap.000334.out 2>&1
    ./overlap.sh 335 > ./overlap.000335.out 2>&1
    ./overlap.sh 336 > ./overlap.000336.out 2>&1
    ./overlap.sh 337 > ./overlap.000337.out 2>&1
    ./overlap.sh 338 > ./overlap.000338.out 2>&1
    ./overlap.sh 339 > ./overlap.000339.out 2>&1
    ./overlap.sh 340 > ./overlap.000340.out 2>&1
    ./overlap.sh 341 > ./overlap.000341.out 2>&1
    ./overlap.sh 342 > ./overlap.000342.out 2>&1
    ./overlap.sh 343 > ./overlap.000343.out 2>&1
    ./overlap.sh 344 > ./overlap.000344.out 2>&1
    ./overlap.sh 345 > ./overlap.000345.out 2>&1
    ./overlap.sh 346 > ./overlap.000346.out 2>&1
    ./overlap.sh 347 > ./overlap.000347.out 2>&1
    ./overlap.sh 348 > ./overlap.000348.out 2>&1
    ./overlap.sh 349 > ./overlap.000349.out 2>&1
    ./overlap.sh 350 > ./overlap.000350.out 2>&1
    ./overlap.sh 351 > ./overlap.000351.out 2>&1
    ./overlap.sh 352 > ./overlap.000352.out 2>&1
    ./overlap.sh 353 > ./overlap.000353.out 2>&1
    ./overlap.sh 354 > ./overlap.000354.out 2>&1
    ./overlap.sh 355 > ./overlap.000355.out 2>&1
    ./overlap.sh 356 > ./overlap.000356.out 2>&1
    ./overlap.sh 357 > ./overlap.000357.out 2>&1
    ./overlap.sh 358 > ./overlap.000358.out 2>&1
    ./overlap.sh 359 > ./overlap.000359.out 2>&1
    ./overlap.sh 360 > ./overlap.000360.out 2>&1
    ./overlap.sh 361 > ./overlap.000361.out 2>&1
    ./overlap.sh 362 > ./overlap.000362.out 2>&1
    ./overlap.sh 363 > ./overlap.000363.out 2>&1
    ./overlap.sh 364 > ./overlap.000364.out 2>&1
    ./overlap.sh 365 > ./overlap.000365.out 2>&1
    ./overlap.sh 366 > ./overlap.000366.out 2>&1
    ./overlap.sh 367 > ./overlap.000367.out 2>&1
    ./overlap.sh 368 > ./overlap.000368.out 2>&1
    ./overlap.sh 369 > ./overlap.000369.out 2>&1
    ./overlap.sh 370 > ./overlap.000370.out 2>&1
    ./overlap.sh 371 > ./overlap.000371.out 2>&1
    ./overlap.sh 372 > ./overlap.000372.out 2>&1
    ./overlap.sh 373 > ./overlap.000373.out 2>&1
    ./overlap.sh 374 > ./overlap.000374.out 2>&1
    ./overlap.sh 375 > ./overlap.000375.out 2>&1
    ./overlap.sh 376 > ./overlap.000376.out 2>&1
    ./overlap.sh 377 > ./overlap.000377.out 2>&1
    ./overlap.sh 378 > ./overlap.000378.out 2>&1
    ./overlap.sh 379 > ./overlap.000379.out 2>&1
    ./overlap.sh 380 > ./overlap.000380.out 2>&1
    ./overlap.sh 381 > ./overlap.000381.out 2>&1
    ./overlap.sh 382 > ./overlap.000382.out 2>&1
    ./overlap.sh 383 > ./overlap.000383.out 2>&1
    ./overlap.sh 384 > ./overlap.000384.out 2>&1
    ./overlap.sh 385 > ./overlap.000385.out 2>&1
    ./overlap.sh 386 > ./overlap.000386.out 2>&1
    ./overlap.sh 387 > ./overlap.000387.out 2>&1
    ./overlap.sh 388 > ./overlap.000388.out 2>&1
    ./overlap.sh 389 > ./overlap.000389.out 2>&1
    ./overlap.sh 390 > ./overlap.000390.out 2>&1
    ./overlap.sh 391 > ./overlap.000391.out 2>&1
    ./overlap.sh 392 > ./overlap.000392.out 2>&1
    ./overlap.sh 393 > ./overlap.000393.out 2>&1
    ./overlap.sh 394 > ./overlap.000394.out 2>&1
    ./overlap.sh 395 > ./overlap.000395.out 2>&1
    ./overlap.sh 396 > ./overlap.000396.out 2>&1
    ./overlap.sh 397 > ./overlap.000397.out 2>&1
    ./overlap.sh 398 > ./overlap.000398.out 2>&1
    ./overlap.sh 399 > ./overlap.000399.out 2>&1
    ./overlap.sh 400 > ./overlap.000400.out 2>&1
    ./overlap.sh 401 > ./overlap.000401.out 2>&1
    ./overlap.sh 402 > ./overlap.000402.out 2>&1
    ./overlap.sh 403 > ./overlap.000403.out 2>&1
    ./overlap.sh 404 > ./overlap.000404.out 2>&1
    ./overlap.sh 405 > ./overlap.000405.out 2>&1
    ./overlap.sh 406 > ./overlap.000406.out 2>&1
    ./overlap.sh 407 > ./overlap.000407.out 2>&1
    ./overlap.sh 408 > ./overlap.000408.out 2>&1
    ./overlap.sh 409 > ./overlap.000409.out 2>&1
    ./overlap.sh 410 > ./overlap.000410.out 2>&1
    ./overlap.sh 411 > ./overlap.000411.out 2>&1
    ./overlap.sh 412 > ./overlap.000412.out 2>&1
    ./overlap.sh 413 > ./overlap.000413.out 2>&1
    ./overlap.sh 414 > ./overlap.000414.out 2>&1
    ./overlap.sh 415 > ./overlap.000415.out 2>&1
    ./overlap.sh 416 > ./overlap.000416.out 2>&1
    ./overlap.sh 417 > ./overlap.000417.out 2>&1
    ./overlap.sh 418 > ./overlap.000418.out 2>&1
    ./overlap.sh 419 > ./overlap.000419.out 2>&1
    ./overlap.sh 420 > ./overlap.000420.out 2>&1

-- Finished on Sat Sep  2 20:32:48 2023 (39209 seconds) with 798077.515 GB free disk space
----------------------------------------
-- Found 420 overlapInCore output files.
--
-- overlapInCore compute 'trimming/1-overlapper':
--   kmer hits
--     with no overlap      10874247601  65.7166667 +- 10065973.723
--     with an overlap          3958869  7857142857 +- 3780.123
--
--   overlaps                   3958866  7142857143 +- 3780.116
--     contained                      0           0 +- 0
--     dovetail                       0           0 +- 0
--
--   overlaps rejected
--     multiple per pair              0           0 +- 0
--     bad short window               0           0 +- 0
--     bad long window                0           0 +- 0
-- Finished stage 'obt-overlapCheck', reset canuIteration.
----------------------------------------
-- Starting command on Sat Sep  2 20:32:48 2023 with 798077.468 GB free disk space

    cd trimming
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/ovStoreConfig \
     -S ../tf.seqStore \
     -M 4-8 \
     -L ./1-overlapper/ovljob.files \
     -create ./tf.ovlStore.config \
     > ./tf.ovlStore.config.txt \
    2> ./tf.ovlStore.config.err

-- Finished on Sat Sep  2 20:33:06 2023 (18 seconds) with 798077.437 GB free disk space
----------------------------------------
--
-- Creating overlap store trimming/tf.ovlStore using:
--      2 buckets
--      2 slices
--        using at most 1 GB memory each
-- Finished stage 'obt-overlapStoreConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovB' concurrent execution on Sat Sep  2 20:33:06 2023 with 798077.406 GB free disk space (2 processes; 40 concurrently)

    cd trimming/tf.ovlStore.BUILDING
    ./scripts/1-bucketize.sh 1 > ./logs/1-bucketize.000001.out 2>&1
    ./scripts/1-bucketize.sh 2 > ./logs/1-bucketize.000002.out 2>&1

-- Finished on Sat Sep  2 20:33:10 2023 (4 seconds) with 798050.718 GB free disk space
----------------------------------------
-- Overlap store bucketizer finished.
-- Finished stage 'obt-overlapStoreBucketizerCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovS' concurrent execution on Sat Sep  2 20:33:10 2023 with 798050.687 GB free disk space (2 processes; 40 concurrently)

    cd trimming/tf.ovlStore.BUILDING
    ./scripts/2-sort.sh 1 > ./logs/2-sort.000001.out 2>&1
    ./scripts/2-sort.sh 2 > ./logs/2-sort.000002.out 2>&1

-- Finished on Sat Sep  2 20:33:11 2023 (one second) with 798050.343 GB free disk space
----------------------------------------
-- Overlap store sorter finished.
-- Finished stage 'obt-overlapStoreSorterCheck', reset canuIteration.
----------------------------------------
-- Starting command on Sat Sep  2 20:33:11 2023 with 798050.328 GB free disk space

    cd trimming
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/ovStoreIndexer \
      -O  ./tf.ovlStore.BUILDING \
      -S ../tf.seqStore \
      -C  ./tf.ovlStore.config \
      -delete \
    > ./tf.ovlStore.BUILDING.index.err 2>&1

-- Finished on Sat Sep  2 20:33:12 2023 (one second) with 798050.187 GB free disk space
----------------------------------------
-- Overlap store indexer finished.
-- Checking store.
----------------------------------------
-- Starting command on Sat Sep  2 20:33:12 2023 with 798050.187 GB free disk space

    cd trimming
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/ovStoreDump \
     -S ../tf.seqStore \
     -O  ./tf.ovlStore \
     -counts \
     > ./tf.ovlStore/counts.dat 2> ./tf.ovlStore/counts.err

-- Finished on Sat Sep  2 20:33:12 2023 (like a bat out of hell) with 798050.171 GB free disk space
----------------------------------------
--
-- Overlap store 'trimming/tf.ovlStore' successfully constructed.
-- Found 7917738 overlaps for 135218 reads; 1275855 reads have no overlaps.
--
--
-- Purged 2.29 GB in 1260 overlap output files.
-- Finished stage 'obt-createOverlapStore', reset canuIteration.
----------------------------------------
-- Starting command on Sat Sep  2 20:33:13 2023 with 798050.828 GB free disk space

    cd trimming/3-overlapbasedtrimming
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/trimReads \
      -S  ../../tf.seqStore \
      -O  ../tf.ovlStore \
      -Co ./tf.1.trimReads.clear \
      -e  0.045 \
      -minlength 1000 \
      -ol 500 \
      -oc 2 \
      -o  ./tf.1.trimReads \
    >     ./tf.1.trimReads.err 2>&1

-- Finished on Sat Sep  2 20:33:19 2023 (6 seconds) with 798050.812 GB free disk space
----------------------------------------
--  PARAMETERS:
--  ----------
--     1000    (reads trimmed below this many bases are deleted)
--   0.0450    (use overlaps at or below this fraction error)
--      500    (break region if overlap is less than this long, for 'largest covered' algorithm)
--        2    (break region if overlap coverage is less than this many reads, for 'largest covered' algorithm)
--  
--  INPUT READS:
--  -----------
--  1411073 reads  23257214780 bases (reads processed)
--       0 reads            0 bases (reads not processed, previously deleted)
--       0 reads            0 bases (reads not processed, in a library where trimming isn't allowed)
--  
--  OUTPUT READS:
--  ------------
--   27223 reads     51357655 bases (trimmed reads output)
--       9 reads        18796 bases (reads with no change, kept as is)
--  1275855 reads  19820693833 bases (reads with no overlaps, deleted)
--  107986 reads   2738001899 bases (reads with short trimmed length, deleted)
--  
--  TRIMMING DETAILS:
--  ----------------
--   26957 reads    366012675 bases (bases trimmed from the 5' end of a read)
--   26818 reads    281129922 bases (bases trimmed from the 3' end of a read)
-- Finished stage 'obt-trimReads', reset canuIteration.
----------------------------------------
-- Starting command on Sat Sep  2 20:33:19 2023 with 798050.796 GB free disk space

    cd trimming/3-overlapbasedtrimming
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/splitReads \
      -S  ../../tf.seqStore \
      -O  ../tf.ovlStore \
      -Ci ./tf.1.trimReads.clear \
      -Co ./tf.2.splitReads.clear \
      -e  0.045 \
      -minlength 1000 \
      -o  ./tf.2.splitReads \
    >     ./tf.2.splitReads.err 2>&1

-- Finished on Sat Sep  2 20:33:20 2023 (one second) with 798050.781 GB free disk space
----------------------------------------
--  PARAMETERS:
--  ----------
--     1000    (reads trimmed below this many bases are deleted)
--   0.0450    (use overlaps at or below this fraction error)
--  INPUT READS:
--  -----------
--   27232 reads    698519048 bases (reads processed)
--  1383841 reads  22558695732 bases (reads not processed, previously deleted)
--       0 reads            0 bases (reads not processed, in a library where trimming isn't allowed)
--  
--  PROCESSED:
--  --------
--       0 reads            0 bases (no overlaps)
--    1503 reads     38389698 bases (no coverage after adjusting for trimming done already)
--       0 reads            0 bases (processed for chimera)
--       0 reads            0 bases (processed for spur)
--   25729 reads    660129350 bases (processed for subreads)
--  
--  READS WITH SIGNALS:
--  ------------------
--       0 reads            0 signals (number of 5' spur signal)
--       0 reads            0 signals (number of 3' spur signal)
--       0 reads            0 signals (number of chimera signal)
--       0 reads            0 signals (number of subread signal)
--  
--  SIGNALS:
--  -------
--       0 reads            0 bases (size of 5' spur signal)
--       0 reads            0 bases (size of 3' spur signal)
--       0 reads            0 bases (size of chimera signal)
--       0 reads            0 bases (size of subread signal)
--  
--  TRIMMING:
--  --------
--       0 reads            0 bases (trimmed from the 5' end of the read)
--       0 reads            0 bases (trimmed from the 3' end of the read)
-- Finished stage 'obt-splitReads', reset canuIteration.
----------------------------------------
-- Starting command on Sat Sep  2 20:33:20 2023 with 798050.765 GB free disk space

    cd trimming/3-overlapbasedtrimming
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/loadTrimmedReads \
      -S ../../tf.seqStore \
      -c ./tf.2.splitReads.clear \
    > ./tf.loadTrimmedReads.err 2>&1

-- Finished on Sat Sep  2 20:33:27 2023 (7 seconds) with 798050.593 GB free disk space
----------------------------------------
--
-- In sequence store './tf.seqStore':
--   Found 27232 reads.
--   Found 51376451 bases (0.31 times coverage).
--    Histogram of corrected-trimmed reads:
--    
--    G=51376451                         sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010         5335       732      5141089  ||       1000-1233         8608|---------------------------------------------------------------
--    00020         3746      1908     10276177  ||       1234-1467         5204|---------------------------------------
--    00030         2891      3483     15413323  ||       1468-1701         3338|-------------------------
--    00040         2316      5478     20550860  ||       1702-1935         2223|-----------------
--    00050         1929      7916     25689492  ||       1936-2169         1583|------------
--    00060         1643     10813     30826519  ||       2170-2403         1180|---------
--    00070         1427     14175     35964232  ||       2404-2637          878|-------
--    00080         1255     18018     41101773  ||       2638-2871          669|-----
--    00090         1116     22364     46239649  ||       2872-3105          596|-----
--    00100         1000     27231     51376451  ||       3106-3339          441|----
--    001.000x               27232     51376451  ||       3340-3573          367|---
--                                               ||       3574-3807          302|---
--                                               ||       3808-4041          311|---
--                                               ||       4042-4275          218|--
--                                               ||       4276-4509          170|--
--                                               ||       4510-4743          138|--
--                                               ||       4744-4977          107|-
--                                               ||       4978-5211          115|-
--                                               ||       5212-5445           92|-
--                                               ||       5446-5679           71|-
--                                               ||       5680-5913           84|-
--                                               ||       5914-6147           74|-
--                                               ||       6148-6381           51|-
--                                               ||       6382-6615           47|-
--                                               ||       6616-6849           46|-
--                                               ||       6850-7083           43|-
--                                               ||       7084-7317           23|-
--                                               ||       7318-7551           36|-
--                                               ||       7552-7785           29|-
--                                               ||       7786-8019           26|-
--                                               ||       8020-8253           25|-
--                                               ||       8254-8487           20|-
--                                               ||       8488-8721           24|-
--                                               ||       8722-8955           23|-
--                                               ||       8956-9189           13|-
--                                               ||       9190-9423            7|-
--                                               ||       9424-9657            5|-
--                                               ||       9658-9891            5|-
--                                               ||       9892-10125           5|-
--                                               ||      10126-10359           6|-
--                                               ||      10360-10593           6|-
--                                               ||      10594-10827           5|-
--                                               ||      10828-11061           3|-
--                                               ||      11062-11295           2|-
--                                               ||      11296-11529           2|-
--                                               ||      11530-11763           1|-
--                                               ||      11764-11997           5|-
--                                               ||      11998-12231           3|-
--                                               ||      12232-12465           1|-
--                                               ||      12466-12699           1|-
--    
--
-- No trimmed reads generated, overlaps used for trimming saved.
-- Finished stage 'obt-dumpReads', reset canuIteration.
----------------------------------------
-- Starting command on Sat Sep  2 20:33:27 2023 with 798050.531 GB free disk space

    cd .
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/sqStoreDumpFASTQ \
      -trimmed \
      -S ./tf.seqStore \
      -o ./tf.trimmedReads.gz \
      -fasta \
      -trimmed -normal -nolibname \
    > ./tf.trimmedReads.fasta.err 2>&1

-- Finished on Sat Sep  2 20:33:31 2023 (4 seconds) with 798050.5 GB free disk space
----------------------------------------
--
-- Trimmed reads saved in 'tf.trimmedReads.fasta.gz'.
-- Finished stage 'cor-dumpTrimmedReads', reset canuIteration.
--
-- ERROR:  Read coverage (0.31) lower than allowed.
-- ERROR:    stopOnLowCoverage = 10
-- ERROR:
-- ERROR:  This could be caused by an incorrect genomeSize or poor
-- ERROR:  quality reads that cound not be sufficiently corrected.
-- ERROR:
-- ERROR:  You can force Canu to continue by decreasing parameter
-- ERROR:  stopOnLowCoverage (and possibly minInputCoverage too).
-- ERROR:  Be warned that the quality of corrected reads and/or
-- ERROR:  contiguity of contigs will be poor.
--

ABORT:
ABORT: canu 2.2
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.
ABORT:

I have tried to add the corMhapSensitivity=normal, but nothing changed.

skoren commented 1 year ago

The corMhapSensitivity option only applies to the built-in correction which you're not using so that is why it made no difference.

The issue is your corrected reads don't really look corrected. There's no histogram peak in the k-mers that you'd expect after correction. There are also almost no overlaps being found at 4.5% error, which is why you're ending up with no coverage:

--  OUTPUT READS:
--  ------------
--   27223 reads     51357655 bases (trimmed reads output)
--       9 reads        18796 bases (reads with no change, kept as is)
--  1275855 reads  19820693833 bases (reads with no overlaps, deleted)
--  107986 reads   2738001899 bases (reads with short trimmed length, deleted)

I'd suggest running with the built-in canu correction and see what results you obtain.

yaoxkkkkk commented 1 year ago

Thank you for your kindly reply! I have tried to use Canu correction to correct and assembly the raw Pacbio data, but I got similar results, the command I use is:

canu useGrid=false \
    -p tf -d tf-pacbio \
    genomeSize=164.4m \
    corMhapSensitivity=normal \
    -pacbio /dssg/home/acct-jiang.lu/jiang.lu/Tf/Genome_data/SRR23272336_1.fastq

And the log is showing below:

-- canu 2.2
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
--   Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
--   Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
--   Genome Res. 2017 May;27(5):722-736.
--   http://doi.org/10.1101/gr.215087.116
-- 
-- Read and contig alignments during correction and consensus use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_312' (from 'java') with -d64 support.
-- Detected gnuplot version '5.2 patchlevel 4   ' (from 'gnuplot') and image format 'png'.
--
-- Detected 40 CPUs and 320000 gigabytes of memory on the local machine.
--
-- Detected Slurm with 'sinfo' binary in /usr/bin/sinfo.
--          Slurm disabled by useGrid=false
--
-- Local machine mode enabled; grid support not detected or not allowed.
--
--                                (tag)Concurrency
--                         (tag)Threads          |
--                (tag)Memory         |          |
--        (tag)             |         |          |       total usage      algorithm
--        -------  ----------  --------   --------  --------------------  -----------------------------
-- Local: meryl     24.000 GB    8 CPUs x   5 jobs   120.000 GB  40 CPUs  (k-mer counting)
-- Local: hap       12.000 GB   20 CPUs x   2 jobs    24.000 GB  40 CPUs  (read-to-haplotype assignment)
-- Local: cormhap   13.000 GB   10 CPUs x   4 jobs    52.000 GB  40 CPUs  (overlap detection with mhap)
-- Local: obtovl     8.000 GB    8 CPUs x   5 jobs    40.000 GB  40 CPUs  (overlap detection)
-- Local: utgovl     8.000 GB    8 CPUs x   5 jobs    40.000 GB  40 CPUs  (overlap detection)
-- Local: cor        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (read correction)
-- Local: ovb        4.000 GB    1 CPU  x  40 jobs   160.000 GB  40 CPUs  (overlap store bucketizer)
-- Local: ovs        8.000 GB    1 CPU  x  40 jobs   320.000 GB  40 CPUs  (overlap store sorting)
-- Local: red       16.000 GB    5 CPUs x   8 jobs   128.000 GB  40 CPUs  (read error detection)
-- Local: oea        8.000 GB    1 CPU  x  40 jobs   320.000 GB  40 CPUs  (overlap error adjustment)
-- Local: bat       64.000 GB    8 CPUs x   1 job     64.000 GB   8 CPUs  (contig construction with bogart)
-- Local: cns        -.--- GB    8 CPUs x   - jobs     -.--- GB   - CPUs  (consensus)
--
-- Found untrimmed raw PacBio CLR reads in the input files.
--
-- Generating assembly 'tf' in '/dssg/home/acct-jiang.lu/jiang.lu/Tf/way1/tf-pacbio':
--   genomeSize:
--     164400000
--
--   Overlap Generation Limits:
--     corOvlErrorRate 0.2400 ( 24.00%)
--     obtOvlErrorRate 0.0450 (  4.50%)
--     utgOvlErrorRate 0.0450 (  4.50%)
--
--   Overlap Processing Limits:
--     corErrorRate    0.2500 ( 25.00%)
--     obtErrorRate    0.0450 (  4.50%)
--     utgErrorRate    0.0450 (  4.50%)
--     cnsErrorRate    0.0750 (  7.50%)
--
--   Stages to run:
--     correct raw reads.
--     trim corrected reads.
--     assemble corrected and trimmed reads.
--
--
-- BEGIN CORRECTION
----------------------------------------
-- Starting command on Sun Sep  3 11:13:40 2023 with 799284.812 GB free disk space

    cd .
    ./tf.seqStore.sh \
    > ./tf.seqStore.err 2>&1

-- Finished on Sun Sep  3 11:18:31 2023 (291 seconds) with 799398.953 GB free disk space
----------------------------------------
--
-- In sequence store './tf.seqStore':
--   Found 1411139 reads.
--   Found 23273378703 bases (141.56 times coverage).
--    Histogram of raw reads:
--    
--    G=23273378703                      sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010        52545     36156   2327338168  ||       1000-7389       261130|------------------------------
--    00020        40702     86884   4654688563  ||       7390-13779      554721|---------------------------------------------------------------
--    00030        32780    150830   6982041779  ||      13780-20169      237248|---------------------------
--    00040        26594    229783   9309375917  ||      20170-26559      127710|---------------
--    00050        21449    327346  11636705582  ||      26560-32949       81198|----------
--    00060        17036    449211  13964032384  ||      32950-39339       53539|-------
--    00070        13663    602685  16291366224  ||      39340-45729       35069|----
--    00080        11543    788708  18618708039  ||      45730-52119       23158|---
--    00090         9377   1010164  20946041426  ||      52120-58509       15135|--
--    00100         1000   1411138  23273378703  ||      58510-64899        9343|--
--    001.000x             1411139  23273378703  ||      64900-71289        5513|-
--                                               ||      71290-77679        3186|-
--                                               ||      77680-84069        1769|-
--                                               ||      84070-90459        1090|-
--                                               ||      90460-96849         607|-
--                                               ||      96850-103239        329|-
--                                               ||     103240-109629        178|-
--                                               ||     109630-116019        100|-
--                                               ||     116020-122409         50|-
--                                               ||     122410-128799         25|-
--                                               ||     128800-135189         15|-
--                                               ||     135190-141579          8|-
--                                               ||     141580-147969          1|-
--                                               ||     147970-154359          0|
--                                               ||     154360-160749          3|-
--                                               ||     160750-167139          2|-
--                                               ||     167140-173529          1|-
--                                               ||     173530-179919          2|-
--                                               ||     179920-186309          4|-
--                                               ||     186310-192699          1|-
--                                               ||     192700-199089          0|
--                                               ||     199090-205479          0|
--                                               ||     205480-211869          1|-
--                                               ||     211870-218259          0|
--                                               ||     218260-224649          2|-
--                                               ||     224650-231039          0|
--                                               ||     231040-237429          0|
--                                               ||     237430-243819          0|
--                                               ||     243820-250209          0|
--                                               ||     250210-256599          0|
--                                               ||     256600-262989          0|
--                                               ||     262990-269379          0|
--                                               ||     269380-275769          0|
--                                               ||     275770-282159          0|
--                                               ||     282160-288549          0|
--                                               ||     288550-294939          0|
--                                               ||     294940-301329          0|
--                                               ||     301330-307719          0|
--                                               ||     307720-314109          0|
--                                               ||     314110-320499          1|-
--    
----------------------------------------
-- Starting command on Sun Sep  3 11:18:33 2023 with 799398.921 GB free disk space

    cd correction/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Sun Sep  3 11:18:34 2023 (one second) with 799398.921 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--        01 14.00 GB       3
--        02 13.50 GB       2
--        04 10.91 GB       2
--        06  7.46 GB       2
--        08  5.71 GB       2
--        12  3.98 GB       2
--        16  3.12 GB       2
--        20  2.48 GB       2
--        24  2.28 GB       2
--        32  1.65 GB       2
--        40  1.50 GB       2
--        48  1.25 GB       2
--        56  1.07 GB       2
--        64  0.94 GB       2
--        96  0.62 GB       2
--
--  For 1411139 reads with 23273378703 bases, limit to 232 batches.
--  Will count kmers using 02 jobs, each using 15 GB and 8 threads.
--
-- Finished stage 'merylConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Sun Sep  3 11:18:34 2023 with 799398.921 GB free disk space (2 processes; 5 concurrently)

    cd correction/0-mercounts
    ./meryl-count.sh 1 > ./meryl-count.000001.out 2>&1
    ./meryl-count.sh 2 > ./meryl-count.000002.out 2>&1

-- Finished on Sun Sep  3 11:26:36 2023 (482 seconds) with 798748.828 GB free disk space
----------------------------------------
-- Found 2 Kmer counting (meryl) outputs.
-- Finished stage 'cor-merylCountCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Sun Sep  3 11:26:36 2023 with 798748.828 GB free disk space (1 processes; 5 concurrently)

    cd correction/0-mercounts
    ./meryl-process.sh 1 > ./meryl-process.000001.out 2>&1

-- Finished on Sun Sep  3 11:27:14 2023 (38 seconds) with 798976.625 GB free disk space
----------------------------------------
-- Meryl finished successfully.  Kmer frequency histogram:
--
--  16-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2 201552521 *************************                                              0.1017 0.0174
--       3-     4 486987755 *************************************************************          0.2241 0.0489
--       5-     7 553940866 ********************************************************************** 0.4580 0.1386
--       8-    11 347900759 *******************************************                            0.6867 0.2730
--      12-    16 170999161 *********************                                                  0.8265 0.3956
--      17-    22  84385157 **********                                                             0.8982 0.4857
--      23-    29  45001864 *****                                                                  0.9353 0.5495
--      30-    37  26723373 ***                                                                    0.9558 0.5963
--      38-    46  17563534 **                                                                     0.9683 0.6330
--      47-    56  11614248 *                                                                      0.9767 0.6636
--      57-    67   7685728                                                                        0.9822 0.6884
--      68-    79   5297520                                                                        0.9859 0.7083
--      80-    92   3878583                                                                        0.9885 0.7246
--      93-   106   3166670                                                                        0.9904 0.7388
--     107-   121   2812153                                                                        0.9920 0.7523
--     122-   137   2193861                                                                        0.9934 0.7661
--     138-   154   1595830                                                                        0.9945 0.7781
--     155-   172   1346205                                                                        0.9953 0.7880
--     173-   191   1152753                                                                        0.9959 0.7975
--     192-   211    906803                                                                        0.9965 0.8065
--     212-   232    703670                                                                        0.9970 0.8143
--     233-   254    584340                                                                        0.9973 0.8209
--     255-   277    500318                                                                        0.9976 0.8271
--     278-   301    413453                                                                        0.9979 0.8328
--     302-   326    364479                                                                        0.9981 0.8379
--     327-   352    347201                                                                        0.9983 0.8428
--     353-   379    316787                                                                        0.9984 0.8479
--     380-   407    267574                                                                        0.9986 0.8529
--     408-   436    219121                                                                        0.9987 0.8574
--     437-   466    187230                                                                        0.9988 0.8614
--     467-   497    174488                                                                        0.9989 0.8651
--     498-   529    164479                                                                        0.9990 0.8687
--     530-   562    150514                                                                        0.9991 0.8723
--     563-   596    136607                                                                        0.9992 0.8759
--     597-   631    124724                                                                        0.9992 0.8793
--     632-   667    109797                                                                        0.9993 0.8826
--     668-   704     92704                                                                        0.9994 0.8857
--     705-   742     78733                                                                        0.9994 0.8884
--     743-   781     72281                                                                        0.9994 0.8909
--     782-   821     71300                                                                        0.9995 0.8932
--
--           0 (max occurrences)
-- 23128879630 (total mers, non-unique)
--  1982754231 (distinct mers, non-unique)
--           0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'correction/0-mercounts/tf.ms16'.
--
-- OVERLAPPER (mhap) (correction)
--
--
-- PARAMETERS: hashes=512, minMatches=3, threshold=0.78
--
-- Given 11.7 GB, can fit 35100 reads per block.
-- For 42 blocks, set stride to 10 blocks.
-- Logging partitioning to 'correction/1-overlapper/partitioning.log'.
-- Configured 41 mhap precompute jobs.
-- Configured 101 mhap overlap jobs.
-- Finished stage 'cor-mhapConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Sun Sep  3 11:27:14 2023 with 798985 GB free disk space (41 processes; 4 concurrently)

    cd correction/1-overlapper
    ./precompute.sh 1 > ./precompute.000001.out 2>&1
    ./precompute.sh 2 > ./precompute.000002.out 2>&1
    ./precompute.sh 3 > ./precompute.000003.out 2>&1
    ./precompute.sh 4 > ./precompute.000004.out 2>&1
    ./precompute.sh 5 > ./precompute.000005.out 2>&1
    ./precompute.sh 6 > ./precompute.000006.out 2>&1
    ./precompute.sh 7 > ./precompute.000007.out 2>&1
    ./precompute.sh 8 > ./precompute.000008.out 2>&1
    ./precompute.sh 9 > ./precompute.000009.out 2>&1
    ./precompute.sh 10 > ./precompute.000010.out 2>&1
    ./precompute.sh 11 > ./precompute.000011.out 2>&1
    ./precompute.sh 12 > ./precompute.000012.out 2>&1
    ./precompute.sh 13 > ./precompute.000013.out 2>&1
    ./precompute.sh 14 > ./precompute.000014.out 2>&1
    ./precompute.sh 15 > ./precompute.000015.out 2>&1
    ./precompute.sh 16 > ./precompute.000016.out 2>&1
    ./precompute.sh 17 > ./precompute.000017.out 2>&1
    ./precompute.sh 18 > ./precompute.000018.out 2>&1
    ./precompute.sh 19 > ./precompute.000019.out 2>&1
    ./precompute.sh 20 > ./precompute.000020.out 2>&1
    ./precompute.sh 21 > ./precompute.000021.out 2>&1
    ./precompute.sh 22 > ./precompute.000022.out 2>&1
    ./precompute.sh 23 > ./precompute.000023.out 2>&1
    ./precompute.sh 24 > ./precompute.000024.out 2>&1
    ./precompute.sh 25 > ./precompute.000025.out 2>&1
    ./precompute.sh 26 > ./precompute.000026.out 2>&1
    ./precompute.sh 27 > ./precompute.000027.out 2>&1
    ./precompute.sh 28 > ./precompute.000028.out 2>&1
    ./precompute.sh 29 > ./precompute.000029.out 2>&1
    ./precompute.sh 30 > ./precompute.000030.out 2>&1
    ./precompute.sh 31 > ./precompute.000031.out 2>&1
    ./precompute.sh 32 > ./precompute.000032.out 2>&1
    ./precompute.sh 33 > ./precompute.000033.out 2>&1
    ./precompute.sh 34 > ./precompute.000034.out 2>&1
    ./precompute.sh 35 > ./precompute.000035.out 2>&1
    ./precompute.sh 36 > ./precompute.000036.out 2>&1
    ./precompute.sh 37 > ./precompute.000037.out 2>&1
    ./precompute.sh 38 > ./precompute.000038.out 2>&1
    ./precompute.sh 39 > ./precompute.000039.out 2>&1
    ./precompute.sh 40 > ./precompute.000040.out 2>&1
    ./precompute.sh 41 > ./precompute.000041.out 2>&1

-- Finished on Sun Sep  3 15:07:24 2023 (13210 seconds, like watching paint dry) with 862199.531 GB free disk space
----------------------------------------
-- All 41 mhap precompute jobs finished successfully.
-- Finished stage 'cor-mhapPrecomputeCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Sun Sep  3 15:07:24 2023 with 862199.5 GB free disk space (101 processes; 4 concurrently)

    cd correction/1-overlapper
    ./mhap.sh 1 > ./mhap.000001.out 2>&1
    ./mhap.sh 2 > ./mhap.000002.out 2>&1
    ./mhap.sh 3 > ./mhap.000003.out 2>&1
    ./mhap.sh 4 > ./mhap.000004.out 2>&1
    ./mhap.sh 5 > ./mhap.000005.out 2>&1
    ./mhap.sh 6 > ./mhap.000006.out 2>&1
    ./mhap.sh 7 > ./mhap.000007.out 2>&1
    ./mhap.sh 8 > ./mhap.000008.out 2>&1
    ./mhap.sh 9 > ./mhap.000009.out 2>&1
    ./mhap.sh 10 > ./mhap.000010.out 2>&1
    ./mhap.sh 11 > ./mhap.000011.out 2>&1
    ./mhap.sh 12 > ./mhap.000012.out 2>&1
    ./mhap.sh 13 > ./mhap.000013.out 2>&1
    ./mhap.sh 14 > ./mhap.000014.out 2>&1
    ./mhap.sh 15 > ./mhap.000015.out 2>&1
    ./mhap.sh 16 > ./mhap.000016.out 2>&1
    ./mhap.sh 17 > ./mhap.000017.out 2>&1
    ./mhap.sh 18 > ./mhap.000018.out 2>&1
    ./mhap.sh 19 > ./mhap.000019.out 2>&1
    ./mhap.sh 20 > ./mhap.000020.out 2>&1
    ./mhap.sh 21 > ./mhap.000021.out 2>&1
    ./mhap.sh 22 > ./mhap.000022.out 2>&1
    ./mhap.sh 23 > ./mhap.000023.out 2>&1
    ./mhap.sh 24 > ./mhap.000024.out 2>&1
    ./mhap.sh 25 > ./mhap.000025.out 2>&1
    ./mhap.sh 26 > ./mhap.000026.out 2>&1
    ./mhap.sh 27 > ./mhap.000027.out 2>&1
    ./mhap.sh 28 > ./mhap.000028.out 2>&1
    ./mhap.sh 29 > ./mhap.000029.out 2>&1
    ./mhap.sh 30 > ./mhap.000030.out 2>&1
    ./mhap.sh 31 > ./mhap.000031.out 2>&1
    ./mhap.sh 32 > ./mhap.000032.out 2>&1
    ./mhap.sh 33 > ./mhap.000033.out 2>&1
    ./mhap.sh 34 > ./mhap.000034.out 2>&1
    ./mhap.sh 35 > ./mhap.000035.out 2>&1
    ./mhap.sh 36 > ./mhap.000036.out 2>&1
    ./mhap.sh 37 > ./mhap.000037.out 2>&1
    ./mhap.sh 38 > ./mhap.000038.out 2>&1
    ./mhap.sh 39 > ./mhap.000039.out 2>&1
    ./mhap.sh 40 > ./mhap.000040.out 2>&1
    ./mhap.sh 41 > ./mhap.000041.out 2>&1
    ./mhap.sh 42 > ./mhap.000042.out 2>&1
    ./mhap.sh 43 > ./mhap.000043.out 2>&1
    ./mhap.sh 44 > ./mhap.000044.out 2>&1
    ./mhap.sh 45 > ./mhap.000045.out 2>&1
    ./mhap.sh 46 > ./mhap.000046.out 2>&1
    ./mhap.sh 47 > ./mhap.000047.out 2>&1
    ./mhap.sh 48 > ./mhap.000048.out 2>&1
    ./mhap.sh 49 > ./mhap.000049.out 2>&1
    ./mhap.sh 50 > ./mhap.000050.out 2>&1
    ./mhap.sh 51 > ./mhap.000051.out 2>&1
    ./mhap.sh 52 > ./mhap.000052.out 2>&1
    ./mhap.sh 53 > ./mhap.000053.out 2>&1
    ./mhap.sh 54 > ./mhap.000054.out 2>&1
    ./mhap.sh 55 > ./mhap.000055.out 2>&1
    ./mhap.sh 56 > ./mhap.000056.out 2>&1
    ./mhap.sh 57 > ./mhap.000057.out 2>&1
    ./mhap.sh 58 > ./mhap.000058.out 2>&1
    ./mhap.sh 59 > ./mhap.000059.out 2>&1
    ./mhap.sh 60 > ./mhap.000060.out 2>&1
    ./mhap.sh 61 > ./mhap.000061.out 2>&1
    ./mhap.sh 62 > ./mhap.000062.out 2>&1
    ./mhap.sh 63 > ./mhap.000063.out 2>&1
    ./mhap.sh 64 > ./mhap.000064.out 2>&1
    ./mhap.sh 65 > ./mhap.000065.out 2>&1
    ./mhap.sh 66 > ./mhap.000066.out 2>&1
    ./mhap.sh 67 > ./mhap.000067.out 2>&1
    ./mhap.sh 68 > ./mhap.000068.out 2>&1
    ./mhap.sh 69 > ./mhap.000069.out 2>&1
    ./mhap.sh 70 > ./mhap.000070.out 2>&1
    ./mhap.sh 71 > ./mhap.000071.out 2>&1
    ./mhap.sh 72 > ./mhap.000072.out 2>&1
    ./mhap.sh 73 > ./mhap.000073.out 2>&1
    ./mhap.sh 74 > ./mhap.000074.out 2>&1
    ./mhap.sh 75 > ./mhap.000075.out 2>&1
    ./mhap.sh 76 > ./mhap.000076.out 2>&1
    ./mhap.sh 77 > ./mhap.000077.out 2>&1
    ./mhap.sh 78 > ./mhap.000078.out 2>&1
    ./mhap.sh 79 > ./mhap.000079.out 2>&1
    ./mhap.sh 80 > ./mhap.000080.out 2>&1
    ./mhap.sh 81 > ./mhap.000081.out 2>&1
    ./mhap.sh 82 > ./mhap.000082.out 2>&1
    ./mhap.sh 83 > ./mhap.000083.out 2>&1
    ./mhap.sh 84 > ./mhap.000084.out 2>&1
    ./mhap.sh 85 > ./mhap.000085.out 2>&1
    ./mhap.sh 86 > ./mhap.000086.out 2>&1
    ./mhap.sh 87 > ./mhap.000087.out 2>&1
    ./mhap.sh 88 > ./mhap.000088.out 2>&1
    ./mhap.sh 89 > ./mhap.000089.out 2>&1
    ./mhap.sh 90 > ./mhap.000090.out 2>&1
    ./mhap.sh 91 > ./mhap.000091.out 2>&1
    ./mhap.sh 92 > ./mhap.000092.out 2>&1
    ./mhap.sh 93 > ./mhap.000093.out 2>&1
    ./mhap.sh 94 > ./mhap.000094.out 2>&1
    ./mhap.sh 95 > ./mhap.000095.out 2>&1
    ./mhap.sh 96 > ./mhap.000096.out 2>&1
    ./mhap.sh 97 > ./mhap.000097.out 2>&1
    ./mhap.sh 98 > ./mhap.000098.out 2>&1
    ./mhap.sh 99 > ./mhap.000099.out 2>&1
    ./mhap.sh 100 > ./mhap.000100.out 2>&1
    ./mhap.sh 101 > ./mhap.000101.out 2>&1

-- Finished on Sun Sep  3 17:57:22 2023 (10198 seconds, fashionably late) with 860200.187 GB free disk space
----------------------------------------
-- Found 101 mhap overlap output files.
-- Finished stage 'cor-mhapCheck', reset canuIteration.
----------------------------------------
-- Starting command on Sun Sep  3 17:57:22 2023 with 860200.156 GB free disk space

    cd correction
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/ovStoreConfig \
     -S ../tf.seqStore \
     -M 4-8 \
     -L ./1-overlapper/ovljob.files \
     -create ./tf.ovlStore.config \
     > ./tf.ovlStore.config.txt \
    2> ./tf.ovlStore.config.err

-- Finished on Sun Sep  3 17:57:26 2023 (4 seconds) with 860200.125 GB free disk space
----------------------------------------
--
-- Creating overlap store correction/tf.ovlStore using:
--     31 buckets
--     31 slices
--        using at most 8 GB memory each
-- Finished stage 'cor-overlapStoreConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovB' concurrent execution on Sun Sep  3 17:57:26 2023 with 860200.093 GB free disk space (31 processes; 40 concurrently)

    cd correction/tf.ovlStore.BUILDING
    ./scripts/1-bucketize.sh 1 > ./logs/1-bucketize.000001.out 2>&1
    ./scripts/1-bucketize.sh 2 > ./logs/1-bucketize.000002.out 2>&1
    ./scripts/1-bucketize.sh 3 > ./logs/1-bucketize.000003.out 2>&1
    ./scripts/1-bucketize.sh 4 > ./logs/1-bucketize.000004.out 2>&1
    ./scripts/1-bucketize.sh 5 > ./logs/1-bucketize.000005.out 2>&1
    ./scripts/1-bucketize.sh 6 > ./logs/1-bucketize.000006.out 2>&1
    ./scripts/1-bucketize.sh 7 > ./logs/1-bucketize.000007.out 2>&1
    ./scripts/1-bucketize.sh 8 > ./logs/1-bucketize.000008.out 2>&1
    ./scripts/1-bucketize.sh 9 > ./logs/1-bucketize.000009.out 2>&1
    ./scripts/1-bucketize.sh 10 > ./logs/1-bucketize.000010.out 2>&1
    ./scripts/1-bucketize.sh 11 > ./logs/1-bucketize.000011.out 2>&1
    ./scripts/1-bucketize.sh 12 > ./logs/1-bucketize.000012.out 2>&1
    ./scripts/1-bucketize.sh 13 > ./logs/1-bucketize.000013.out 2>&1
    ./scripts/1-bucketize.sh 14 > ./logs/1-bucketize.000014.out 2>&1
    ./scripts/1-bucketize.sh 15 > ./logs/1-bucketize.000015.out 2>&1
    ./scripts/1-bucketize.sh 16 > ./logs/1-bucketize.000016.out 2>&1
    ./scripts/1-bucketize.sh 17 > ./logs/1-bucketize.000017.out 2>&1
    ./scripts/1-bucketize.sh 18 > ./logs/1-bucketize.000018.out 2>&1
    ./scripts/1-bucketize.sh 19 > ./logs/1-bucketize.000019.out 2>&1
    ./scripts/1-bucketize.sh 20 > ./logs/1-bucketize.000020.out 2>&1
    ./scripts/1-bucketize.sh 21 > ./logs/1-bucketize.000021.out 2>&1
    ./scripts/1-bucketize.sh 22 > ./logs/1-bucketize.000022.out 2>&1
    ./scripts/1-bucketize.sh 23 > ./logs/1-bucketize.000023.out 2>&1
    ./scripts/1-bucketize.sh 24 > ./logs/1-bucketize.000024.out 2>&1
    ./scripts/1-bucketize.sh 25 > ./logs/1-bucketize.000025.out 2>&1
    ./scripts/1-bucketize.sh 26 > ./logs/1-bucketize.000026.out 2>&1
    ./scripts/1-bucketize.sh 27 > ./logs/1-bucketize.000027.out 2>&1
    ./scripts/1-bucketize.sh 28 > ./logs/1-bucketize.000028.out 2>&1
    ./scripts/1-bucketize.sh 29 > ./logs/1-bucketize.000029.out 2>&1
    ./scripts/1-bucketize.sh 30 > ./logs/1-bucketize.000030.out 2>&1
    ./scripts/1-bucketize.sh 31 > ./logs/1-bucketize.000031.out 2>&1

-- Finished on Sun Sep  3 17:58:55 2023 (89 seconds) with 859947.25 GB free disk space
----------------------------------------
-- Overlap store bucketizer finished.
-- Finished stage 'cor-overlapStoreBucketizerCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovS' concurrent execution on Sun Sep  3 17:58:55 2023 with 859947.218 GB free disk space (31 processes; 40 concurrently)

    cd correction/tf.ovlStore.BUILDING
    ./scripts/2-sort.sh 1 > ./logs/2-sort.000001.out 2>&1
    ./scripts/2-sort.sh 2 > ./logs/2-sort.000002.out 2>&1
    ./scripts/2-sort.sh 3 > ./logs/2-sort.000003.out 2>&1
    ./scripts/2-sort.sh 4 > ./logs/2-sort.000004.out 2>&1
    ./scripts/2-sort.sh 5 > ./logs/2-sort.000005.out 2>&1
    ./scripts/2-sort.sh 6 > ./logs/2-sort.000006.out 2>&1
    ./scripts/2-sort.sh 7 > ./logs/2-sort.000007.out 2>&1
    ./scripts/2-sort.sh 8 > ./logs/2-sort.000008.out 2>&1
    ./scripts/2-sort.sh 9 > ./logs/2-sort.000009.out 2>&1
    ./scripts/2-sort.sh 10 > ./logs/2-sort.000010.out 2>&1
    ./scripts/2-sort.sh 11 > ./logs/2-sort.000011.out 2>&1
    ./scripts/2-sort.sh 12 > ./logs/2-sort.000012.out 2>&1
    ./scripts/2-sort.sh 13 > ./logs/2-sort.000013.out 2>&1
    ./scripts/2-sort.sh 14 > ./logs/2-sort.000014.out 2>&1
    ./scripts/2-sort.sh 15 > ./logs/2-sort.000015.out 2>&1
    ./scripts/2-sort.sh 16 > ./logs/2-sort.000016.out 2>&1
    ./scripts/2-sort.sh 17 > ./logs/2-sort.000017.out 2>&1
    ./scripts/2-sort.sh 18 > ./logs/2-sort.000018.out 2>&1
    ./scripts/2-sort.sh 19 > ./logs/2-sort.000019.out 2>&1
    ./scripts/2-sort.sh 20 > ./logs/2-sort.000020.out 2>&1
    ./scripts/2-sort.sh 21 > ./logs/2-sort.000021.out 2>&1
    ./scripts/2-sort.sh 22 > ./logs/2-sort.000022.out 2>&1
    ./scripts/2-sort.sh 23 > ./logs/2-sort.000023.out 2>&1
    ./scripts/2-sort.sh 24 > ./logs/2-sort.000024.out 2>&1
    ./scripts/2-sort.sh 25 > ./logs/2-sort.000025.out 2>&1
    ./scripts/2-sort.sh 26 > ./logs/2-sort.000026.out 2>&1
    ./scripts/2-sort.sh 27 > ./logs/2-sort.000027.out 2>&1
    ./scripts/2-sort.sh 28 > ./logs/2-sort.000028.out 2>&1
    ./scripts/2-sort.sh 29 > ./logs/2-sort.000029.out 2>&1
    ./scripts/2-sort.sh 30 > ./logs/2-sort.000030.out 2>&1
    ./scripts/2-sort.sh 31 > ./logs/2-sort.000031.out 2>&1

-- Finished on Sun Sep  3 18:00:29 2023 (94 seconds) with 859955.703 GB free disk space
----------------------------------------
-- Overlap store sorter finished.
-- Finished stage 'cor-overlapStoreSorterCheck', reset canuIteration.
----------------------------------------
-- Starting command on Sun Sep  3 18:00:29 2023 with 859955.687 GB free disk space

    cd correction
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/ovStoreIndexer \
      -O  ./tf.ovlStore.BUILDING \
      -S ../tf.seqStore \
      -C  ./tf.ovlStore.config \
      -delete \
    > ./tf.ovlStore.BUILDING.index.err 2>&1

-- Finished on Sun Sep  3 18:00:31 2023 (2 seconds) with 859956.875 GB free disk space
----------------------------------------
-- Overlap store indexer finished.
-- Checking store.
----------------------------------------
-- Starting command on Sun Sep  3 18:00:31 2023 with 859956.875 GB free disk space

    cd correction
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/ovStoreDump \
     -S ../tf.seqStore \
     -O  ./tf.ovlStore \
     -counts \
     > ./tf.ovlStore/counts.dat 2> ./tf.ovlStore/counts.err

-- Finished on Sun Sep  3 18:00:31 2023 (in the blink of an eye) with 859956.859 GB free disk space
----------------------------------------
--
-- Overlap store 'correction/tf.ovlStore' successfully constructed.
-- Found 8984816058 overlaps for 1056338 reads; 354801 reads have no overlaps.
--
--
-- Purged 109.985 GB in 243 overlap output files.
-- Finished stage 'cor-createOverlapStore', reset canuIteration.
-- Set corMinCoverage=4 based on read coverage of 141.56.
-- Computing correction layouts.
--   Local  filter coverage   80
--   Global filter coverage   40
----------------------------------------
-- Starting command on Sun Sep  3 18:00:33 2023 with 860065.234 GB free disk space

    cd correction
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/generateCorrectionLayouts \
      -S ../tf.seqStore \
      -O  ./tf.ovlStore \
      -C  ./tf.corStore.WORKING \
      -eC 80 \
      -xC 40 \
    > ./tf.corStore.err 2>&1

-- Finished on Sun Sep  3 18:05:57 2023 (324 seconds) with 859913.421 GB free disk space
----------------------------------------
-- Finished stage 'cor-buildCorrectionLayoutsConfigure', reset canuIteration.
-- Computing correction layouts.
----------------------------------------
-- Starting command on Sun Sep  3 18:05:57 2023 with 859913.406 GB free disk space

    cd correction/2-correction
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/filterCorrectionLayouts \
      -S  ../../tf.seqStore \
      -C     ../tf.corStore \
      -R      ./tf.readsToCorrect.WORKING \
      -cc 4 \
      -cl 1000 \
      -g  164400000 \
      -c  40 \
    > ./tf.readsToCorrect.err 2>&1

-- Finished on Sun Sep  3 18:06:02 2023 (5 seconds) with 859913.265 GB free disk space
----------------------------------------
--                             original      original
--                            raw reads     raw reads
--   category                w/overlaps  w/o/overlaps
--   -------------------- ------------- -------------
--   Number of Reads             586101        825038
--   Number of Bases         9944907745    7449268760
--   Coverage                    60.492        45.312
--   Median                       12918          6468
--   Mean                         16967          9029
--   N50                          19918         19439
--   Minimum                       1000             0
--   Maximum                     320451        173539
--   
--                                        --------corrected---------  ----------rescued----------
--                             evidence                     expected                     expected
--   category                     reads            raw     corrected            raw     corrected
--   -------------------- -------------  ------------- -------------  ------------- -------------
--   Number of Reads             988282         278569        278569              0             0
--   Number of Bases        16233258060     5695258786    3875213746              0             0
--   Coverage                    98.742         34.643        23.572          0.000         0.000
--   Median                       12372          16341         11075              0             0
--   Mean                         16425          20444         13911              0             0
--   N50                          19576          25012         21769              0             0
--   Minimum                       1000           1000             1              0             0
--   Maximum                     320451         224529        223909              0             0
--   
--                        --------uncorrected--------
--                                           expected
--   category                       raw     corrected
--   -------------------- ------------- -------------
--   Number of Reads            1132570       1132570
--   Number of Bases        11698917719        265224
--   Coverage                    71.161         0.002
--   Median                        9247             0
--   Mean                         10329             0
--   N50                          16737             0
--   Minimum                          0             0
--   Maximum                     320451        265224
--   
--   Maximum Memory          3862323740
-- Finished stage 'cor-filterCorrectionLayouts', reset canuIteration.
--
-- Correction jobs estimated to need at most 3.597 GB for computation.
-- Correction jobs will request 12 GB each.
--
-- Local: cor       12.000 GB    4 CPUs x  10 jobs   120.000 GB  40 CPUs  (read correction)
--
--
-- Configuring correction jobs:
--   Reads estimated to need at most 3.597 GB for computation.
--   Jobs will request 12 GB each.
----------------------------------------
-- Starting command on Sun Sep  3 18:06:02 2023 with 859913.25 GB free disk space

    cd correction/2-correction
    ./correctReadsPartition.sh \
    > ./correctReadsPartition.err 2>&1

-- Finished on Sun Sep  3 18:06:03 2023 (one second) with 859913.25 GB free disk space
----------------------------------------
-- Finished stage 'cor-generateCorrectedReadsConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cor' concurrent execution on Sun Sep  3 18:06:03 2023 with 859913.218 GB free disk space (14 processes; 10 concurrently)

    cd correction/2-correction
    ./correctReads.sh 1 > ./correctReads.000001.out 2>&1
    ./correctReads.sh 2 > ./correctReads.000002.out 2>&1
    ./correctReads.sh 3 > ./correctReads.000003.out 2>&1
    ./correctReads.sh 4 > ./correctReads.000004.out 2>&1
    ./correctReads.sh 5 > ./correctReads.000005.out 2>&1
    ./correctReads.sh 6 > ./correctReads.000006.out 2>&1
    ./correctReads.sh 7 > ./correctReads.000007.out 2>&1
    ./correctReads.sh 8 > ./correctReads.000008.out 2>&1
    ./correctReads.sh 9 > ./correctReads.000009.out 2>&1
    ./correctReads.sh 10 > ./correctReads.000010.out 2>&1
    ./correctReads.sh 11 > ./correctReads.000011.out 2>&1
    ./correctReads.sh 12 > ./correctReads.000012.out 2>&1
    ./correctReads.sh 13 > ./correctReads.000013.out 2>&1
    ./correctReads.sh 14 > ./correctReads.000014.out 2>&1

-- Finished on Sun Sep  3 19:59:25 2023 (6802 seconds) with 859022.562 GB free disk space
----------------------------------------
-- Found 14 read correction output files.
-- Finished stage 'cor-generateCorrectedReadsCheck', reset canuIteration.
-- Found 14 read correction output files.
-- Finished stage 'cor-generateCorrectedReadsCheck', reset canuIteration.
--
-- Loading corrected reads into corStore and seqStore.
----------------------------------------
-- Starting command on Sun Sep  3 19:59:25 2023 with 859022.531 GB free disk space

    cd correction
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/loadCorrectedReads \
      -S ../tf.seqStore \
      -C ./tf.corStore \
      -L ./2-correction/corjob.files \
    >  ./tf.loadCorrectedReads.log \
    2> ./tf.loadCorrectedReads.err

-- Finished on Sun Sep  3 19:59:30 2023 (5 seconds) with 859022.187 GB free disk space
----------------------------------------
--
-- In sequence store './tf.seqStore':
--   Found 10366 reads.
--   Found 90002428 bases (0.54 times coverage).
--    Histogram of corrected reads:
--    
--    G=90002428                         sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010        57906       132      9029984  ||       1000-3860         5156|---------------------------------------------------------------
--    00020        47381       303     18027143  ||       3861-6721         1939|------------------------
--    00030        38132       514     27005835  ||       6722-9582          993|-------------
--    00040        30212       780     36019258  ||       9583-12443         520|-------
--    00050        21854      1130     45005384  ||      12444-15304         238|---
--    00060        13609      1654     54004605  ||      15305-18165         222|---
--    00070         8755      2500     63002627  ||      18166-21026         129|--
--    00080         5769      3765     72004349  ||      21027-23887         143|--
--    00090         3312      5809     81004665  ||      23888-26748         115|--
--    00100         1000     10365     90002428  ||      26749-29609         104|--
--    001.000x               10366     90002428  ||      29610-32470         114|--
--                                               ||      32471-35331          92|--
--                                               ||      35332-38192          89|--
--                                               ||      38193-41053          70|-
--                                               ||      41054-43914          62|-
--                                               ||      43915-46775          66|-
--                                               ||      46776-49636          40|-
--                                               ||      49637-52497          61|-
--                                               ||      52498-55358          42|-
--                                               ||      55359-58219          40|-
--                                               ||      58220-61080          28|-
--                                               ||      61081-63941          25|-
--                                               ||      63942-66802          29|-
--                                               ||      66803-69663          20|-
--                                               ||      69664-72524          16|-
--                                               ||      72525-75385           5|-
--                                               ||      75386-78246           2|-
--                                               ||      78247-81107           0|
--                                               ||      81108-83968           1|-
--                                               ||      83969-86829           0|
--                                               ||      86830-89690           0|
--                                               ||      89691-92551           0|
--                                               ||      92552-95412           0|
--                                               ||      95413-98273           0|
--                                               ||      98274-101134          0|
--                                               ||     101135-103995          0|
--                                               ||     103996-106856          0|
--                                               ||     106857-109717          0|
--                                               ||     109718-112578          0|
--                                               ||     112579-115439          0|
--                                               ||     115440-118300          0|
--                                               ||     118301-121161          0|
--                                               ||     121162-124022          0|
--                                               ||     124023-126883          0|
--                                               ||     126884-129744          0|
--                                               ||     129745-132605          2|-
--                                               ||     132606-135466          0|
--                                               ||     135467-138327          1|-
--                                               ||     138328-141188          0|
--                                               ||     141189-144049          2|-
--    
--
-- Purging correctReads output after loading into stores.
-- Purged 14 .cns outputs.
-- Purged 28 .out job log outputs.
--
-- No corrected reads generated, overlaps used for correction saved.
-- Finished stage 'cor-loadCorrectedReads', reset canuIteration.
----------------------------------------
-- Starting command on Sun Sep  3 19:59:30 2023 with 859022.125 GB free disk space

    cd .
    /dssg/home/acct-jiang.lu/jiang.lu/biotools/canu-2.2/bin/sqStoreDumpFASTQ \
      -corrected \
      -S ./tf.seqStore \
      -o ./tf.correctedReads.gz \
      -fasta \
      -nolibname \
    > tf.correctedReads.fasta.err 2>&1

-- Finished on Sun Sep  3 19:59:32 2023 (2 seconds) with 859022.078 GB free disk space
----------------------------------------
--
-- Corrected reads saved in 'tf.correctedReads.fasta.gz'.
-- Finished stage 'cor-dumpCorrectedReads', reset canuIteration.
--
-- ERROR:  Read coverage (0.54) lower than allowed.
-- ERROR:    stopOnLowCoverage = 10
-- ERROR:
-- ERROR:  This could be caused by an incorrect genomeSize or poor
-- ERROR:  quality reads that cound not be sufficiently corrected.
-- ERROR:
-- ERROR:  You can force Canu to continue by decreasing parameter
-- ERROR:  stopOnLowCoverage (and possibly minInputCoverage too).
-- ERROR:  Be warned that the quality of corrected reads and/or
-- ERROR:  contiguity of contigs will be poor.
--

ABORT:
ABORT: canu 2.2
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.
ABORT:
skoren commented 1 year ago

It seems these reads may be lower quality than expected. You could try corMhapSensitivity=high corMinCoverage=0 and see if that helps. The reads are overall pretty short, mostly under 15kb which isn't ideal for a good assembly. How old are these reads? Have you tried converting them to HiFi data to see how much coverage you end up with after that? This would make the 10kb reads more useful for assembly if they were high accuracy.

yaoxkkkkk commented 1 year ago

The data was uploaded to NCBI on 2023-01-30 (SRR23272336) and the preprint was published to Biorxiv on 2023-06-22. According to their preprint:

Moreover, a total of 2.2 million reads larger than 500 bp were obtained by PacBio Sequel sequencing, with a coverage depth of approximately 79 X (Fig. S3). 149,029 reads (about 52% of the total) were larger than 5 kb in length, of which 81.57% had an average base length of 10 kb.

Any ideas to deal with this quality data? Their assembly pipeline is Falcon + Pilon +LACHESIS. But I think it is too old since LACHESIS doesn't develop anymore.

Thanks for your time!

skoren commented 1 year ago

I am not sure that this was properly submitted to NCBI. The original file name here: https://trace.ncbi.nlm.nih.gov/Traces/index.html?view=run_browser&page_size=10&acc=SRR23272336&display=data-access is listed as: s3://sra-pub-src-18/SRR23272336/m54180_171129_064641.scraps.bam.1. Scraps.bam is the failed reads from the run, the reads are normally named subreads.bam (https://pacbio.gs.washington.edu/documents/Raw_Data_files.pdf). The read statistics also don't match what you have (coverage and number of reads are both different). I'd suggest contacting the authors and confirming the NCBI submission is accurate.

yaoxkkkkk commented 1 year ago

Oh my god! I can't believe that the raw data is wrong... Thank you for helping me get out of the delimma.

yaoxkkkkk commented 1 year ago

I find another NCBI Pacbio SRA, and here is the quality graph: image

And the quality graph of SRR23272336 is like: image

So it's indeed abnormal...