search

marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
659 stars 179 forks source link

No overlaps found with mhap #2052

Closed jonhultqvist closed 2 years ago

jonhultqvist commented 2 years ago

Hi,

I'm having problems with canu not finding any overlaps during the mhap step. The dataset is nanopore reads basecalled using Guppy and the SUP model, adaptor-trimmed using porechop and filtered to 100x coverage using Filtlong. The data has assembled well using Flye 2.9. I have looked at the some of the logs but it's not obvious to me what is the issue.

I'm running canu 2.2, Linux. Ubuntu 21.04. Local computer.

Thanks -Jon

Command:

 /home/jon/software/canu-2.2/bin/canu -p WB_SUP -d WB_SUP_canu genomeSize=12m maxInputCoverage=100 -nanopore /home/jon/scratch/MinION/Giardia/TRIMMED_READS/WB_RBK_210804_SUP.chop.filt.fq.gz

Output

-- canu 2.2
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
--   Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
--   Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
--   Genome Res. 2017 May;27(5):722-736.
--   http://doi.org/10.1101/gr.215087.116
-- 
-- Read and contig alignments during correction and consensus use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '11.0.12' (from 'java') without -d64 support.
--
-- WARNING:
-- WARNING:  Failed to run gnuplot using command 'gnuplot'.
-- WARNING:  Plots will be disabled.
-- WARNING:
--
--
-- Detected 16 CPUs and 62 gigabytes of memory on the local machine.
--
-- Local machine mode enabled; grid support not detected or not allowed.
--
--                                (tag)Concurrency
--                         (tag)Threads          |
--                (tag)Memory         |          |
--        (tag)             |         |          |       total usage      algorithm
--        -------  ----------  --------   --------  --------------------  -----------------------------
-- Local: meryl     12.000 GB    4 CPUs x   4 jobs    48.000 GB  16 CPUs  (k-mer counting)
-- Local: hap        8.000 GB    4 CPUs x   4 jobs    32.000 GB  16 CPUs  (read-to-haplotype assignment)
-- Local: cormhap    6.000 GB   16 CPUs x   1 job      6.000 GB  16 CPUs  (overlap detection with mhap)
-- Local: obtovl     4.000 GB    8 CPUs x   2 jobs     8.000 GB  16 CPUs  (overlap detection)
-- Local: utgovl     4.000 GB    8 CPUs x   2 jobs     8.000 GB  16 CPUs  (overlap detection)
-- Local: cor        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (read correction)
-- Local: ovb        4.000 GB    1 CPU  x  15 jobs    60.000 GB  15 CPUs  (overlap store bucketizer)
-- Local: ovs        8.000 GB    1 CPU  x   7 jobs    56.000 GB   7 CPUs  (overlap store sorting)
-- Local: red       15.000 GB    4 CPUs x   4 jobs    60.000 GB  16 CPUs  (read error detection)
-- Local: oea        8.000 GB    1 CPU  x   7 jobs    56.000 GB   7 CPUs  (overlap error adjustment)
-- Local: bat       16.000 GB    4 CPUs x   1 job     16.000 GB   4 CPUs  (contig construction with bogart)
-- Local: cns        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (consensus)
--
-- Found untrimmed raw Nanopore reads in the input files.
--
-- Generating assembly 'WB_SUP' in '/media/jon/BackupPlus/MinION_data/ASSEMBLY/canu/WB_SUP_canu':
--   genomeSize:
--     12000000
--
--   Overlap Generation Limits:
--     corOvlErrorRate 0.3200 ( 32.00%)
--     obtOvlErrorRate 0.1200 ( 12.00%)
--     utgOvlErrorRate 0.1200 ( 12.00%)
--
--   Overlap Processing Limits:
--     corErrorRate    0.3000 ( 30.00%)
--     obtErrorRate    0.1200 ( 12.00%)
--     utgErrorRate    0.1200 ( 12.00%)
--     cnsErrorRate    0.2000 ( 20.00%)
--
--   Stages to run:
--     correct raw reads.
--     trim corrected reads.
--     assemble corrected and trimmed reads.
--
--
-- BEGIN CORRECTION
----------------------------------------
-- Starting command on Fri Nov 12 16:47:47 2021 with 2053.637 GB free disk space

    cd .
    ./WB_SUP.seqStore.sh \
    > ./WB_SUP.seqStore.err 2>&1

-- Finished on Fri Nov 12 16:48:00 2021 (13 seconds) with 2053.307 GB free disk space
----------------------------------------
--
-- In sequence store './WB_SUP.seqStore':
--   Found 119829 reads.
--   Found 1200003459 bases (100 times coverage).
--    Histogram of raw reads:
--    
--    G=1200003459                       sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010        16599      6019    120006185  ||       6509-7479        25078|---------------------------------------------------------------
--    00020        13824     14009    240004191  ||       7480-8450        23815|------------------------------------------------------------
--    00030        12119     23317    360008458  ||       8451-9421        18105|----------------------------------------------
--    00040        10886     33785    480012264  ||       9422-10392       13684|-----------------------------------
--    00050         9917     45352    600009243  ||      10393-11363        9851|-------------------------
--    00060         9116     57980    720006607  ||      11364-12334        7459|-------------------
--    00070         8414     71692    840006018  ||      12335-13305        5562|--------------
--    00080         7780     86530    960006224  ||      13306-14276        4080|-----------
--    00090         7222    102548   1080010087  ||      14277-15247        3064|--------
--    00100         6509    119828   1200003459  ||      15248-16218        2380|------
--    001.000x              119829   1200003459  ||      16219-17189        1776|-----
--                                               ||      17190-18160        1278|----
--                                               ||      18161-19131         924|---
--                                               ||      19132-20102         670|--
--                                               ||      20103-21073         524|--
--                                               ||      21074-22044         383|-
--                                               ||      22045-23015         316|-
--                                               ||      23016-23986         214|-
--                                               ||      23987-24957         161|-
--                                               ||      24958-25928         127|-
--                                               ||      25929-26899          91|-
--                                               ||      26900-27870          78|-
--                                               ||      27871-28841          41|-
--                                               ||      28842-29812          41|-
--                                               ||      29813-30783          34|-
--                                               ||      30784-31754          18|-
--                                               ||      31755-32725          24|-
--                                               ||      32726-33696          12|-
--                                               ||      33697-34667           9|-
--                                               ||      34668-35638          10|-
--                                               ||      35639-36609           2|-
--                                               ||      36610-37580           6|-
--                                               ||      37581-38551           0|
--                                               ||      38552-39522           1|-
--                                               ||      39523-40493           3|-
--                                               ||      40494-41464           0|
--                                               ||      41465-42435           3|-
--                                               ||      42436-43406           0|
--                                               ||      43407-44377           0|
--                                               ||      44378-45348           1|-
--                                               ||      45349-46319           0|
--                                               ||      46320-47290           0|
--                                               ||      47291-48261           1|-
--                                               ||      48262-49232           0|
--                                               ||      49233-50203           0|
--                                               ||      50204-51174           1|-
--                                               ||      51175-52145           0|
--                                               ||      52146-53116           1|-
--                                               ||      53117-54087           0|
--                                               ||      54088-55058           1|-
--    
----------------------------------------
-- Starting command on Fri Nov 12 16:48:00 2021 with 2053.304 GB free disk space

    cd correction/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Fri Nov 12 16:48:00 2021 (lickety-split) with 2053.302 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--        01  2.45 GB       2
--        02  1.54 GB       2
--        04  0.77 GB       2
--        06  0.52 GB       2
--        08  0.39 GB       2
--        12  0.26 GB       2
--
--  For 119829 reads with 1200003459 bases, limit to 12 batches.
--  Will count kmers using 01 jobs, each using 4 GB and 4 threads.
--
-- Finished stage 'merylConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Fri Nov 12 16:48:00 2021 with 2053.302 GB free disk space (1 processes; 4 concurrently)

    cd correction/0-mercounts
    ./meryl-count.sh 1 > ./meryl-count.000001.out 2>&1

-- Finished on Fri Nov 12 16:49:18 2021 (78 seconds) with 2052.176 GB free disk space
----------------------------------------
-- Found 1 Kmer counting (meryl) outputs.
-- Finished stage 'cor-merylCountCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Fri Nov 12 16:49:18 2021 with 2052.176 GB free disk space (1 processes; 4 concurrently)

    cd correction/0-mercounts
    ./meryl-process.sh 1 > ./meryl-process.000001.out 2>&1

-- Finished on Fri Nov 12 16:49:22 2021 (4 seconds) with 2053.002 GB free disk space
----------------------------------------
-- Meryl finished successfully.  Kmer frequency histogram:
--
-- WARNING: gnuplot failed.
--
----------------------------------------
--
--  16-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2  30618151 ********************************************************************** 0.5176 0.0607
--       3-     5  14629875 *********************************                                      0.6694 0.0874
--       6-    10   2823746 ******                                                                 0.7835 0.1182
--      11-    17    698332 *                                                                      0.8157 0.1340
--      18-    26    219074                                                                        0.8252 0.1418
--      27-    37    129810                                                                        0.8284 0.1460
--      38-    50    461993 *                                                                      0.8306 0.1504
--      51-    65   2396119 *****                                                                  0.8395 0.1745
--      66-    82   4697091 **********                                                             0.8832 0.3296
--      83-   101   1690516 ***                                                                    0.9614 0.6712
--     102-   122    181367                                                                        0.9870 0.8056
--     123-   145    150944                                                                        0.9898 0.8240
--     146-   170    163395                                                                        0.9924 0.8446
--     171-   197     79326                                                                        0.9951 0.8697
--     198-   226     42001                                                                        0.9964 0.8835
--     227-   257     34262                                                                        0.9971 0.8923
--     258-   290     23468                                                                        0.9977 0.9004
--     291-   325     20345                                                                        0.9981 0.9067
--     326-   362     14938                                                                        0.9984 0.9129
--     363-   401     10268                                                                        0.9987 0.9179
--     402-   442      8177                                                                        0.9989 0.9218
--     443-   485      6279                                                                        0.9990 0.9252
--     486-   530      5630                                                                        0.9991 0.9280
--     531-   577      4776                                                                        0.9992 0.9309
--     578-   626      4506                                                                        0.9993 0.9335
--     627-   677      4262                                                                        0.9994 0.9362
--     678-   730      3877                                                                        0.9994 0.9389
--     731-   785      2623                                                                        0.9995 0.9416
--     786-   842      2198                                                                        0.9995 0.9436
--     843-   901      1975                                                                        0.9996 0.9453
--     902-   962      1952                                                                        0.9996 0.9470
--     963-  1025      1810                                                                        0.9996 0.9488
--    1026-  1090      1671                                                                        0.9997 0.9506
--    1091-  1157      1534                                                                        0.9997 0.9524
--    1158-  1226      1400                                                                        0.9997 0.9541
--    1227-  1297      1256                                                                        0.9997 0.9557
--    1298-  1370      1048                                                                        0.9998 0.9573
--    1371-  1445       972                                                                        0.9998 0.9587
--    1446-  1522       709                                                                        0.9998 0.9600
--    1523-  1601       520                                                                        0.9998 0.9611
--
--           0 (max occurrences)
--  1008990965 (total mers, non-unique)
--    59152920 (distinct mers, non-unique)
--           0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'correction/0-mercounts/WB_SUP.ms16'.
--
-- OVERLAPPER (mhap) (correction)
--
-- Set corMhapSensitivity=low based on read coverage of 100.
--
-- PARAMETERS: hashes=256, minMatches=3, threshold=0.8
--
-- Given 5.4 GB, can fit 16200 reads per block.
-- For 9 blocks, set stride to 2 blocks.
-- Logging partitioning to 'correction/1-overlapper/partitioning.log'.
-- Configured 8 mhap precompute jobs.
-- Configured 17 mhap overlap jobs.
-- Finished stage 'cor-mhapConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Fri Nov 12 16:49:22 2021 with 2053.295 GB free disk space (8 processes; 1 concurrently)

    cd correction/1-overlapper
    ./precompute.sh 1 > ./precompute.000001.out 2>&1
    ./precompute.sh 2 > ./precompute.000002.out 2>&1
    ./precompute.sh 3 > ./precompute.000003.out 2>&1
    ./precompute.sh 4 > ./precompute.000004.out 2>&1
    ./precompute.sh 5 > ./precompute.000005.out 2>&1
    ./precompute.sh 6 > ./precompute.000006.out 2>&1
    ./precompute.sh 7 > ./precompute.000007.out 2>&1
    ./precompute.sh 8 > ./precompute.000008.out 2>&1

-- Finished on Fri Nov 12 17:05:32 2021 (970 seconds) with 2051.268 GB free disk space
----------------------------------------
-- All 8 mhap precompute jobs finished successfully.
-- Finished stage 'cor-mhapPrecomputeCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Fri Nov 12 17:05:32 2021 with 2051.268 GB free disk space (17 processes; 1 concurrently)

    cd correction/1-overlapper
    ./mhap.sh 1 > ./mhap.000001.out 2>&1
    ./mhap.sh 2 > ./mhap.000002.out 2>&1
    ./mhap.sh 3 > ./mhap.000003.out 2>&1
    ./mhap.sh 4 > ./mhap.000004.out 2>&1
    ./mhap.sh 5 > ./mhap.000005.out 2>&1
    ./mhap.sh 6 > ./mhap.000006.out 2>&1
    ./mhap.sh 7 > ./mhap.000007.out 2>&1
    ./mhap.sh 8 > ./mhap.000008.out 2>&1
    ./mhap.sh 9 > ./mhap.000009.out 2>&1
    ./mhap.sh 10 > ./mhap.000010.out 2>&1
    ./mhap.sh 11 > ./mhap.000011.out 2>&1
    ./mhap.sh 12 > ./mhap.000012.out 2>&1
    ./mhap.sh 13 > ./mhap.000013.out 2>&1
    ./mhap.sh 14 > ./mhap.000014.out 2>&1
    ./mhap.sh 15 > ./mhap.000015.out 2>&1
    ./mhap.sh 16 > ./mhap.000016.out 2>&1
    ./mhap.sh 17 > ./mhap.000017.out 2>&1

-- Finished on Fri Nov 12 17:06:41 2021 (69 seconds) with 2051.255 GB free disk space
----------------------------------------
-- Found 17 mhap overlap output files.
-- Finished stage 'cor-mhapCheck', reset canuIteration.
----------------------------------------
-- Starting command on Fri Nov 12 17:06:41 2021 with 2051.255 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/ovStoreConfig \
     -S ../WB_SUP.seqStore \
     -M 4-8 \
     -L ./1-overlapper/ovljob.files \
     -create ./WB_SUP.ovlStore.config \
     > ./WB_SUP.ovlStore.config.txt \
    2> ./WB_SUP.ovlStore.config.err

-- Finished on Fri Nov 12 17:06:41 2021 (furiously fast) with 2051.254 GB free disk space
----------------------------------------
--
-- Creating overlap store correction/WB_SUP.ovlStore using:
--      1 bucket
--      1 slice
--        using at most 1 GB memory each
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovS' concurrent execution on Fri Nov 12 17:06:41 2021 with 2051.254 GB free disk space (1 processes; 7 concurrently)

    cd correction
    ./WB_SUP.ovlStore.sh 1 > ./WB_SUP.ovlStore.000001.out 2>&1

-- Finished on Fri Nov 12 17:06:41 2021 (like a bat out of hell) with 2051.25 GB free disk space
----------------------------------------
-- Checking store.
----------------------------------------
-- Starting command on Fri Nov 12 17:06:41 2021 with 2051.25 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/ovStoreDump \
     -S ../WB_SUP.seqStore \
     -O  ./WB_SUP.ovlStore \
     -counts \
     > ./WB_SUP.ovlStore/counts.dat 2> ./WB_SUP.ovlStore/counts.err

-- Finished on Fri Nov 12 17:06:41 2021 (in the blink of an eye) with 2051.249 GB free disk space
----------------------------------------
--
-- Overlap store 'correction/WB_SUP.ovlStore' successfully constructed.
-- Found 0 overlaps for 0 reads; 119829 reads have no overlaps.
--
--
-- Purged 2.031 GB in 42 overlap output files.
-- Finished stage 'cor-createOverlapStore', reset canuIteration.
-- Set corMinCoverage=4 based on read coverage of 100.
-- Computing correction layouts.
--   Local  filter coverage   80
--   Global filter coverage   40
----------------------------------------
-- Starting command on Fri Nov 12 17:06:42 2021 with 2053.288 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/generateCorrectionLayouts \
      -S ../WB_SUP.seqStore \
      -O  ./WB_SUP.ovlStore \
      -C  ./WB_SUP.corStore.WORKING \
      -eC 80 \
      -xC 40 \
    > ./WB_SUP.corStore.err 2>&1

-- Finished on Fri Nov 12 17:06:42 2021 (fast as lightning) with 2053.287 GB free disk space
----------------------------------------
-- Finished stage 'cor-buildCorrectionLayoutsConfigure', reset canuIteration.
-- Computing correction layouts.
----------------------------------------
-- Starting command on Fri Nov 12 17:06:42 2021 with 2053.287 GB free disk space

    cd correction/2-correction
    /home/jon/software/canu-2.2/bin/filterCorrectionLayouts \
      -S  ../../WB_SUP.seqStore \
      -C     ../WB_SUP.corStore \
      -R      ./WB_SUP.readsToCorrect.WORKING \
      -cc 4 \
      -cl 1000 \
      -g  12000000 \
      -c  40 \
    > ./WB_SUP.readsToCorrect.err 2>&1

-- Finished on Fri Nov 12 17:06:42 2021 (fast as lightning) with 2053.278 GB free disk space
----------------------------------------
--                             original      original
--                            raw reads     raw reads
--   category                w/overlaps  w/o/overlaps
--   -------------------- ------------- -------------
--   Number of Reads                  0        119829
--   Number of Bases                  0             0
--   Coverage                     0.000         0.000
--   Median                           0             0
--   Mean                             0             0
--   N50                              0             0
--   Minimum                          0             0
--   Maximum                          0             0
--   
--                                        --------corrected---------  ----------rescued----------
--                             evidence                     expected                     expected
--   category                     reads            raw     corrected            raw     corrected
--   -------------------- -------------  ------------- -------------  ------------- -------------
--   Number of Reads                  0              0             0              0             0
--   Number of Bases                  0              0             0              0             0
--   Coverage                     0.000          0.000         0.000          0.000         0.000
--   Median                           0              0             0              0             0
--   Mean                             0              0             0              0             0
--   N50                              0              0             0              0             0
--   Minimum                          0              0             0              0             0
--   Maximum                          0              0             0              0             0
--   
--                        --------uncorrected--------
--                                           expected
--   category                       raw     corrected
--   -------------------- ------------- -------------
--   Number of Reads             119829        119829
--   Number of Bases                  0             0
--   Coverage                     0.000         0.000
--   Median                           0             0
--   Mean                             0             0
--   N50                              0             0
--   Minimum                          0             0
--   Maximum                          0             0
--   
--   Maximum Memory                   0
-- Finished stage 'cor-filterCorrectionLayouts', reset canuIteration.
--
-- Correction jobs estimated to need at most 0 GB for computation.
-- Correction jobs will request 6 GB each.
--
-- Local: cor        6.000 GB    4 CPUs x   4 jobs    24.000 GB  16 CPUs  (read correction)
--
--
-- Configuring correction jobs:
--   Reads estimated to need at most 0 GB for computation.
--   Jobs will request 6 GB each.
----------------------------------------
-- Starting command on Fri Nov 12 17:06:42 2021 with 2053.278 GB free disk space

    cd correction/2-correction
    ./correctReadsPartition.sh \
    > ./correctReadsPartition.err 2>&1

-- Finished on Fri Nov 12 17:06:42 2021 (like a bat out of hell) with 2053.277 GB free disk space
----------------------------------------
-- Finished stage 'cor-generateCorrectedReadsConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cor' concurrent execution on Fri Nov 12 17:06:42 2021 with 2053.277 GB free disk space (1 processes; 4 concurrently)

    cd correction/2-correction
    ./correctReads.sh 1 > ./correctReads.000001.out 2>&1

-- Finished on Fri Nov 12 17:06:42 2021 (lickety-split) with 2053.277 GB free disk space
----------------------------------------
-- Found 1 read correction output files.
-- Finished stage 'cor-generateCorrectedReadsCheck', reset canuIteration.
-- Found 1 read correction output files.
-- Finished stage 'cor-generateCorrectedReadsCheck', reset canuIteration.
--
-- Loading corrected reads into corStore and seqStore.
----------------------------------------
-- Starting command on Fri Nov 12 17:06:42 2021 with 2053.277 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/loadCorrectedReads \
      -S ../WB_SUP.seqStore \
      -C ./WB_SUP.corStore \
      -L ./2-correction/corjob.files \
    >  ./WB_SUP.loadCorrectedReads.log \
    2> ./WB_SUP.loadCorrectedReads.err

-- Finished on Fri Nov 12 17:06:42 2021 (like a bat out of hell) with 2053.267 GB free disk space
----------------------------------------
--
-- No corrected reads generated; correctReads output saved.
--
-- Purging overlaps used for correction.
-- Finished stage 'cor-loadCorrectedReads', reset canuIteration.
----------------------------------------
-- Starting command on Fri Nov 12 17:06:42 2021 with 2053.271 GB free disk space

    cd .
    /home/jon/software/canu-2.2/bin/sqStoreDumpFASTQ \
      -corrected \
      -S ./WB_SUP.seqStore \
      -o ./WB_SUP.correctedReads.gz \
      -fasta \
      -nolibname \
    > WB_SUP.correctedReads.fasta.err 2>&1

-- Finished on Fri Nov 12 17:06:42 2021 (lickety-split) with 2053.27 GB free disk space
----------------------------------------
--
-- Corrected reads saved in 'WB_SUP.correctedReads.fasta.gz'.
-- Finished stage 'cor-dumpCorrectedReads', reset canuIteration.
--
-- Trimming skipped; no corrected reads exist in WB_SUP.seqStore.
--
-- Unitigging skipped; no corrected reads to assemble.
--
-- Bye.
skoren commented 2 years ago

Interesting, I don't see anything that'd indicate a problem in the logs. Try adding the options mhapPipe=false purgeOverlaps=false saveOverlaps=true and run from scratch. Post the tail (last 20 or so lines) of a few of the mhap.*.out files as well.

Based on the k-mer histogram, it looks like you've got a pretty clear peak in the reads so you might not need correction. In that case you can run direct assembly (see the quick start: https://canu.readthedocs.io/en/latest/quick-start.html#assembling-with-multiple-technologies-and-multiple-files).

jonhultqvist commented 2 years ago

Thank you, I will re-run with your suggested changes. The assembly has moved past were it previously failed and into the 'cor' step with falconsense running.

Meanwhile, here are the mhap out-files. Something seems to go wrong from the with some of the batches. Batches 2, 3, 4, 6, 7, 9, 10, 12 and 14 (out of 17) show NaN as shown for 2 and 3 below.

mhap.000001.out (last 20 lines)

Current # sequences stored: 5000...
Current # sequences stored: 10000...
Current # sequences stored: 15000...
Current # sequences stored: 20000...
Current # sequences stored: 25000...
Current # sequences stored: 30000...
Stored 32400 sequences in the index.
Processed 32400 unique sequences (fwd and rev).
Time (s) to read and hash from file: 1.097224137
Time (s) to score and output to self: 5.500190358
Total scoring time (s): 5.5008108920000005
Total time (s): 6.598120403
MinHash search time (s): 4.304200432
Total matches found: 600746
Average number of matches per lookup: 37.08308641975309
Average number of table elements processed per lookup: 1983.488950617284
Average number of table elements processed per match: 53.48769862803914
Average % of hashed sequences hit per lookup: 0.5746141975308642
Average % of hashed sequences hit that are matches: 19.918402803680994
Average % of hashed sequences fully compared that are matches: 97.47100998972957

mhap.000002.out (last 20 lines)

Current # sequences stored: 5000...
Current # sequences stored: 10000...
Current # sequences stored: 15000...
Current # sequences stored: 20000...
Current # sequences stored: 25000...
Current # sequences stored: 30000...
Stored 32400 sequences in the index.
Processed 32400 unique sequences (fwd and rev).
Time (s) to read and hash from file: 1.062150128
Total scoring time (s): 5.45419E-4
Total time (s): 1.062840979
MinHash search time (s): 0.0
Total matches found: 0
Average number of matches per lookup: NaN
Average number of table elements processed per lookup: NaN
Average number of table elements processed per match: NaN
Average % of hashed sequences hit per lookup: NaN
Average % of hashed sequences hit that are matches: NaN
Average % of hashed sequences fully compared that are matches: NaN

mhap.000003.out (last 20 lines)

Current # sequences stored: 5000...
Current # sequences stored: 10000...
Current # sequences stored: 15000...
Current # sequences stored: 20000...
Current # sequences stored: 25000...
Current # sequences stored: 30000...
Stored 32400 sequences in the index.
Processed 32400 unique sequences (fwd and rev).
Time (s) to read and hash from file: 1.232457538
Total scoring time (s): 4.9817E-4
Total time (s): 1.2330863330000001
MinHash search time (s): 0.0
Total matches found: 0
Average number of matches per lookup: NaN
Average number of table elements processed per lookup: NaN
Average number of table elements processed per match: NaN
Average % of hashed sequences hit per lookup: NaN
Average % of hashed sequences hit that are matches: NaN
Average % of hashed sequences fully compared that are matches: NaN
skoren commented 2 years ago

Are you able to share your data (see FAQ for info on sending it to us)?

Did the assembly that moved past this step actually find overlaps and have an updated report? If it didn't find overlaps the correct reads wouldn't be very useful since there is no info to correct them properly.

jonhultqvist commented 2 years ago

Hi, I've tried sharing the data as per the FAQ. The file is named "WB_RBK_210804_SUP.chop.filt.fq.gz"

It seems like some overlaps might have been calculated, but I also see a lot of zeros in the statistics. Not sure what to make of it.

-- canu 2.2
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
--   Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
--   Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
--   Genome Res. 2017 May;27(5):722-736.
--   http://doi.org/10.1101/gr.215087.116
-- 
-- Read and contig alignments during correction and consensus use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '11.0.12' (from 'java') without -d64 support.
--
-- WARNING:
-- WARNING:  Failed to run gnuplot using command 'gnuplot'.
-- WARNING:  Plots will be disabled.
-- WARNING:
--
--
-- Detected 16 CPUs and 62 gigabytes of memory on the local machine.
--
-- Local machine mode enabled; grid support not detected or not allowed.
--
--                                (tag)Concurrency
--                         (tag)Threads          |
--                (tag)Memory         |          |
--        (tag)             |         |          |       total usage      algorithm
--        -------  ----------  --------   --------  --------------------  -----------------------------
-- Local: meryl     12.000 GB    4 CPUs x   4 jobs    48.000 GB  16 CPUs  (k-mer counting)
-- Local: hap        8.000 GB    4 CPUs x   4 jobs    32.000 GB  16 CPUs  (read-to-haplotype assignment)
-- Local: cormhap    6.000 GB   16 CPUs x   1 job      6.000 GB  16 CPUs  (overlap detection with mhap)
-- Local: obtovl     4.000 GB    8 CPUs x   2 jobs     8.000 GB  16 CPUs  (overlap detection)
-- Local: utgovl     4.000 GB    8 CPUs x   2 jobs     8.000 GB  16 CPUs  (overlap detection)
-- Local: cor        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (read correction)
-- Local: ovb        4.000 GB    1 CPU  x  15 jobs    60.000 GB  15 CPUs  (overlap store bucketizer)
-- Local: ovs        8.000 GB    1 CPU  x   7 jobs    56.000 GB   7 CPUs  (overlap store sorting)
-- Local: red       15.000 GB    4 CPUs x   4 jobs    60.000 GB  16 CPUs  (read error detection)
-- Local: oea        8.000 GB    1 CPU  x   7 jobs    56.000 GB   7 CPUs  (overlap error adjustment)
-- Local: bat       16.000 GB    4 CPUs x   1 job     16.000 GB   4 CPUs  (contig construction with bogart)
-- Local: cns        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (consensus)
--
-- Found untrimmed raw Nanopore reads in the input files.
--
-- Generating assembly 'WB_SUP' in '/media/jon/BackupPlus/MinION_data/ASSEMBLY/canu/WB_SUP_canu_no_mhap':
--   genomeSize:
--     12000000
--
--   Overlap Generation Limits:
--     corOvlErrorRate 0.3200 ( 32.00%)
--     obtOvlErrorRate 0.1200 ( 12.00%)
--     utgOvlErrorRate 0.1200 ( 12.00%)
--
--   Overlap Processing Limits:
--     corErrorRate    0.3000 ( 30.00%)
--     obtErrorRate    0.1200 ( 12.00%)
--     utgErrorRate    0.1200 ( 12.00%)
--     cnsErrorRate    0.2000 ( 20.00%)
--
--   Stages to run:
--     correct raw reads.
--     trim corrected reads.
--     assemble corrected and trimmed reads.
--
--
-- BEGIN CORRECTION
----------------------------------------
-- Starting command on Tue Nov 16 16:10:37 2021 with 2051.814 GB free disk space

    cd .
    ./WB_SUP.seqStore.sh \
    > ./WB_SUP.seqStore.err 2>&1

-- Finished on Tue Nov 16 16:10:51 2021 (14 seconds) with 2051.484 GB free disk space
----------------------------------------
--
-- In sequence store './WB_SUP.seqStore':
--   Found 119829 reads.
--   Found 1200003459 bases (100 times coverage).
--    Histogram of raw reads:
--    
--    G=1200003459                       sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010        16599      6019    120006185  ||       6509-7479        25078|---------------------------------------------------------------
--    00020        13824     14009    240004191  ||       7480-8450        23815|------------------------------------------------------------
--    00030        12119     23317    360008458  ||       8451-9421        18105|----------------------------------------------
--    00040        10886     33785    480012264  ||       9422-10392       13684|-----------------------------------
--    00050         9917     45352    600009243  ||      10393-11363        9851|-------------------------
--    00060         9116     57980    720006607  ||      11364-12334        7459|-------------------
--    00070         8414     71692    840006018  ||      12335-13305        5562|--------------
--    00080         7780     86530    960006224  ||      13306-14276        4080|-----------
--    00090         7222    102548   1080010087  ||      14277-15247        3064|--------
--    00100         6509    119828   1200003459  ||      15248-16218        2380|------
--    001.000x              119829   1200003459  ||      16219-17189        1776|-----
--                                               ||      17190-18160        1278|----
--                                               ||      18161-19131         924|---
--                                               ||      19132-20102         670|--
--                                               ||      20103-21073         524|--
--                                               ||      21074-22044         383|-
--                                               ||      22045-23015         316|-
--                                               ||      23016-23986         214|-
--                                               ||      23987-24957         161|-
--                                               ||      24958-25928         127|-
--                                               ||      25929-26899          91|-
--                                               ||      26900-27870          78|-
--                                               ||      27871-28841          41|-
--                                               ||      28842-29812          41|-
--                                               ||      29813-30783          34|-
--                                               ||      30784-31754          18|-
--                                               ||      31755-32725          24|-
--                                               ||      32726-33696          12|-
--                                               ||      33697-34667           9|-
--                                               ||      34668-35638          10|-
--                                               ||      35639-36609           2|-
--                                               ||      36610-37580           6|-
--                                               ||      37581-38551           0|
--                                               ||      38552-39522           1|-
--                                               ||      39523-40493           3|-
--                                               ||      40494-41464           0|
--                                               ||      41465-42435           3|-
--                                               ||      42436-43406           0|
--                                               ||      43407-44377           0|
--                                               ||      44378-45348           1|-
--                                               ||      45349-46319           0|
--                                               ||      46320-47290           0|
--                                               ||      47291-48261           1|-
--                                               ||      48262-49232           0|
--                                               ||      49233-50203           0|
--                                               ||      50204-51174           1|-
--                                               ||      51175-52145           0|
--                                               ||      52146-53116           1|-
--                                               ||      53117-54087           0|
--                                               ||      54088-55058           1|-
--    
----------------------------------------
-- Starting command on Tue Nov 16 16:10:51 2021 with 2051.482 GB free disk space

    cd correction/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Tue Nov 16 16:10:51 2021 (like a bat out of hell) with 2051.48 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--        01  2.45 GB       2
--        02  1.54 GB       2
--        04  0.77 GB       2
--        06  0.52 GB       2
--        08  0.39 GB       2
--        12  0.26 GB       2
--
--  For 119829 reads with 1200003459 bases, limit to 12 batches.
--  Will count kmers using 01 jobs, each using 4 GB and 4 threads.
--
-- Finished stage 'merylConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 16:10:51 2021 with 2051.479 GB free disk space (1 processes; 4 concurrently)

    cd correction/0-mercounts
    ./meryl-count.sh 1 > ./meryl-count.000001.out 2>&1

-- Finished on Tue Nov 16 16:12:10 2021 (79 seconds) with 2050.354 GB free disk space
----------------------------------------
-- Found 1 Kmer counting (meryl) outputs.
-- Finished stage 'cor-merylCountCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 16:12:10 2021 with 2050.354 GB free disk space (1 processes; 4 concurrently)

    cd correction/0-mercounts
    ./meryl-process.sh 1 > ./meryl-process.000001.out 2>&1

-- Finished on Tue Nov 16 16:12:14 2021 (4 seconds) with 2051.179 GB free disk space
----------------------------------------
-- Meryl finished successfully.  Kmer frequency histogram:
--
-- WARNING: gnuplot failed.
--
----------------------------------------
--
--  16-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2  30618151 ********************************************************************** 0.5176 0.0607
--       3-     5  14629875 *********************************                                      0.6694 0.0874
--       6-    10   2823746 ******                                                                 0.7835 0.1182
--      11-    17    698332 *                                                                      0.8157 0.1340
--      18-    26    219074                                                                        0.8252 0.1418
--      27-    37    129810                                                                        0.8284 0.1460
--      38-    50    461993 *                                                                      0.8306 0.1504
--      51-    65   2396119 *****                                                                  0.8395 0.1745
--      66-    82   4697091 **********                                                             0.8832 0.3296
--      83-   101   1690516 ***                                                                    0.9614 0.6712
--     102-   122    181367                                                                        0.9870 0.8056
--     123-   145    150944                                                                        0.9898 0.8240
--     146-   170    163395                                                                        0.9924 0.8446
--     171-   197     79326                                                                        0.9951 0.8697
--     198-   226     42001                                                                        0.9964 0.8835
--     227-   257     34262                                                                        0.9971 0.8923
--     258-   290     23468                                                                        0.9977 0.9004
--     291-   325     20345                                                                        0.9981 0.9067
--     326-   362     14938                                                                        0.9984 0.9129
--     363-   401     10268                                                                        0.9987 0.9179
--     402-   442      8177                                                                        0.9989 0.9218
--     443-   485      6279                                                                        0.9990 0.9252
--     486-   530      5630                                                                        0.9991 0.9280
--     531-   577      4776                                                                        0.9992 0.9309
--     578-   626      4506                                                                        0.9993 0.9335
--     627-   677      4262                                                                        0.9994 0.9362
--     678-   730      3877                                                                        0.9994 0.9389
--     731-   785      2623                                                                        0.9995 0.9416
--     786-   842      2198                                                                        0.9995 0.9436
--     843-   901      1975                                                                        0.9996 0.9453
--     902-   962      1952                                                                        0.9996 0.9470
--     963-  1025      1810                                                                        0.9996 0.9488
--    1026-  1090      1671                                                                        0.9997 0.9506
--    1091-  1157      1534                                                                        0.9997 0.9524
--    1158-  1226      1400                                                                        0.9997 0.9541
--    1227-  1297      1256                                                                        0.9997 0.9557
--    1298-  1370      1048                                                                        0.9998 0.9573
--    1371-  1445       972                                                                        0.9998 0.9587
--    1446-  1522       709                                                                        0.9998 0.9600
--    1523-  1601       520                                                                        0.9998 0.9611
--
--           0 (max occurrences)
--  1008990965 (total mers, non-unique)
--    59152920 (distinct mers, non-unique)
--           0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'correction/0-mercounts/WB_SUP.ms16'.
--
-- OVERLAPPER (mhap) (correction)
--
-- Set corMhapSensitivity=low based on read coverage of 100.
--
-- PARAMETERS: hashes=256, minMatches=3, threshold=0.8
--
-- Given 5.4 GB, can fit 16200 reads per block.
-- For 9 blocks, set stride to 2 blocks.
-- Logging partitioning to 'correction/1-overlapper/partitioning.log'.
-- Configured 8 mhap precompute jobs.
-- Configured 17 mhap overlap jobs.
-- Finished stage 'cor-mhapConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Tue Nov 16 16:12:14 2021 with 2051.472 GB free disk space (8 processes; 1 concurrently)

    cd correction/1-overlapper
    ./precompute.sh 1 > ./precompute.000001.out 2>&1
    ./precompute.sh 2 > ./precompute.000002.out 2>&1
    ./precompute.sh 3 > ./precompute.000003.out 2>&1
    ./precompute.sh 4 > ./precompute.000004.out 2>&1
    ./precompute.sh 5 > ./precompute.000005.out 2>&1
    ./precompute.sh 6 > ./precompute.000006.out 2>&1
    ./precompute.sh 7 > ./precompute.000007.out 2>&1
    ./precompute.sh 8 > ./precompute.000008.out 2>&1

-- Finished on Tue Nov 16 16:29:01 2021 (1007 seconds) with 2049.446 GB free disk space
----------------------------------------
-- All 8 mhap precompute jobs finished successfully.
-- Finished stage 'cor-mhapPrecomputeCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Tue Nov 16 16:29:01 2021 with 2049.446 GB free disk space (17 processes; 1 concurrently)

    cd correction/1-overlapper
    ./mhap.sh 1 > ./mhap.000001.out 2>&1
    ./mhap.sh 2 > ./mhap.000002.out 2>&1
    ./mhap.sh 3 > ./mhap.000003.out 2>&1
    ./mhap.sh 4 > ./mhap.000004.out 2>&1
    ./mhap.sh 5 > ./mhap.000005.out 2>&1
    ./mhap.sh 6 > ./mhap.000006.out 2>&1
    ./mhap.sh 7 > ./mhap.000007.out 2>&1
    ./mhap.sh 8 > ./mhap.000008.out 2>&1
    ./mhap.sh 9 > ./mhap.000009.out 2>&1
    ./mhap.sh 10 > ./mhap.000010.out 2>&1
    ./mhap.sh 11 > ./mhap.000011.out 2>&1
    ./mhap.sh 12 > ./mhap.000012.out 2>&1
    ./mhap.sh 13 > ./mhap.000013.out 2>&1
    ./mhap.sh 14 > ./mhap.000014.out 2>&1
    ./mhap.sh 15 > ./mhap.000015.out 2>&1
    ./mhap.sh 16 > ./mhap.000016.out 2>&1
    ./mhap.sh 17 > ./mhap.000017.out 2>&1

-- Finished on Tue Nov 16 16:30:15 2021 (74 seconds) with 2049.002 GB free disk space
----------------------------------------
-- Found 17 mhap overlap output files.
-- Finished stage 'cor-mhapCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 16:30:15 2021 with 2049.001 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/ovStoreConfig \
     -S ../WB_SUP.seqStore \
     -M 4-8 \
     -L ./1-overlapper/ovljob.files \
     -create ./WB_SUP.ovlStore.config \
     > ./WB_SUP.ovlStore.config.txt \
    2> ./WB_SUP.ovlStore.config.err

-- Finished on Tue Nov 16 16:30:15 2021 (furiously fast) with 2049.001 GB free disk space
----------------------------------------
--
-- Creating overlap store correction/WB_SUP.ovlStore using:
--      2 buckets
--      2 slices
--        using at most 1 GB memory each
-- Finished stage 'cor-overlapStoreConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovB' concurrent execution on Tue Nov 16 16:30:15 2021 with 2049 GB free disk space (2 processes; 15 concurrently)

    cd correction/WB_SUP.ovlStore.BUILDING
    ./scripts/1-bucketize.sh 1 > ./logs/1-bucketize.000001.out 2>&1
    ./scripts/1-bucketize.sh 2 > ./logs/1-bucketize.000002.out 2>&1

-- Finished on Tue Nov 16 16:30:16 2021 (one second) with 2048.833 GB free disk space
----------------------------------------
-- Overlap store bucketizer finished.
-- Finished stage 'cor-overlapStoreBucketizerCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovS' concurrent execution on Tue Nov 16 16:30:16 2021 with 2048.833 GB free disk space (2 processes; 7 concurrently)

    cd correction/WB_SUP.ovlStore.BUILDING
    ./scripts/2-sort.sh 1 > ./logs/2-sort.000001.out 2>&1
    ./scripts/2-sort.sh 2 > ./logs/2-sort.000002.out 2>&1

-- Finished on Tue Nov 16 16:30:17 2021 (one second) with 2048.64 GB free disk space
----------------------------------------
-- Overlap store sorter finished.
-- Finished stage 'cor-overlapStoreSorterCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 16:30:17 2021 with 2048.64 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/ovStoreIndexer \
      -O  ./WB_SUP.ovlStore.BUILDING \
      -S ../WB_SUP.seqStore \
      -C  ./WB_SUP.ovlStore.config \
      -delete \
    > ./WB_SUP.ovlStore.BUILDING.index.err 2>&1

-- Finished on Tue Nov 16 16:30:17 2021 (lickety-split) with 2048.81 GB free disk space
----------------------------------------
-- Overlap store indexer finished.
-- Checking store.
----------------------------------------
-- Starting command on Tue Nov 16 16:30:17 2021 with 2048.81 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/ovStoreDump \
     -S ../WB_SUP.seqStore \
     -O  ./WB_SUP.ovlStore \
     -counts \
     > ./WB_SUP.ovlStore/counts.dat 2> ./WB_SUP.ovlStore/counts.err

-- Finished on Tue Nov 16 16:30:18 2021 (one second) with 2048.808 GB free disk space
----------------------------------------
--
-- Overlap store 'correction/WB_SUP.ovlStore' successfully constructed.
-- Found 9588688 overlaps for 119798 reads; 31 reads have no overlaps.
--
-- Finished stage 'cor-createOverlapStore', reset canuIteration.
-- Set corMinCoverage=4 based on read coverage of 100.
-- Computing correction layouts.
--   Local  filter coverage   80
--   Global filter coverage   40
----------------------------------------
-- Starting command on Tue Nov 16 16:30:18 2021 with 2048.81 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/generateCorrectionLayouts \
      -S ../WB_SUP.seqStore \
      -O  ./WB_SUP.ovlStore \
      -C  ./WB_SUP.corStore.WORKING \
      -eC 80 \
      -xC 40 \
    > ./WB_SUP.corStore.err 2>&1

-- Finished on Tue Nov 16 16:30:18 2021 (lickety-split) with 2048.651 GB free disk space
----------------------------------------
-- Finished stage 'cor-buildCorrectionLayoutsConfigure', reset canuIteration.
-- Computing correction layouts.
----------------------------------------
-- Starting command on Tue Nov 16 16:30:18 2021 with 2048.652 GB free disk space

    cd correction/2-correction
    /home/jon/software/canu-2.2/bin/filterCorrectionLayouts \
      -S  ../../WB_SUP.seqStore \
      -C     ../WB_SUP.corStore \
      -R      ./WB_SUP.readsToCorrect.WORKING \
      -cc 4 \
      -cl 1000 \
      -g  12000000 \
      -c  40 \
    > ./WB_SUP.readsToCorrect.err 2>&1

-- Finished on Tue Nov 16 16:30:19 2021 (one second) with 2048.643 GB free disk space
----------------------------------------
--                             original      original
--                            raw reads     raw reads
--   category                w/overlaps  w/o/overlaps
--   -------------------- ------------- -------------
--   Number of Reads             119043           786
--   Number of Bases         1193749376       5977470
--   Coverage                    99.479         0.498
--   Median                        9020          7519
--   Mean                         10027          7604
--   N50                           9933          7713
--   Minimum                       6509             0
--   Maximum                      55030         12700
--   
--                                        --------corrected---------  ----------rescued----------
--                             evidence                     expected                     expected
--   category                     reads            raw     corrected            raw     corrected
--   -------------------- -------------  ------------- -------------  ------------- -------------
--   Number of Reads             118379          34340         34340            813           813
--   Number of Bases         1186819479      483565899     480003777        7432964       3969296
--   Coverage                    98.902         40.297        40.000          0.619         0.331
--   Median                        9016          13060         12969           8384          4661
--   Mean                         10025          14081         13977           9142          4882
--   N50                           9929          13719         13607           8974          6575
--   Minimum                       6509          10678         10676           6610          1004
--   Maximum                      55030          52653         52640          35383         10640
--   
--                        --------uncorrected--------
--                                           expected
--   category                       raw     corrected
--   -------------------- ------------- -------------
--   Number of Reads              84676         84676
--   Number of Bases          708727983     677804341
--   Coverage                    59.061        56.484
--   Median                        8167          8025
--   Mean                          8369          8004
--   N50                           8399          8305
--   Minimum                          0             0
--   Maximum                      55030         54876
--   
--   Maximum Memory          1032441278
-- Finished stage 'cor-filterCorrectionLayouts', reset canuIteration.
--
-- Correction jobs estimated to need at most 0.961 GB for computation.
-- Correction jobs will request 6 GB each.
--
-- Local: cor        6.000 GB    4 CPUs x   4 jobs    24.000 GB  16 CPUs  (read correction)
--
--
-- Configuring correction jobs:
--   Reads estimated to need at most 0.961 GB for computation.
--   Jobs will request 6 GB each.
----------------------------------------
-- Starting command on Tue Nov 16 16:30:19 2021 with 2048.642 GB free disk space

    cd correction/2-correction
    ./correctReadsPartition.sh \
    > ./correctReadsPartition.err 2>&1

-- Finished on Tue Nov 16 16:30:19 2021 (in the blink of an eye) with 2048.642 GB free disk space
----------------------------------------
-- Finished stage 'cor-generateCorrectedReadsConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cor' concurrent execution on Tue Nov 16 16:30:19 2021 with 2048.642 GB free disk space (4 processes; 4 concurrently)

    cd correction/2-correction
    ./correctReads.sh 1 > ./correctReads.000001.out 2>&1
    ./correctReads.sh 2 > ./correctReads.000002.out 2>&1
    ./correctReads.sh 3 > ./correctReads.000003.out 2>&1
    ./correctReads.sh 4 > ./correctReads.000004.out 2>&1

-- Finished on Tue Nov 16 16:49:55 2021 (1176 seconds) with 2047.676 GB free disk space
----------------------------------------
-- Found 4 read correction output files.
-- Finished stage 'cor-generateCorrectedReadsCheck', reset canuIteration.
-- Found 4 read correction output files.
-- Finished stage 'cor-generateCorrectedReadsCheck', reset canuIteration.
--
-- Loading corrected reads into corStore and seqStore.
----------------------------------------
-- Starting command on Tue Nov 16 16:49:55 2021 with 2047.676 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/loadCorrectedReads \
      -S ../WB_SUP.seqStore \
      -C ./WB_SUP.corStore \
      -L ./2-correction/corjob.files \
    >  ./WB_SUP.loadCorrectedReads.log \
    2> ./WB_SUP.loadCorrectedReads.err

-- Finished on Tue Nov 16 16:49:59 2021 (4 seconds) with 2047.435 GB free disk space
----------------------------------------
--
-- In sequence store './WB_SUP.seqStore':
--   Found 35115 reads.
--   Found 484668327 bases (40.38 times coverage).
--    Histogram of corrected reads:
--    
--    G=484668327                        sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010        19954      2081     48473710  ||       1016-2057          109|-
--    00020        17226      4718     96947799  ||       2058-3099          112|-
--    00030        15678      7675    145405887  ||       3100-4141          117|-
--    00040        14528     10891    193877888  ||       4142-5183           83|-
--    00050        13615     14340    242336543  ||       5184-6225           90|-
--    00060        12848     18009    290805282  ||       6226-7267          145|--
--    00070        12218     21879    339270602  ||       7268-8309          118|-
--    00080        11646     25945    387736371  ||       8310-9351          110|-
--    00090        11133     30202    436210644  ||       9352-10393         195|--
--    00100         1016     35114    484668327  ||      10394-11435        6395|-------------------------------------------------------
--    001.000x               35115    484668327  ||      11436-12477        7442|---------------------------------------------------------------
--                                               ||      12478-13519        5437|-----------------------------------------------
--                                               ||      13520-14561        3989|----------------------------------
--                                               ||      14562-15603        2925|-------------------------
--                                               ||      15604-16645        2165|-------------------
--                                               ||      16646-17687        1605|--------------
--                                               ||      17688-18729        1109|----------
--                                               ||      18730-19771         760|-------
--                                               ||      19772-20813         585|-----
--                                               ||      20814-21855         445|----
--                                               ||      21856-22897         332|---
--                                               ||      22898-23939         216|--
--                                               ||      23940-24981         175|--
--                                               ||      24982-26023         127|--
--                                               ||      26024-27065          92|-
--                                               ||      27066-28107          67|-
--                                               ||      28108-29149          42|-
--                                               ||      29150-30191          28|-
--                                               ||      30192-31233          31|-
--                                               ||      31234-32275          20|-
--                                               ||      32276-33317          12|-
--                                               ||      33318-34359          12|-
--                                               ||      34360-35401           5|-
--                                               ||      35402-36443           4|-
--                                               ||      36444-37485           4|-
--                                               ||      37486-38527           1|-
--                                               ||      38528-39569           2|-
--                                               ||      39570-40611           4|-
--                                               ||      40612-41653           1|-
--                                               ||      41654-42695           2|-
--                                               ||      42696-43737           0|
--                                               ||      43738-44779           0|
--                                               ||      44780-45821           1|-
--                                               ||      45822-46863           0|
--                                               ||      46864-47905           0|
--                                               ||      47906-48947           0|
--                                               ||      48948-49989           0|
--                                               ||      49990-51031           0|
--                                               ||      51032-52073           0|
--                                               ||      52074-53115           1|-
--    
--
-- Purging correctReads output after loading into stores.
-- Purged 4 .cns outputs.
-- Purged 8 .out job log outputs.
--
-- No corrected reads generated, overlaps used for correction saved.
-- Finished stage 'cor-loadCorrectedReads', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 16:49:59 2021 with 2048.401 GB free disk space

    cd .
    /home/jon/software/canu-2.2/bin/sqStoreDumpFASTQ \
      -corrected \
      -S ./WB_SUP.seqStore \
      -o ./WB_SUP.correctedReads.gz \
      -fasta \
      -nolibname \
    > WB_SUP.correctedReads.fasta.err 2>&1

-- Finished on Tue Nov 16 16:50:05 2021 (6 seconds) with 2048.251 GB free disk space
----------------------------------------
--
-- Corrected reads saved in 'WB_SUP.correctedReads.fasta.gz'.
-- Finished stage 'cor-dumpCorrectedReads', reset canuIteration.
--
-- BEGIN TRIMMING
----------------------------------------
-- Starting command on Tue Nov 16 16:50:05 2021 with 2048.25 GB free disk space

    cd trimming/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Tue Nov 16 16:50:05 2021 (in the blink of an eye) with 2048.249 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--        01  1.93 GB       2
--        02  0.97 GB       2
--        04  0.48 GB       2
--
--  For 35115 reads with 484668327 bases, limit to 4 batches.
--  Will count kmers using 01 jobs, each using 3 GB and 4 threads.
--
-- Finished stage 'merylConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 16:50:05 2021 with 2048.249 GB free disk space (1 processes; 4 concurrently)

    cd trimming/0-mercounts
    ./meryl-count.sh 1 > ./meryl-count.000001.out 2>&1

-- Finished on Tue Nov 16 16:50:29 2021 (24 seconds) with 2048.134 GB free disk space
----------------------------------------
-- Found 1 Kmer counting (meryl) outputs.
-- Finished stage 'obt-merylCountCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 16:50:29 2021 with 2048.134 GB free disk space (1 processes; 4 concurrently)

    cd trimming/0-mercounts
    ./meryl-process.sh 1 > ./meryl-process.000001.out 2>&1

-- Finished on Tue Nov 16 16:50:30 2021 (one second) with 2048.152 GB free disk space
----------------------------------------
-- Meryl finished successfully.  Kmer frequency histogram:
--
-- WARNING: gnuplot failed.
--
----------------------------------------
--
--  22-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2    425271 *******                                                                0.0384 0.0018
--       3-     4    272688 ****                                                                   0.0543 0.0029
--       5-     7    137931 **                                                                     0.0687 0.0043
--       8-    11     78052 *                                                                      0.0777 0.0058
--      12-    16     81181 *                                                                      0.0839 0.0072
--      17-    22    170305 **                                                                     0.0916 0.0099
--      23-    29    621191 **********                                                             0.1090 0.0183
--      30-    37   2530747 ********************************************                           0.1784 0.0628
--      38-    46   3979530 ********************************************************************** 0.4313 0.2673
--      47-    56   1912434 *********************************                                      0.7795 0.6107
--      57-    67    312569 *****                                                                  0.9267 0.7842
--      68-    79    121255 **                                                                     0.9505 0.8180
--      80-    92    127137 **                                                                     0.9611 0.8362
--      93-   106     99408 *                                                                      0.9727 0.8593
--     107-   121     49353                                                                        0.9811 0.8787
--     122-   137     31706                                                                        0.9854 0.8899
--     138-   154     24575                                                                        0.9882 0.8985
--     155-   172     19431                                                                        0.9904 0.9058
--     173-   191     15457                                                                        0.9921 0.9125
--     192-   211     10088                                                                        0.9935 0.9181
--     212-   232      7241                                                                        0.9944 0.9222
--     233-   254      5846                                                                        0.9950 0.9256
--     255-   277      4759                                                                        0.9956 0.9286
--     278-   301      3781                                                                        0.9960 0.9311
--     302-   326      4948                                                                        0.9963 0.9334
--     327-   352      4768                                                                        0.9968 0.9367
--     353-   379      4175                                                                        0.9972 0.9400
--     380-   407      3084                                                                        0.9976 0.9432
--     408-   436      1891                                                                        0.9978 0.9457
--     437-   466      2010                                                                        0.9980 0.9473
--     467-   497      1768                                                                        0.9982 0.9492
--     498-   529      1631                                                                        0.9983 0.9510
--     530-   562      1770                                                                        0.9985 0.9528
--     563-   596       882                                                                        0.9986 0.9547
--     597-   631       905                                                                        0.9987 0.9558
--     632-   667       469                                                                        0.9988 0.9569
--     668-   704       778                                                                        0.9988 0.9576
--     705-   742       431                                                                        0.9989 0.9587
--     743-   781       246                                                                        0.9990 0.9593
--     782-   821       650                                                                        0.9990 0.9597
--
--           0 (max occurrences)
--   480552570 (total mers, non-unique)
--    11083010 (distinct mers, non-unique)
--           0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'trimming/0-mercounts/WB_SUP.ms22'.
--
-- OVERLAPPER (normal) (trimming) erate=0.12
--
----------------------------------------
-- Starting command on Tue Nov 16 16:50:30 2021 with 2048.247 GB free disk space

    cd trimming/1-overlapper
    /home/jon/software/canu-2.2/bin/overlapInCorePartition \
     -S  ../../WB_SUP.seqStore \
     -hl 80000000 \
     -rl 1000000000 \
     -ol 500 \
     -o  ./WB_SUP.partition \
    > ./WB_SUP.partition.err 2>&1

-- Finished on Tue Nov 16 16:50:30 2021 (in the blink of an eye) with 2048.246 GB free disk space
----------------------------------------
--
-- Configured 7 overlapInCore jobs.
-- Finished stage 'obt-overlapConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'obtovl' concurrent execution on Tue Nov 16 16:50:30 2021 with 2048.245 GB free disk space (7 processes; 2 concurrently)

    cd trimming/1-overlapper
    ./overlap.sh 1 > ./overlap.000001.out 2>&1
    ./overlap.sh 2 > ./overlap.000002.out 2>&1
    ./overlap.sh 3 > ./overlap.000003.out 2>&1
    ./overlap.sh 4 > ./overlap.000004.out 2>&1
    ./overlap.sh 5 > ./overlap.000005.out 2>&1
    ./overlap.sh 6 > ./overlap.000006.out 2>&1
    ./overlap.sh 7 > ./overlap.000007.out 2>&1

-- Finished on Tue Nov 16 17:13:31 2021 (1381 seconds) with 2048.153 GB free disk space
----------------------------------------
-- Found 7 overlapInCore output files.
--
-- overlapInCore compute 'trimming/1-overlapper':
--   kmer hits
--     with no overlap         16407799  1.28571429 +- 2058715.237
--     with an overlap          4515620  .571428571 +- 548925.22
--
--   overlaps                   4515309  .142857143 +- 548875.39
--     contained                      0           0 +- 0
--     dovetail                       0           0 +- 0
--
--   overlaps rejected
--     multiple per pair              0           0 +- 0
--     bad short window               0           0 +- 0
--     bad long window                0           0 +- 0
-- Finished stage 'obt-overlapCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:31 2021 with 2048.152 GB free disk space

    cd trimming
    /home/jon/software/canu-2.2/bin/ovStoreConfig \
     -S ../WB_SUP.seqStore \
     -M 3 \
     -L ./1-overlapper/ovljob.files \
     -create ./WB_SUP.ovlStore.config \
     > ./WB_SUP.ovlStore.config.txt \
    2> ./WB_SUP.ovlStore.config.err

-- Finished on Tue Nov 16 17:13:31 2021 (furiously fast) with 2048.152 GB free disk space
----------------------------------------
--
-- Creating overlap store trimming/WB_SUP.ovlStore using:
--      2 buckets
--      2 slices
--        using at most 1 GB memory each
-- Finished stage 'obt-overlapStoreConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovB' concurrent execution on Tue Nov 16 17:13:31 2021 with 2048.151 GB free disk space (2 processes; 15 concurrently)

    cd trimming/WB_SUP.ovlStore.BUILDING
    ./scripts/1-bucketize.sh 1 > ./logs/1-bucketize.000001.out 2>&1
    ./scripts/1-bucketize.sh 2 > ./logs/1-bucketize.000002.out 2>&1

-- Finished on Tue Nov 16 17:13:32 2021 (one second) with 2047.994 GB free disk space
----------------------------------------
-- Overlap store bucketizer finished.
-- Finished stage 'obt-overlapStoreBucketizerCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovS' concurrent execution on Tue Nov 16 17:13:32 2021 with 2047.994 GB free disk space (2 processes; 7 concurrently)

    cd trimming/WB_SUP.ovlStore.BUILDING
    ./scripts/2-sort.sh 1 > ./logs/2-sort.000001.out 2>&1
    ./scripts/2-sort.sh 2 > ./logs/2-sort.000002.out 2>&1

-- Finished on Tue Nov 16 17:13:33 2021 (one second) with 2047.811 GB free disk space
----------------------------------------
-- Overlap store sorter finished.
-- Finished stage 'obt-overlapStoreSorterCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:33 2021 with 2047.811 GB free disk space

    cd trimming
    /home/jon/software/canu-2.2/bin/ovStoreIndexer \
      -O  ./WB_SUP.ovlStore.BUILDING \
      -S ../WB_SUP.seqStore \
      -C  ./WB_SUP.ovlStore.config \
      -delete \
    > ./WB_SUP.ovlStore.BUILDING.index.err 2>&1

-- Finished on Tue Nov 16 17:13:33 2021 (fast as lightning) with 2047.971 GB free disk space
----------------------------------------
-- Overlap store indexer finished.
-- Checking store.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:33 2021 with 2047.971 GB free disk space

    cd trimming
    /home/jon/software/canu-2.2/bin/ovStoreDump \
     -S ../WB_SUP.seqStore \
     -O  ./WB_SUP.ovlStore \
     -counts \
     > ./WB_SUP.ovlStore/counts.dat 2> ./WB_SUP.ovlStore/counts.err

-- Finished on Tue Nov 16 17:13:33 2021 (furiously fast) with 2047.97 GB free disk space
----------------------------------------
--
-- Overlap store 'trimming/WB_SUP.ovlStore' successfully constructed.
-- Found 9031240 overlaps for 35114 reads; 84715 reads have no overlaps.
--
-- Finished stage 'obt-createOverlapStore', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:34 2021 with 2047.971 GB free disk space

    cd trimming/3-overlapbasedtrimming
    /home/jon/software/canu-2.2/bin/trimReads \
      -S  ../../WB_SUP.seqStore \
      -O  ../WB_SUP.ovlStore \
      -Co ./WB_SUP.1.trimReads.clear \
      -e  0.12 \
      -minlength 1000 \
      -ol 500 \
      -oc 2 \
      -o  ./WB_SUP.1.trimReads \
    >     ./WB_SUP.1.trimReads.err 2>&1

-- Finished on Tue Nov 16 17:13:35 2021 (one second) with 2047.961 GB free disk space
----------------------------------------
--  PARAMETERS:
--  ----------
--     1000    (reads trimmed below this many bases are deleted)
--   0.1200    (use overlaps at or below this fraction error)
--      500    (break region if overlap is less than this long, for 'largest covered' algorithm)
--        2    (break region if overlap coverage is less than this many reads, for 'largest covered' algorithm)
--  
--  INPUT READS:
--  -----------
--  119829 reads    484668327 bases (reads processed)
--       0 reads            0 bases (reads not processed, previously deleted)
--       0 reads            0 bases (reads not processed, in a library where trimming isn't allowed)
--  
--  OUTPUT READS:
--  ------------
--    9021 reads    125442180 bases (trimmed reads output)
--   26091 reads    358326849 bases (reads with no change, kept as is)
--   84715 reads         1096 bases (reads with no overlaps, deleted)
--       2 reads         2766 bases (reads with short trimmed length, deleted)
--  
--  TRIMMING DETAILS:
--  ----------------
--    1202 reads       353933 bases (bases trimmed from the 5' end of a read)
--    8095 reads       541503 bases (bases trimmed from the 3' end of a read)
-- Finished stage 'obt-trimReads', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:35 2021 with 2047.962 GB free disk space

    cd trimming/3-overlapbasedtrimming
    /home/jon/software/canu-2.2/bin/splitReads \
      -S  ../../WB_SUP.seqStore \
      -O  ../WB_SUP.ovlStore \
      -Ci ./WB_SUP.1.trimReads.clear \
      -Co ./WB_SUP.2.splitReads.clear \
      -e  0.12 \
      -minlength 1000 \
      -o  ./WB_SUP.2.splitReads \
    >     ./WB_SUP.2.splitReads.err 2>&1

-- Finished on Tue Nov 16 17:13:36 2021 (one second) with 2047.96 GB free disk space
----------------------------------------
--  PARAMETERS:
--  ----------
--     1000    (reads trimmed below this many bases are deleted)
--   0.1200    (use overlaps at or below this fraction error)
--  INPUT READS:
--  -----------
--   35112 reads    484664465 bases (reads processed)
--   84717 reads         3862 bases (reads not processed, previously deleted)
--       0 reads            0 bases (reads not processed, in a library where trimming isn't allowed)
--  
--  PROCESSED:
--  --------
--       0 reads            0 bases (no overlaps)
--       0 reads            0 bases (no coverage after adjusting for trimming done already)
--       0 reads            0 bases (processed for chimera)
--       0 reads            0 bases (processed for spur)
--   35112 reads    484664465 bases (processed for subreads)
--  
--  READS WITH SIGNALS:
--  ------------------
--       0 reads            0 signals (number of 5' spur signal)
--       0 reads            0 signals (number of 3' spur signal)
--       0 reads            0 signals (number of chimera signal)
--       0 reads            0 signals (number of subread signal)
--  
--  SIGNALS:
--  -------
--       0 reads            0 bases (size of 5' spur signal)
--       0 reads            0 bases (size of 3' spur signal)
--       0 reads            0 bases (size of chimera signal)
--       0 reads            0 bases (size of subread signal)
--  
--  TRIMMING:
--  --------
--       0 reads            0 bases (trimmed from the 5' end of the read)
--       0 reads            0 bases (trimmed from the 3' end of the read)
-- Finished stage 'obt-splitReads', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:36 2021 with 2047.96 GB free disk space

    cd trimming/3-overlapbasedtrimming
    /home/jon/software/canu-2.2/bin/loadTrimmedReads \
      -S ../../WB_SUP.seqStore \
      -c ./WB_SUP.2.splitReads.clear \
    > ./WB_SUP.loadTrimmedReads.err 2>&1

-- Finished on Tue Nov 16 17:13:38 2021 (2 seconds) with 2047.948 GB free disk space
----------------------------------------
--
-- In sequence store './WB_SUP.seqStore':
--   Found 35112 reads.
--   Found 483769029 bases (40.31 times coverage).
--    Histogram of corrected-trimmed reads:
--    
--    G=483769029                        sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010        19931      2079     48387893  ||       1016-2057          107|-
--    00020        17214      4713     96767932  ||       2058-3099          117|-
--    00030        15662      7667    145140244  ||       3100-4141          114|-
--    00040        14510     10880    193522007  ||       4142-5183           90|-
--    00050        13601     14326    241887220  ||       5184-6225           95|-
--    00060        12833     17992    290262083  ||       6226-7267          151|--
--    00070        12202     21860    338640074  ||       7268-8309          141|--
--    00080        11632     25924    387015451  ||       8310-9351          142|--
--    00090        11122     30178    435398442  ||       9352-10393         248|---
--    00100         1016     35111    483769029  ||      10394-11435        6401|-------------------------------------------------------
--    001.000x               35112    483769029  ||      11436-12477        7426|---------------------------------------------------------------
--                                               ||      12478-13519        5396|----------------------------------------------
--                                               ||      13520-14561        3975|----------------------------------
--                                               ||      14562-15603        2900|-------------------------
--                                               ||      15604-16645        2155|-------------------
--                                               ||      16646-17687        1599|--------------
--                                               ||      17688-18729        1100|----------
--                                               ||      18730-19771         759|-------
--                                               ||      19772-20813         589|-----
--                                               ||      20814-21855         435|----
--                                               ||      21856-22897         327|---
--                                               ||      22898-23939         216|--
--                                               ||      23940-24981         176|--
--                                               ||      24982-26023         126|--
--                                               ||      26024-27065          94|-
--                                               ||      27066-28107          64|-
--                                               ||      28108-29149          42|-
--                                               ||      29150-30191          28|-
--                                               ||      30192-31233          32|-
--                                               ||      31234-32275          18|-
--                                               ||      32276-33317          12|-
--                                               ||      33318-34359          12|-
--                                               ||      34360-35401           5|-
--                                               ||      35402-36443           4|-
--                                               ||      36444-37485           4|-
--                                               ||      37486-38527           1|-
--                                               ||      38528-39569           2|-
--                                               ||      39570-40611           4|-
--                                               ||      40612-41653           1|-
--                                               ||      41654-42695           2|-
--                                               ||      42696-43737           0|
--                                               ||      43738-44779           0|
--                                               ||      44780-45821           1|-
--                                               ||      45822-46863           0|
--                                               ||      46864-47905           0|
--                                               ||      47906-48947           0|
--                                               ||      48948-49989           0|
--                                               ||      49990-51031           0|
--                                               ||      51032-52073           0|
--                                               ||      52074-53115           1|-
--    
--
-- No trimmed reads generated, overlaps used for trimming saved.
-- Finished stage 'obt-dumpReads', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:38 2021 with 2047.951 GB free disk space

    cd .
    /home/jon/software/canu-2.2/bin/sqStoreDumpFASTQ \
      -trimmed \
      -S ./WB_SUP.seqStore \
      -o ./WB_SUP.trimmedReads.gz \
      -fasta \
      -trimmed -normal -nolibname \
    > ./WB_SUP.trimmedReads.fasta.err 2>&1

-- Finished on Tue Nov 16 17:13:44 2021 (6 seconds) with 2047.801 GB free disk space
----------------------------------------
--
-- Trimmed reads saved in 'WB_SUP.trimmedReads.fasta.gz'.
-- Finished stage 'cor-dumpTrimmedReads', reset canuIteration.
--
-- BEGIN ASSEMBLY
----------------------------------------
-- Starting command on Tue Nov 16 17:13:44 2021 with 2047.801 GB free disk space

    cd unitigging/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Tue Nov 16 17:13:44 2021 (like a bat out of hell) with 2047.8 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--        01  1.93 GB       2
--        02  0.97 GB       2
--        04  0.48 GB       2
--
--  For 35112 reads with 483769029 bases, limit to 4 batches.
--  Will count kmers using 01 jobs, each using 3 GB and 4 threads.
--
-- Finished stage 'merylConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 17:13:44 2021 with 2047.799 GB free disk space (1 processes; 4 concurrently)

    cd unitigging/0-mercounts
    ./meryl-count.sh 1 > ./meryl-count.000001.out 2>&1

-- Finished on Tue Nov 16 17:14:07 2021 (23 seconds) with 2047.685 GB free disk space
----------------------------------------
-- Found 1 Kmer counting (meryl) outputs.
-- Finished stage 'utg-merylCountCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 17:14:07 2021 with 2047.685 GB free disk space (1 processes; 4 concurrently)

    cd unitigging/0-mercounts
    ./meryl-process.sh 1 > ./meryl-process.000001.out 2>&1

-- Finished on Tue Nov 16 17:14:07 2021 (in the blink of an eye) with 2047.703 GB free disk space
----------------------------------------
-- Meryl finished successfully.  Kmer frequency histogram:
--
-- WARNING: gnuplot failed.
--
----------------------------------------
--
--  22-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2    417553 *******                                                                0.0377 0.0017
--       3-     4    269194 ****                                                                   0.0535 0.0028
--       5-     7    136442 **                                                                     0.0677 0.0043
--       8-    11     77914 *                                                                      0.0767 0.0057
--      12-    16     81614 *                                                                      0.0828 0.0072
--      17-    22    171240 ***                                                                    0.0906 0.0099
--      23-    29    626430 ***********                                                            0.1080 0.0183
--      30-    37   2540501 ********************************************                           0.1782 0.0632
--      38-    46   3977690 ********************************************************************** 0.4320 0.2684
--      47-    56   1898392 *********************************                                      0.7804 0.6121
--      57-    67    311243 *****                                                                  0.9268 0.7847
--      68-    79    121725 **                                                                     0.9505 0.8183
--      80-    92    127437 **                                                                     0.9612 0.8366
--      93-   106     98467 *                                                                      0.9727 0.8598
--     107-   121     49168                                                                        0.9811 0.8791
--     122-   137     31832                                                                        0.9854 0.8902
--     138-   154     24469                                                                        0.9882 0.8989
--     155-   172     19471                                                                        0.9904 0.9062
--     173-   191     15298                                                                        0.9922 0.9129
--     192-   211     10053                                                                        0.9935 0.9185
--     212-   232      7275                                                                        0.9944 0.9226
--     233-   254      5960                                                                        0.9950 0.9260
--     255-   277      4676                                                                        0.9956 0.9289
--     278-   301      3771                                                                        0.9960 0.9315
--     302-   326      4987                                                                        0.9963 0.9338
--     327-   352      4907                                                                        0.9968 0.9371
--     353-   379      4304                                                                        0.9972 0.9405
--     380-   407      2777                                                                        0.9976 0.9438
--     408-   436      1895                                                                        0.9978 0.9460
--     437-   466      1962                                                                        0.9980 0.9477
--     467-   497      1709                                                                        0.9982 0.9495
--     498-   529      1701                                                                        0.9983 0.9512
--     530-   562      1692                                                                        0.9985 0.9531
--     563-   596       919                                                                        0.9987 0.9550
--     597-   631       854                                                                        0.9987 0.9561
--     632-   667       566                                                                        0.9988 0.9571
--     668-   704       671                                                                        0.9989 0.9579
--     705-   742       413                                                                        0.9989 0.9589
--     743-   781       222                                                                        0.9990 0.9595
--     782-   821       671                                                                        0.9990 0.9598
--
--           0 (max occurrences)
--   479850855 (total mers, non-unique)
--    11068693 (distinct mers, non-unique)
--           0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'unitigging/0-mercounts/WB_SUP.ms22'.
--
-- OVERLAPPER (normal) (assembly) erate=0.12
--
----------------------------------------
-- Starting command on Tue Nov 16 17:14:07 2021 with 2047.797 GB free disk space

    cd unitigging/1-overlapper
    /home/jon/software/canu-2.2/bin/overlapInCorePartition \
     -S  ../../WB_SUP.seqStore \
     -hl 80000000 \
     -rl 1000000000 \
     -ol 500 \
     -o  ./WB_SUP.partition \
    > ./WB_SUP.partition.err 2>&1

-- Finished on Tue Nov 16 17:14:07 2021 (furiously fast) with 2047.796 GB free disk space
----------------------------------------
--
-- Configured 7 overlapInCore jobs.
-- Finished stage 'utg-overlapConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'utgovl' concurrent execution on Tue Nov 16 17:14:07 2021 with 2047.796 GB free disk space (7 processes; 2 concurrently)

    cd unitigging/1-overlapper
    ./overlap.sh 1 > ./overlap.000001.out 2>&1
    ./overlap.sh 2 > ./overlap.000002.out 2>&1
    ./overlap.sh 3 > ./overlap.000003.out 2>&1
    ./overlap.sh 4 > ./overlap.000004.out 2>&1
    ./overlap.sh 5 > ./overlap.000005.out 2>&1
    ./overlap.sh 6 > ./overlap.000006.out 2>&1
skoren commented 2 years ago

That log looks. OK, the 0s are for trimming chimeric (e.g. accidental 1d^2 reads) but it's OK that they were not picked up in the dataset. I presume this was a completely new run? Did you change the command/parameters from the initial attempt?

jonhultqvist commented 2 years ago

Hi, Yes this was fresh run and with the parameters you suggested added (mhapPipe=false purgeOverlaps=false saveOverlaps=true). I haven't tried the direct assembly option you suggested yet.

/home/jon/software/canu-2.2/bin/canu \
-p WB_SUP -d WB_SUP_canu_no_mhap \
genomeSize=12m maxInputCoverage=100 mhapPipe=false purgeOverlaps=false saveOverlaps=true \
-nanopore /home/jon/scratch/MinION/Giardia/TRIMMED_READS/WB_RBK_210804_SUP.chop.filt.fq.gz

The run finished and produced output. The assembly results are similar to those one would expect. However, canu appears to have only used a subset of the data. The coverage is expected to be around 100x but canu only says ~40x in the main contigs (WB_SUP.contigs.layout.tigInfo). The final output of the run is below. Output of WB_SUP.contigs.layout.tigInfo follows after.

(base) jon@pickle:/media/jon/BackupPlus/MinION_data/ASSEMBLY/canu$ bash WB_SUP_canu_2.2_no_mhap.sh 
-- canu 2.2
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
--   Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
--   Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
--   Genome Res. 2017 May;27(5):722-736.
--   http://doi.org/10.1101/gr.215087.116
-- 
-- Read and contig alignments during correction and consensus use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
-- 
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
-- 
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
-- 
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
-- 
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
-- 
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '11.0.12' (from 'java') without -d64 support.
--
-- WARNING:
-- WARNING:  Failed to run gnuplot using command 'gnuplot'.
-- WARNING:  Plots will be disabled.
-- WARNING:
--
--
-- Detected 16 CPUs and 62 gigabytes of memory on the local machine.
--
-- Local machine mode enabled; grid support not detected or not allowed.
--
--                                (tag)Concurrency
--                         (tag)Threads          |
--                (tag)Memory         |          |
--        (tag)             |         |          |       total usage      algorithm
--        -------  ----------  --------   --------  --------------------  -----------------------------
-- Local: meryl     12.000 GB    4 CPUs x   4 jobs    48.000 GB  16 CPUs  (k-mer counting)
-- Local: hap        8.000 GB    4 CPUs x   4 jobs    32.000 GB  16 CPUs  (read-to-haplotype assignment)
-- Local: cormhap    6.000 GB   16 CPUs x   1 job      6.000 GB  16 CPUs  (overlap detection with mhap)
-- Local: obtovl     4.000 GB    8 CPUs x   2 jobs     8.000 GB  16 CPUs  (overlap detection)
-- Local: utgovl     4.000 GB    8 CPUs x   2 jobs     8.000 GB  16 CPUs  (overlap detection)
-- Local: cor        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (read correction)
-- Local: ovb        4.000 GB    1 CPU  x  15 jobs    60.000 GB  15 CPUs  (overlap store bucketizer)
-- Local: ovs        8.000 GB    1 CPU  x   7 jobs    56.000 GB   7 CPUs  (overlap store sorting)
-- Local: red       15.000 GB    4 CPUs x   4 jobs    60.000 GB  16 CPUs  (read error detection)
-- Local: oea        8.000 GB    1 CPU  x   7 jobs    56.000 GB   7 CPUs  (overlap error adjustment)
-- Local: bat       16.000 GB    4 CPUs x   1 job     16.000 GB   4 CPUs  (contig construction with bogart)
-- Local: cns        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (consensus)
--
-- Found untrimmed raw Nanopore reads in the input files.
--
-- Generating assembly 'WB_SUP' in '/media/jon/BackupPlus/MinION_data/ASSEMBLY/canu/WB_SUP_canu_no_mhap':
--   genomeSize:
--     12000000
--
--   Overlap Generation Limits:
--     corOvlErrorRate 0.3200 ( 32.00%)
--     obtOvlErrorRate 0.1200 ( 12.00%)
--     utgOvlErrorRate 0.1200 ( 12.00%)
--
--   Overlap Processing Limits:
--     corErrorRate    0.3000 ( 30.00%)
--     obtErrorRate    0.1200 ( 12.00%)
--     utgErrorRate    0.1200 ( 12.00%)
--     cnsErrorRate    0.2000 ( 20.00%)
--
--   Stages to run:
--     correct raw reads.
--     trim corrected reads.
--     assemble corrected and trimmed reads.
--
--
-- BEGIN CORRECTION
----------------------------------------
-- Starting command on Tue Nov 16 16:10:37 2021 with 2051.814 GB free disk space

    cd .
    ./WB_SUP.seqStore.sh \
    > ./WB_SUP.seqStore.err 2>&1

-- Finished on Tue Nov 16 16:10:51 2021 (14 seconds) with 2051.484 GB free disk space
----------------------------------------
--
-- In sequence store './WB_SUP.seqStore':
--   Found 119829 reads.
--   Found 1200003459 bases (100 times coverage).
--    Histogram of raw reads:
--    
--    G=1200003459                       sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010        16599      6019    120006185  ||       6509-7479        25078|---------------------------------------------------------------
--    00020        13824     14009    240004191  ||       7480-8450        23815|------------------------------------------------------------
--    00030        12119     23317    360008458  ||       8451-9421        18105|----------------------------------------------
--    00040        10886     33785    480012264  ||       9422-10392       13684|-----------------------------------
--    00050         9917     45352    600009243  ||      10393-11363        9851|-------------------------
--    00060         9116     57980    720006607  ||      11364-12334        7459|-------------------
--    00070         8414     71692    840006018  ||      12335-13305        5562|--------------
--    00080         7780     86530    960006224  ||      13306-14276        4080|-----------
--    00090         7222    102548   1080010087  ||      14277-15247        3064|--------
--    00100         6509    119828   1200003459  ||      15248-16218        2380|------
--    001.000x              119829   1200003459  ||      16219-17189        1776|-----
--                                               ||      17190-18160        1278|----
--                                               ||      18161-19131         924|---
--                                               ||      19132-20102         670|--
--                                               ||      20103-21073         524|--
--                                               ||      21074-22044         383|-
--                                               ||      22045-23015         316|-
--                                               ||      23016-23986         214|-
--                                               ||      23987-24957         161|-
--                                               ||      24958-25928         127|-
--                                               ||      25929-26899          91|-
--                                               ||      26900-27870          78|-
--                                               ||      27871-28841          41|-
--                                               ||      28842-29812          41|-
--                                               ||      29813-30783          34|-
--                                               ||      30784-31754          18|-
--                                               ||      31755-32725          24|-
--                                               ||      32726-33696          12|-
--                                               ||      33697-34667           9|-
--                                               ||      34668-35638          10|-
--                                               ||      35639-36609           2|-
--                                               ||      36610-37580           6|-
--                                               ||      37581-38551           0|
--                                               ||      38552-39522           1|-
--                                               ||      39523-40493           3|-
--                                               ||      40494-41464           0|
--                                               ||      41465-42435           3|-
--                                               ||      42436-43406           0|
--                                               ||      43407-44377           0|
--                                               ||      44378-45348           1|-
--                                               ||      45349-46319           0|
--                                               ||      46320-47290           0|
--                                               ||      47291-48261           1|-
--                                               ||      48262-49232           0|
--                                               ||      49233-50203           0|
--                                               ||      50204-51174           1|-
--                                               ||      51175-52145           0|
--                                               ||      52146-53116           1|-
--                                               ||      53117-54087           0|
--                                               ||      54088-55058           1|-
--    
----------------------------------------
-- Starting command on Tue Nov 16 16:10:51 2021 with 2051.482 GB free disk space

    cd correction/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Tue Nov 16 16:10:51 2021 (like a bat out of hell) with 2051.48 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--        01  2.45 GB       2
--        02  1.54 GB       2
--        04  0.77 GB       2
--        06  0.52 GB       2
--        08  0.39 GB       2
--        12  0.26 GB       2
--
--  For 119829 reads with 1200003459 bases, limit to 12 batches.
--  Will count kmers using 01 jobs, each using 4 GB and 4 threads.
--
-- Finished stage 'merylConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 16:10:51 2021 with 2051.479 GB free disk space (1 processes; 4 concurrently)

    cd correction/0-mercounts
    ./meryl-count.sh 1 > ./meryl-count.000001.out 2>&1

-- Finished on Tue Nov 16 16:12:10 2021 (79 seconds) with 2050.354 GB free disk space
----------------------------------------
-- Found 1 Kmer counting (meryl) outputs.
-- Finished stage 'cor-merylCountCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 16:12:10 2021 with 2050.354 GB free disk space (1 processes; 4 concurrently)

    cd correction/0-mercounts
    ./meryl-process.sh 1 > ./meryl-process.000001.out 2>&1

-- Finished on Tue Nov 16 16:12:14 2021 (4 seconds) with 2051.179 GB free disk space
----------------------------------------
-- Meryl finished successfully.  Kmer frequency histogram:
--
-- WARNING: gnuplot failed.
--
----------------------------------------
--
--  16-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2  30618151 ********************************************************************** 0.5176 0.0607
--       3-     5  14629875 *********************************                                      0.6694 0.0874
--       6-    10   2823746 ******                                                                 0.7835 0.1182
--      11-    17    698332 *                                                                      0.8157 0.1340
--      18-    26    219074                                                                        0.8252 0.1418
--      27-    37    129810                                                                        0.8284 0.1460
--      38-    50    461993 *                                                                      0.8306 0.1504
--      51-    65   2396119 *****                                                                  0.8395 0.1745
--      66-    82   4697091 **********                                                             0.8832 0.3296
--      83-   101   1690516 ***                                                                    0.9614 0.6712
--     102-   122    181367                                                                        0.9870 0.8056
--     123-   145    150944                                                                        0.9898 0.8240
--     146-   170    163395                                                                        0.9924 0.8446
--     171-   197     79326                                                                        0.9951 0.8697
--     198-   226     42001                                                                        0.9964 0.8835
--     227-   257     34262                                                                        0.9971 0.8923
--     258-   290     23468                                                                        0.9977 0.9004
--     291-   325     20345                                                                        0.9981 0.9067
--     326-   362     14938                                                                        0.9984 0.9129
--     363-   401     10268                                                                        0.9987 0.9179
--     402-   442      8177                                                                        0.9989 0.9218
--     443-   485      6279                                                                        0.9990 0.9252
--     486-   530      5630                                                                        0.9991 0.9280
--     531-   577      4776                                                                        0.9992 0.9309
--     578-   626      4506                                                                        0.9993 0.9335
--     627-   677      4262                                                                        0.9994 0.9362
--     678-   730      3877                                                                        0.9994 0.9389
--     731-   785      2623                                                                        0.9995 0.9416
--     786-   842      2198                                                                        0.9995 0.9436
--     843-   901      1975                                                                        0.9996 0.9453
--     902-   962      1952                                                                        0.9996 0.9470
--     963-  1025      1810                                                                        0.9996 0.9488
--    1026-  1090      1671                                                                        0.9997 0.9506
--    1091-  1157      1534                                                                        0.9997 0.9524
--    1158-  1226      1400                                                                        0.9997 0.9541
--    1227-  1297      1256                                                                        0.9997 0.9557
--    1298-  1370      1048                                                                        0.9998 0.9573
--    1371-  1445       972                                                                        0.9998 0.9587
--    1446-  1522       709                                                                        0.9998 0.9600
--    1523-  1601       520                                                                        0.9998 0.9611
--
--           0 (max occurrences)
--  1008990965 (total mers, non-unique)
--    59152920 (distinct mers, non-unique)
--           0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'correction/0-mercounts/WB_SUP.ms16'.
--
-- OVERLAPPER (mhap) (correction)
--
-- Set corMhapSensitivity=low based on read coverage of 100.
--
-- PARAMETERS: hashes=256, minMatches=3, threshold=0.8
--
-- Given 5.4 GB, can fit 16200 reads per block.
-- For 9 blocks, set stride to 2 blocks.
-- Logging partitioning to 'correction/1-overlapper/partitioning.log'.
-- Configured 8 mhap precompute jobs.
-- Configured 17 mhap overlap jobs.
-- Finished stage 'cor-mhapConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Tue Nov 16 16:12:14 2021 with 2051.472 GB free disk space (8 processes; 1 concurrently)

    cd correction/1-overlapper
    ./precompute.sh 1 > ./precompute.000001.out 2>&1
    ./precompute.sh 2 > ./precompute.000002.out 2>&1
    ./precompute.sh 3 > ./precompute.000003.out 2>&1
    ./precompute.sh 4 > ./precompute.000004.out 2>&1
    ./precompute.sh 5 > ./precompute.000005.out 2>&1
    ./precompute.sh 6 > ./precompute.000006.out 2>&1
    ./precompute.sh 7 > ./precompute.000007.out 2>&1
    ./precompute.sh 8 > ./precompute.000008.out 2>&1

-- Finished on Tue Nov 16 16:29:01 2021 (1007 seconds) with 2049.446 GB free disk space
----------------------------------------
-- All 8 mhap precompute jobs finished successfully.
-- Finished stage 'cor-mhapPrecomputeCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Tue Nov 16 16:29:01 2021 with 2049.446 GB free disk space (17 processes; 1 concurrently)

    cd correction/1-overlapper
    ./mhap.sh 1 > ./mhap.000001.out 2>&1
    ./mhap.sh 2 > ./mhap.000002.out 2>&1
    ./mhap.sh 3 > ./mhap.000003.out 2>&1
    ./mhap.sh 4 > ./mhap.000004.out 2>&1
    ./mhap.sh 5 > ./mhap.000005.out 2>&1
    ./mhap.sh 6 > ./mhap.000006.out 2>&1
    ./mhap.sh 7 > ./mhap.000007.out 2>&1
    ./mhap.sh 8 > ./mhap.000008.out 2>&1
    ./mhap.sh 9 > ./mhap.000009.out 2>&1
    ./mhap.sh 10 > ./mhap.000010.out 2>&1
    ./mhap.sh 11 > ./mhap.000011.out 2>&1
    ./mhap.sh 12 > ./mhap.000012.out 2>&1
    ./mhap.sh 13 > ./mhap.000013.out 2>&1
    ./mhap.sh 14 > ./mhap.000014.out 2>&1
    ./mhap.sh 15 > ./mhap.000015.out 2>&1
    ./mhap.sh 16 > ./mhap.000016.out 2>&1
    ./mhap.sh 17 > ./mhap.000017.out 2>&1

-- Finished on Tue Nov 16 16:30:15 2021 (74 seconds) with 2049.002 GB free disk space
----------------------------------------
-- Found 17 mhap overlap output files.
-- Finished stage 'cor-mhapCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 16:30:15 2021 with 2049.001 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/ovStoreConfig \
     -S ../WB_SUP.seqStore \
     -M 4-8 \
     -L ./1-overlapper/ovljob.files \
     -create ./WB_SUP.ovlStore.config \
     > ./WB_SUP.ovlStore.config.txt \
    2> ./WB_SUP.ovlStore.config.err

-- Finished on Tue Nov 16 16:30:15 2021 (furiously fast) with 2049.001 GB free disk space
----------------------------------------
--
-- Creating overlap store correction/WB_SUP.ovlStore using:
--      2 buckets
--      2 slices
--        using at most 1 GB memory each
-- Finished stage 'cor-overlapStoreConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovB' concurrent execution on Tue Nov 16 16:30:15 2021 with 2049 GB free disk space (2 processes; 15 concurrently)

    cd correction/WB_SUP.ovlStore.BUILDING
    ./scripts/1-bucketize.sh 1 > ./logs/1-bucketize.000001.out 2>&1
    ./scripts/1-bucketize.sh 2 > ./logs/1-bucketize.000002.out 2>&1

-- Finished on Tue Nov 16 16:30:16 2021 (one second) with 2048.833 GB free disk space
----------------------------------------
-- Overlap store bucketizer finished.
-- Finished stage 'cor-overlapStoreBucketizerCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovS' concurrent execution on Tue Nov 16 16:30:16 2021 with 2048.833 GB free disk space (2 processes; 7 concurrently)

    cd correction/WB_SUP.ovlStore.BUILDING
    ./scripts/2-sort.sh 1 > ./logs/2-sort.000001.out 2>&1
    ./scripts/2-sort.sh 2 > ./logs/2-sort.000002.out 2>&1

-- Finished on Tue Nov 16 16:30:17 2021 (one second) with 2048.64 GB free disk space
----------------------------------------
-- Overlap store sorter finished.
-- Finished stage 'cor-overlapStoreSorterCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 16:30:17 2021 with 2048.64 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/ovStoreIndexer \
      -O  ./WB_SUP.ovlStore.BUILDING \
      -S ../WB_SUP.seqStore \
      -C  ./WB_SUP.ovlStore.config \
      -delete \
    > ./WB_SUP.ovlStore.BUILDING.index.err 2>&1

-- Finished on Tue Nov 16 16:30:17 2021 (lickety-split) with 2048.81 GB free disk space
----------------------------------------
-- Overlap store indexer finished.
-- Checking store.
----------------------------------------
-- Starting command on Tue Nov 16 16:30:17 2021 with 2048.81 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/ovStoreDump \
     -S ../WB_SUP.seqStore \
     -O  ./WB_SUP.ovlStore \
     -counts \
     > ./WB_SUP.ovlStore/counts.dat 2> ./WB_SUP.ovlStore/counts.err

-- Finished on Tue Nov 16 16:30:18 2021 (one second) with 2048.808 GB free disk space
----------------------------------------
--
-- Overlap store 'correction/WB_SUP.ovlStore' successfully constructed.
-- Found 9588688 overlaps for 119798 reads; 31 reads have no overlaps.
--
-- Finished stage 'cor-createOverlapStore', reset canuIteration.
-- Set corMinCoverage=4 based on read coverage of 100.
-- Computing correction layouts.
--   Local  filter coverage   80
--   Global filter coverage   40
----------------------------------------
-- Starting command on Tue Nov 16 16:30:18 2021 with 2048.81 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/generateCorrectionLayouts \
      -S ../WB_SUP.seqStore \
      -O  ./WB_SUP.ovlStore \
      -C  ./WB_SUP.corStore.WORKING \
      -eC 80 \
      -xC 40 \
    > ./WB_SUP.corStore.err 2>&1

-- Finished on Tue Nov 16 16:30:18 2021 (lickety-split) with 2048.651 GB free disk space
----------------------------------------
-- Finished stage 'cor-buildCorrectionLayoutsConfigure', reset canuIteration.
-- Computing correction layouts.
----------------------------------------
-- Starting command on Tue Nov 16 16:30:18 2021 with 2048.652 GB free disk space

    cd correction/2-correction
    /home/jon/software/canu-2.2/bin/filterCorrectionLayouts \
      -S  ../../WB_SUP.seqStore \
      -C     ../WB_SUP.corStore \
      -R      ./WB_SUP.readsToCorrect.WORKING \
      -cc 4 \
      -cl 1000 \
      -g  12000000 \
      -c  40 \
    > ./WB_SUP.readsToCorrect.err 2>&1

-- Finished on Tue Nov 16 16:30:19 2021 (one second) with 2048.643 GB free disk space
----------------------------------------
--                             original      original
--                            raw reads     raw reads
--   category                w/overlaps  w/o/overlaps
--   -------------------- ------------- -------------
--   Number of Reads             119043           786
--   Number of Bases         1193749376       5977470
--   Coverage                    99.479         0.498
--   Median                        9020          7519
--   Mean                         10027          7604
--   N50                           9933          7713
--   Minimum                       6509             0
--   Maximum                      55030         12700
--   
--                                        --------corrected---------  ----------rescued----------
--                             evidence                     expected                     expected
--   category                     reads            raw     corrected            raw     corrected
--   -------------------- -------------  ------------- -------------  ------------- -------------
--   Number of Reads             118379          34340         34340            813           813
--   Number of Bases         1186819479      483565899     480003777        7432964       3969296
--   Coverage                    98.902         40.297        40.000          0.619         0.331
--   Median                        9016          13060         12969           8384          4661
--   Mean                         10025          14081         13977           9142          4882
--   N50                           9929          13719         13607           8974          6575
--   Minimum                       6509          10678         10676           6610          1004
--   Maximum                      55030          52653         52640          35383         10640
--   
--                        --------uncorrected--------
--                                           expected
--   category                       raw     corrected
--   -------------------- ------------- -------------
--   Number of Reads              84676         84676
--   Number of Bases          708727983     677804341
--   Coverage                    59.061        56.484
--   Median                        8167          8025
--   Mean                          8369          8004
--   N50                           8399          8305
--   Minimum                          0             0
--   Maximum                      55030         54876
--   
--   Maximum Memory          1032441278
-- Finished stage 'cor-filterCorrectionLayouts', reset canuIteration.
--
-- Correction jobs estimated to need at most 0.961 GB for computation.
-- Correction jobs will request 6 GB each.
--
-- Local: cor        6.000 GB    4 CPUs x   4 jobs    24.000 GB  16 CPUs  (read correction)
--
--
-- Configuring correction jobs:
--   Reads estimated to need at most 0.961 GB for computation.
--   Jobs will request 6 GB each.
----------------------------------------
-- Starting command on Tue Nov 16 16:30:19 2021 with 2048.642 GB free disk space

    cd correction/2-correction
    ./correctReadsPartition.sh \
    > ./correctReadsPartition.err 2>&1

-- Finished on Tue Nov 16 16:30:19 2021 (in the blink of an eye) with 2048.642 GB free disk space
----------------------------------------
-- Finished stage 'cor-generateCorrectedReadsConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cor' concurrent execution on Tue Nov 16 16:30:19 2021 with 2048.642 GB free disk space (4 processes; 4 concurrently)

    cd correction/2-correction
    ./correctReads.sh 1 > ./correctReads.000001.out 2>&1
    ./correctReads.sh 2 > ./correctReads.000002.out 2>&1
    ./correctReads.sh 3 > ./correctReads.000003.out 2>&1
    ./correctReads.sh 4 > ./correctReads.000004.out 2>&1

-- Finished on Tue Nov 16 16:49:55 2021 (1176 seconds) with 2047.676 GB free disk space
----------------------------------------
-- Found 4 read correction output files.
-- Finished stage 'cor-generateCorrectedReadsCheck', reset canuIteration.
-- Found 4 read correction output files.
-- Finished stage 'cor-generateCorrectedReadsCheck', reset canuIteration.
--
-- Loading corrected reads into corStore and seqStore.
----------------------------------------
-- Starting command on Tue Nov 16 16:49:55 2021 with 2047.676 GB free disk space

    cd correction
    /home/jon/software/canu-2.2/bin/loadCorrectedReads \
      -S ../WB_SUP.seqStore \
      -C ./WB_SUP.corStore \
      -L ./2-correction/corjob.files \
    >  ./WB_SUP.loadCorrectedReads.log \
    2> ./WB_SUP.loadCorrectedReads.err

-- Finished on Tue Nov 16 16:49:59 2021 (4 seconds) with 2047.435 GB free disk space
----------------------------------------
--
-- In sequence store './WB_SUP.seqStore':
--   Found 35115 reads.
--   Found 484668327 bases (40.38 times coverage).
--    Histogram of corrected reads:
--    
--    G=484668327                        sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010        19954      2081     48473710  ||       1016-2057          109|-
--    00020        17226      4718     96947799  ||       2058-3099          112|-
--    00030        15678      7675    145405887  ||       3100-4141          117|-
--    00040        14528     10891    193877888  ||       4142-5183           83|-
--    00050        13615     14340    242336543  ||       5184-6225           90|-
--    00060        12848     18009    290805282  ||       6226-7267          145|--
--    00070        12218     21879    339270602  ||       7268-8309          118|-
--    00080        11646     25945    387736371  ||       8310-9351          110|-
--    00090        11133     30202    436210644  ||       9352-10393         195|--
--    00100         1016     35114    484668327  ||      10394-11435        6395|-------------------------------------------------------
--    001.000x               35115    484668327  ||      11436-12477        7442|---------------------------------------------------------------
--                                               ||      12478-13519        5437|-----------------------------------------------
--                                               ||      13520-14561        3989|----------------------------------
--                                               ||      14562-15603        2925|-------------------------
--                                               ||      15604-16645        2165|-------------------
--                                               ||      16646-17687        1605|--------------
--                                               ||      17688-18729        1109|----------
--                                               ||      18730-19771         760|-------
--                                               ||      19772-20813         585|-----
--                                               ||      20814-21855         445|----
--                                               ||      21856-22897         332|---
--                                               ||      22898-23939         216|--
--                                               ||      23940-24981         175|--
--                                               ||      24982-26023         127|--
--                                               ||      26024-27065          92|-
--                                               ||      27066-28107          67|-
--                                               ||      28108-29149          42|-
--                                               ||      29150-30191          28|-
--                                               ||      30192-31233          31|-
--                                               ||      31234-32275          20|-
--                                               ||      32276-33317          12|-
--                                               ||      33318-34359          12|-
--                                               ||      34360-35401           5|-
--                                               ||      35402-36443           4|-
--                                               ||      36444-37485           4|-
--                                               ||      37486-38527           1|-
--                                               ||      38528-39569           2|-
--                                               ||      39570-40611           4|-
--                                               ||      40612-41653           1|-
--                                               ||      41654-42695           2|-
--                                               ||      42696-43737           0|
--                                               ||      43738-44779           0|
--                                               ||      44780-45821           1|-
--                                               ||      45822-46863           0|
--                                               ||      46864-47905           0|
--                                               ||      47906-48947           0|
--                                               ||      48948-49989           0|
--                                               ||      49990-51031           0|
--                                               ||      51032-52073           0|
--                                               ||      52074-53115           1|-
--    
--
-- Purging correctReads output after loading into stores.
-- Purged 4 .cns outputs.
-- Purged 8 .out job log outputs.
--
-- No corrected reads generated, overlaps used for correction saved.
-- Finished stage 'cor-loadCorrectedReads', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 16:49:59 2021 with 2048.401 GB free disk space

    cd .
    /home/jon/software/canu-2.2/bin/sqStoreDumpFASTQ \
      -corrected \
      -S ./WB_SUP.seqStore \
      -o ./WB_SUP.correctedReads.gz \
      -fasta \
      -nolibname \
    > WB_SUP.correctedReads.fasta.err 2>&1

-- Finished on Tue Nov 16 16:50:05 2021 (6 seconds) with 2048.251 GB free disk space
----------------------------------------
--
-- Corrected reads saved in 'WB_SUP.correctedReads.fasta.gz'.
-- Finished stage 'cor-dumpCorrectedReads', reset canuIteration.
--
-- BEGIN TRIMMING
----------------------------------------
-- Starting command on Tue Nov 16 16:50:05 2021 with 2048.25 GB free disk space

    cd trimming/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Tue Nov 16 16:50:05 2021 (in the blink of an eye) with 2048.249 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--        01  1.93 GB       2
--        02  0.97 GB       2
--        04  0.48 GB       2
--
--  For 35115 reads with 484668327 bases, limit to 4 batches.
--  Will count kmers using 01 jobs, each using 3 GB and 4 threads.
--
-- Finished stage 'merylConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 16:50:05 2021 with 2048.249 GB free disk space (1 processes; 4 concurrently)

    cd trimming/0-mercounts
    ./meryl-count.sh 1 > ./meryl-count.000001.out 2>&1

-- Finished on Tue Nov 16 16:50:29 2021 (24 seconds) with 2048.134 GB free disk space
----------------------------------------
-- Found 1 Kmer counting (meryl) outputs.
-- Finished stage 'obt-merylCountCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 16:50:29 2021 with 2048.134 GB free disk space (1 processes; 4 concurrently)

    cd trimming/0-mercounts
    ./meryl-process.sh 1 > ./meryl-process.000001.out 2>&1

-- Finished on Tue Nov 16 16:50:30 2021 (one second) with 2048.152 GB free disk space
----------------------------------------
-- Meryl finished successfully.  Kmer frequency histogram:
--
-- WARNING: gnuplot failed.
--
----------------------------------------
--
--  22-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2    425271 *******                                                                0.0384 0.0018
--       3-     4    272688 ****                                                                   0.0543 0.0029
--       5-     7    137931 **                                                                     0.0687 0.0043
--       8-    11     78052 *                                                                      0.0777 0.0058
--      12-    16     81181 *                                                                      0.0839 0.0072
--      17-    22    170305 **                                                                     0.0916 0.0099
--      23-    29    621191 **********                                                             0.1090 0.0183
--      30-    37   2530747 ********************************************                           0.1784 0.0628
--      38-    46   3979530 ********************************************************************** 0.4313 0.2673
--      47-    56   1912434 *********************************                                      0.7795 0.6107
--      57-    67    312569 *****                                                                  0.9267 0.7842
--      68-    79    121255 **                                                                     0.9505 0.8180
--      80-    92    127137 **                                                                     0.9611 0.8362
--      93-   106     99408 *                                                                      0.9727 0.8593
--     107-   121     49353                                                                        0.9811 0.8787
--     122-   137     31706                                                                        0.9854 0.8899
--     138-   154     24575                                                                        0.9882 0.8985
--     155-   172     19431                                                                        0.9904 0.9058
--     173-   191     15457                                                                        0.9921 0.9125
--     192-   211     10088                                                                        0.9935 0.9181
--     212-   232      7241                                                                        0.9944 0.9222
--     233-   254      5846                                                                        0.9950 0.9256
--     255-   277      4759                                                                        0.9956 0.9286
--     278-   301      3781                                                                        0.9960 0.9311
--     302-   326      4948                                                                        0.9963 0.9334
--     327-   352      4768                                                                        0.9968 0.9367
--     353-   379      4175                                                                        0.9972 0.9400
--     380-   407      3084                                                                        0.9976 0.9432
--     408-   436      1891                                                                        0.9978 0.9457
--     437-   466      2010                                                                        0.9980 0.9473
--     467-   497      1768                                                                        0.9982 0.9492
--     498-   529      1631                                                                        0.9983 0.9510
--     530-   562      1770                                                                        0.9985 0.9528
--     563-   596       882                                                                        0.9986 0.9547
--     597-   631       905                                                                        0.9987 0.9558
--     632-   667       469                                                                        0.9988 0.9569
--     668-   704       778                                                                        0.9988 0.9576
--     705-   742       431                                                                        0.9989 0.9587
--     743-   781       246                                                                        0.9990 0.9593
--     782-   821       650                                                                        0.9990 0.9597
--
--           0 (max occurrences)
--   480552570 (total mers, non-unique)
--    11083010 (distinct mers, non-unique)
--           0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'trimming/0-mercounts/WB_SUP.ms22'.
--
-- OVERLAPPER (normal) (trimming) erate=0.12
--
----------------------------------------
-- Starting command on Tue Nov 16 16:50:30 2021 with 2048.247 GB free disk space

    cd trimming/1-overlapper
    /home/jon/software/canu-2.2/bin/overlapInCorePartition \
     -S  ../../WB_SUP.seqStore \
     -hl 80000000 \
     -rl 1000000000 \
     -ol 500 \
     -o  ./WB_SUP.partition \
    > ./WB_SUP.partition.err 2>&1

-- Finished on Tue Nov 16 16:50:30 2021 (in the blink of an eye) with 2048.246 GB free disk space
----------------------------------------
--
-- Configured 7 overlapInCore jobs.
-- Finished stage 'obt-overlapConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'obtovl' concurrent execution on Tue Nov 16 16:50:30 2021 with 2048.245 GB free disk space (7 processes; 2 concurrently)

    cd trimming/1-overlapper
    ./overlap.sh 1 > ./overlap.000001.out 2>&1
    ./overlap.sh 2 > ./overlap.000002.out 2>&1
    ./overlap.sh 3 > ./overlap.000003.out 2>&1
    ./overlap.sh 4 > ./overlap.000004.out 2>&1
    ./overlap.sh 5 > ./overlap.000005.out 2>&1
    ./overlap.sh 6 > ./overlap.000006.out 2>&1
    ./overlap.sh 7 > ./overlap.000007.out 2>&1

-- Finished on Tue Nov 16 17:13:31 2021 (1381 seconds) with 2048.153 GB free disk space
----------------------------------------
-- Found 7 overlapInCore output files.
--
-- overlapInCore compute 'trimming/1-overlapper':
--   kmer hits
--     with no overlap         16407799  1.28571429 +- 2058715.237
--     with an overlap          4515620  .571428571 +- 548925.22
--
--   overlaps                   4515309  .142857143 +- 548875.39
--     contained                      0           0 +- 0
--     dovetail                       0           0 +- 0
--
--   overlaps rejected
--     multiple per pair              0           0 +- 0
--     bad short window               0           0 +- 0
--     bad long window                0           0 +- 0
-- Finished stage 'obt-overlapCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:31 2021 with 2048.152 GB free disk space

    cd trimming
    /home/jon/software/canu-2.2/bin/ovStoreConfig \
     -S ../WB_SUP.seqStore \
     -M 3 \
     -L ./1-overlapper/ovljob.files \
     -create ./WB_SUP.ovlStore.config \
     > ./WB_SUP.ovlStore.config.txt \
    2> ./WB_SUP.ovlStore.config.err

-- Finished on Tue Nov 16 17:13:31 2021 (furiously fast) with 2048.152 GB free disk space
----------------------------------------
--
-- Creating overlap store trimming/WB_SUP.ovlStore using:
--      2 buckets
--      2 slices
--        using at most 1 GB memory each
-- Finished stage 'obt-overlapStoreConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovB' concurrent execution on Tue Nov 16 17:13:31 2021 with 2048.151 GB free disk space (2 processes; 15 concurrently)

    cd trimming/WB_SUP.ovlStore.BUILDING
    ./scripts/1-bucketize.sh 1 > ./logs/1-bucketize.000001.out 2>&1
    ./scripts/1-bucketize.sh 2 > ./logs/1-bucketize.000002.out 2>&1

-- Finished on Tue Nov 16 17:13:32 2021 (one second) with 2047.994 GB free disk space
----------------------------------------
-- Overlap store bucketizer finished.
-- Finished stage 'obt-overlapStoreBucketizerCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovS' concurrent execution on Tue Nov 16 17:13:32 2021 with 2047.994 GB free disk space (2 processes; 7 concurrently)

    cd trimming/WB_SUP.ovlStore.BUILDING
    ./scripts/2-sort.sh 1 > ./logs/2-sort.000001.out 2>&1
    ./scripts/2-sort.sh 2 > ./logs/2-sort.000002.out 2>&1

-- Finished on Tue Nov 16 17:13:33 2021 (one second) with 2047.811 GB free disk space
----------------------------------------
-- Overlap store sorter finished.
-- Finished stage 'obt-overlapStoreSorterCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:33 2021 with 2047.811 GB free disk space

    cd trimming
    /home/jon/software/canu-2.2/bin/ovStoreIndexer \
      -O  ./WB_SUP.ovlStore.BUILDING \
      -S ../WB_SUP.seqStore \
      -C  ./WB_SUP.ovlStore.config \
      -delete \
    > ./WB_SUP.ovlStore.BUILDING.index.err 2>&1

-- Finished on Tue Nov 16 17:13:33 2021 (fast as lightning) with 2047.971 GB free disk space
----------------------------------------
-- Overlap store indexer finished.
-- Checking store.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:33 2021 with 2047.971 GB free disk space

    cd trimming
    /home/jon/software/canu-2.2/bin/ovStoreDump \
     -S ../WB_SUP.seqStore \
     -O  ./WB_SUP.ovlStore \
     -counts \
     > ./WB_SUP.ovlStore/counts.dat 2> ./WB_SUP.ovlStore/counts.err

-- Finished on Tue Nov 16 17:13:33 2021 (furiously fast) with 2047.97 GB free disk space
----------------------------------------
--
-- Overlap store 'trimming/WB_SUP.ovlStore' successfully constructed.
-- Found 9031240 overlaps for 35114 reads; 84715 reads have no overlaps.
--
-- Finished stage 'obt-createOverlapStore', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:34 2021 with 2047.971 GB free disk space

    cd trimming/3-overlapbasedtrimming
    /home/jon/software/canu-2.2/bin/trimReads \
      -S  ../../WB_SUP.seqStore \
      -O  ../WB_SUP.ovlStore \
      -Co ./WB_SUP.1.trimReads.clear \
      -e  0.12 \
      -minlength 1000 \
      -ol 500 \
      -oc 2 \
      -o  ./WB_SUP.1.trimReads \
    >     ./WB_SUP.1.trimReads.err 2>&1

-- Finished on Tue Nov 16 17:13:35 2021 (one second) with 2047.961 GB free disk space
----------------------------------------
--  PARAMETERS:
--  ----------
--     1000    (reads trimmed below this many bases are deleted)
--   0.1200    (use overlaps at or below this fraction error)
--      500    (break region if overlap is less than this long, for 'largest covered' algorithm)
--        2    (break region if overlap coverage is less than this many reads, for 'largest covered' algorithm)
--  
--  INPUT READS:
--  -----------
--  119829 reads    484668327 bases (reads processed)
--       0 reads            0 bases (reads not processed, previously deleted)
--       0 reads            0 bases (reads not processed, in a library where trimming isn't allowed)
--  
--  OUTPUT READS:
--  ------------
--    9021 reads    125442180 bases (trimmed reads output)
--   26091 reads    358326849 bases (reads with no change, kept as is)
--   84715 reads         1096 bases (reads with no overlaps, deleted)
--       2 reads         2766 bases (reads with short trimmed length, deleted)
--  
--  TRIMMING DETAILS:
--  ----------------
--    1202 reads       353933 bases (bases trimmed from the 5' end of a read)
--    8095 reads       541503 bases (bases trimmed from the 3' end of a read)
-- Finished stage 'obt-trimReads', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:35 2021 with 2047.962 GB free disk space

    cd trimming/3-overlapbasedtrimming
    /home/jon/software/canu-2.2/bin/splitReads \
      -S  ../../WB_SUP.seqStore \
      -O  ../WB_SUP.ovlStore \
      -Ci ./WB_SUP.1.trimReads.clear \
      -Co ./WB_SUP.2.splitReads.clear \
      -e  0.12 \
      -minlength 1000 \
      -o  ./WB_SUP.2.splitReads \
    >     ./WB_SUP.2.splitReads.err 2>&1

-- Finished on Tue Nov 16 17:13:36 2021 (one second) with 2047.96 GB free disk space
----------------------------------------
--  PARAMETERS:
--  ----------
--     1000    (reads trimmed below this many bases are deleted)
--   0.1200    (use overlaps at or below this fraction error)
--  INPUT READS:
--  -----------
--   35112 reads    484664465 bases (reads processed)
--   84717 reads         3862 bases (reads not processed, previously deleted)
--       0 reads            0 bases (reads not processed, in a library where trimming isn't allowed)
--  
--  PROCESSED:
--  --------
--       0 reads            0 bases (no overlaps)
--       0 reads            0 bases (no coverage after adjusting for trimming done already)
--       0 reads            0 bases (processed for chimera)
--       0 reads            0 bases (processed for spur)
--   35112 reads    484664465 bases (processed for subreads)
--  
--  READS WITH SIGNALS:
--  ------------------
--       0 reads            0 signals (number of 5' spur signal)
--       0 reads            0 signals (number of 3' spur signal)
--       0 reads            0 signals (number of chimera signal)
--       0 reads            0 signals (number of subread signal)
--  
--  SIGNALS:
--  -------
--       0 reads            0 bases (size of 5' spur signal)
--       0 reads            0 bases (size of 3' spur signal)
--       0 reads            0 bases (size of chimera signal)
--       0 reads            0 bases (size of subread signal)
--  
--  TRIMMING:
--  --------
--       0 reads            0 bases (trimmed from the 5' end of the read)
--       0 reads            0 bases (trimmed from the 3' end of the read)
-- Finished stage 'obt-splitReads', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:36 2021 with 2047.96 GB free disk space

    cd trimming/3-overlapbasedtrimming
    /home/jon/software/canu-2.2/bin/loadTrimmedReads \
      -S ../../WB_SUP.seqStore \
      -c ./WB_SUP.2.splitReads.clear \
    > ./WB_SUP.loadTrimmedReads.err 2>&1

-- Finished on Tue Nov 16 17:13:38 2021 (2 seconds) with 2047.948 GB free disk space
----------------------------------------
--
-- In sequence store './WB_SUP.seqStore':
--   Found 35112 reads.
--   Found 483769029 bases (40.31 times coverage).
--    Histogram of corrected-trimmed reads:
--    
--    G=483769029                        sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010        19931      2079     48387893  ||       1016-2057          107|-
--    00020        17214      4713     96767932  ||       2058-3099          117|-
--    00030        15662      7667    145140244  ||       3100-4141          114|-
--    00040        14510     10880    193522007  ||       4142-5183           90|-
--    00050        13601     14326    241887220  ||       5184-6225           95|-
--    00060        12833     17992    290262083  ||       6226-7267          151|--
--    00070        12202     21860    338640074  ||       7268-8309          141|--
--    00080        11632     25924    387015451  ||       8310-9351          142|--
--    00090        11122     30178    435398442  ||       9352-10393         248|---
--    00100         1016     35111    483769029  ||      10394-11435        6401|-------------------------------------------------------
--    001.000x               35112    483769029  ||      11436-12477        7426|---------------------------------------------------------------
--                                               ||      12478-13519        5396|----------------------------------------------
--                                               ||      13520-14561        3975|----------------------------------
--                                               ||      14562-15603        2900|-------------------------
--                                               ||      15604-16645        2155|-------------------
--                                               ||      16646-17687        1599|--------------
--                                               ||      17688-18729        1100|----------
--                                               ||      18730-19771         759|-------
--                                               ||      19772-20813         589|-----
--                                               ||      20814-21855         435|----
--                                               ||      21856-22897         327|---
--                                               ||      22898-23939         216|--
--                                               ||      23940-24981         176|--
--                                               ||      24982-26023         126|--
--                                               ||      26024-27065          94|-
--                                               ||      27066-28107          64|-
--                                               ||      28108-29149          42|-
--                                               ||      29150-30191          28|-
--                                               ||      30192-31233          32|-
--                                               ||      31234-32275          18|-
--                                               ||      32276-33317          12|-
--                                               ||      33318-34359          12|-
--                                               ||      34360-35401           5|-
--                                               ||      35402-36443           4|-
--                                               ||      36444-37485           4|-
--                                               ||      37486-38527           1|-
--                                               ||      38528-39569           2|-
--                                               ||      39570-40611           4|-
--                                               ||      40612-41653           1|-
--                                               ||      41654-42695           2|-
--                                               ||      42696-43737           0|
--                                               ||      43738-44779           0|
--                                               ||      44780-45821           1|-
--                                               ||      45822-46863           0|
--                                               ||      46864-47905           0|
--                                               ||      47906-48947           0|
--                                               ||      48948-49989           0|
--                                               ||      49990-51031           0|
--                                               ||      51032-52073           0|
--                                               ||      52074-53115           1|-
--    
--
-- No trimmed reads generated, overlaps used for trimming saved.
-- Finished stage 'obt-dumpReads', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 17:13:38 2021 with 2047.951 GB free disk space

    cd .
    /home/jon/software/canu-2.2/bin/sqStoreDumpFASTQ \
      -trimmed \
      -S ./WB_SUP.seqStore \
      -o ./WB_SUP.trimmedReads.gz \
      -fasta \
      -trimmed -normal -nolibname \
    > ./WB_SUP.trimmedReads.fasta.err 2>&1

-- Finished on Tue Nov 16 17:13:44 2021 (6 seconds) with 2047.801 GB free disk space
----------------------------------------
--
-- Trimmed reads saved in 'WB_SUP.trimmedReads.fasta.gz'.
-- Finished stage 'cor-dumpTrimmedReads', reset canuIteration.
--
-- BEGIN ASSEMBLY
----------------------------------------
-- Starting command on Tue Nov 16 17:13:44 2021 with 2047.801 GB free disk space

    cd unitigging/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Tue Nov 16 17:13:44 2021 (like a bat out of hell) with 2047.8 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--        01  1.93 GB       2
--        02  0.97 GB       2
--        04  0.48 GB       2
--
--  For 35112 reads with 483769029 bases, limit to 4 batches.
--  Will count kmers using 01 jobs, each using 3 GB and 4 threads.
--
-- Finished stage 'merylConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 17:13:44 2021 with 2047.799 GB free disk space (1 processes; 4 concurrently)

    cd unitigging/0-mercounts
    ./meryl-count.sh 1 > ./meryl-count.000001.out 2>&1

-- Finished on Tue Nov 16 17:14:07 2021 (23 seconds) with 2047.685 GB free disk space
----------------------------------------
-- Found 1 Kmer counting (meryl) outputs.
-- Finished stage 'utg-merylCountCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Tue Nov 16 17:14:07 2021 with 2047.685 GB free disk space (1 processes; 4 concurrently)

    cd unitigging/0-mercounts
    ./meryl-process.sh 1 > ./meryl-process.000001.out 2>&1

-- Finished on Tue Nov 16 17:14:07 2021 (in the blink of an eye) with 2047.703 GB free disk space
----------------------------------------
-- Meryl finished successfully.  Kmer frequency histogram:
--
-- WARNING: gnuplot failed.
--
----------------------------------------
--
--  22-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2    417553 *******                                                                0.0377 0.0017
--       3-     4    269194 ****                                                                   0.0535 0.0028
--       5-     7    136442 **                                                                     0.0677 0.0043
--       8-    11     77914 *                                                                      0.0767 0.0057
--      12-    16     81614 *                                                                      0.0828 0.0072
--      17-    22    171240 ***                                                                    0.0906 0.0099
--      23-    29    626430 ***********                                                            0.1080 0.0183
--      30-    37   2540501 ********************************************                           0.1782 0.0632
--      38-    46   3977690 ********************************************************************** 0.4320 0.2684
--      47-    56   1898392 *********************************                                      0.7804 0.6121
--      57-    67    311243 *****                                                                  0.9268 0.7847
--      68-    79    121725 **                                                                     0.9505 0.8183
--      80-    92    127437 **                                                                     0.9612 0.8366
--      93-   106     98467 *                                                                      0.9727 0.8598
--     107-   121     49168                                                                        0.9811 0.8791
--     122-   137     31832                                                                        0.9854 0.8902
--     138-   154     24469                                                                        0.9882 0.8989
--     155-   172     19471                                                                        0.9904 0.9062
--     173-   191     15298                                                                        0.9922 0.9129
--     192-   211     10053                                                                        0.9935 0.9185
--     212-   232      7275                                                                        0.9944 0.9226
--     233-   254      5960                                                                        0.9950 0.9260
--     255-   277      4676                                                                        0.9956 0.9289
--     278-   301      3771                                                                        0.9960 0.9315
--     302-   326      4987                                                                        0.9963 0.9338
--     327-   352      4907                                                                        0.9968 0.9371
--     353-   379      4304                                                                        0.9972 0.9405
--     380-   407      2777                                                                        0.9976 0.9438
--     408-   436      1895                                                                        0.9978 0.9460
--     437-   466      1962                                                                        0.9980 0.9477
--     467-   497      1709                                                                        0.9982 0.9495
--     498-   529      1701                                                                        0.9983 0.9512
--     530-   562      1692                                                                        0.9985 0.9531
--     563-   596       919                                                                        0.9987 0.9550
--     597-   631       854                                                                        0.9987 0.9561
--     632-   667       566                                                                        0.9988 0.9571
--     668-   704       671                                                                        0.9989 0.9579
--     705-   742       413                                                                        0.9989 0.9589
--     743-   781       222                                                                        0.9990 0.9595
--     782-   821       671                                                                        0.9990 0.9598
--
--           0 (max occurrences)
--   479850855 (total mers, non-unique)
--    11068693 (distinct mers, non-unique)
--           0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'unitigging/0-mercounts/WB_SUP.ms22'.
--
-- OVERLAPPER (normal) (assembly) erate=0.12
--
----------------------------------------
-- Starting command on Tue Nov 16 17:14:07 2021 with 2047.797 GB free disk space

    cd unitigging/1-overlapper
    /home/jon/software/canu-2.2/bin/overlapInCorePartition \
     -S  ../../WB_SUP.seqStore \
     -hl 80000000 \
     -rl 1000000000 \
     -ol 500 \
     -o  ./WB_SUP.partition \
    > ./WB_SUP.partition.err 2>&1

-- Finished on Tue Nov 16 17:14:07 2021 (furiously fast) with 2047.796 GB free disk space
----------------------------------------
--
-- Configured 7 overlapInCore jobs.
-- Finished stage 'utg-overlapConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'utgovl' concurrent execution on Tue Nov 16 17:14:07 2021 with 2047.796 GB free disk space (7 processes; 2 concurrently)

    cd unitigging/1-overlapper
    ./overlap.sh 1 > ./overlap.000001.out 2>&1
    ./overlap.sh 2 > ./overlap.000002.out 2>&1
    ./overlap.sh 3 > ./overlap.000003.out 2>&1
    ./overlap.sh 4 > ./overlap.000004.out 2>&1
    ./overlap.sh 5 > ./overlap.000005.out 2>&1
    ./overlap.sh 6 > ./overlap.000006.out 2>&1
    ./overlap.sh 7 > ./overlap.000007.out 2>&1

-- Finished on Tue Nov 16 18:31:57 2021 (4670 seconds) with 2047.752 GB free disk space
----------------------------------------
-- Found 7 overlapInCore output files.
--
-- overlapInCore compute 'unitigging/1-overlapper':
--   kmer hits
--     with no overlap         18960884  7.71428571 +- 2450123.222
--     with an overlap          1902153  .142857143 +- 221613.684
--
--   overlaps                   1902153  .142857143 +- 221613.684
--     contained                 233060  2857142857 +- 27920.712
--     dovetail                 1669093  .857142857 +- 193792.503
--
--   overlaps rejected
--     multiple per pair              0           0 +- 0
--     bad short window               0           0 +- 0
--     bad long window                0           0 +- 0
-- Finished stage 'utg-overlapCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 18:31:57 2021 with 2047.752 GB free disk space

    cd unitigging
    /home/jon/software/canu-2.2/bin/ovStoreConfig \
     -S ../WB_SUP.seqStore \
     -M 3 \
     -L ./1-overlapper/ovljob.files \
     -create ./WB_SUP.ovlStore.config \
     > ./WB_SUP.ovlStore.config.txt \
    2> ./WB_SUP.ovlStore.config.err

-- Finished on Tue Nov 16 18:31:57 2021 (like a bat out of hell) with 2047.751 GB free disk space
----------------------------------------
--
-- Creating overlap store unitigging/WB_SUP.ovlStore using:
--      2 buckets
--      2 slices
--        using at most 1 GB memory each
-- Finished stage 'utg-overlapStoreConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovB' concurrent execution on Tue Nov 16 18:31:57 2021 with 2047.75 GB free disk space (2 processes; 15 concurrently)

    cd unitigging/WB_SUP.ovlStore.BUILDING
    ./scripts/1-bucketize.sh 1 > ./logs/1-bucketize.000001.out 2>&1
    ./scripts/1-bucketize.sh 2 > ./logs/1-bucketize.000002.out 2>&1

-- Finished on Tue Nov 16 18:31:58 2021 (one second) with 2047.682 GB free disk space
----------------------------------------
-- Overlap store bucketizer finished.
-- Finished stage 'utg-overlapStoreBucketizerCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'ovS' concurrent execution on Tue Nov 16 18:31:58 2021 with 2047.682 GB free disk space (2 processes; 7 concurrently)

    cd unitigging/WB_SUP.ovlStore.BUILDING
    ./scripts/2-sort.sh 1 > ./logs/2-sort.000001.out 2>&1
    ./scripts/2-sort.sh 2 > ./logs/2-sort.000002.out 2>&1

-- Finished on Tue Nov 16 18:31:59 2021 (one second) with 2047.597 GB free disk space
----------------------------------------
-- Overlap store sorter finished.
-- Finished stage 'utg-overlapStoreSorterCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 18:31:59 2021 with 2047.597 GB free disk space

    cd unitigging
    /home/jon/software/canu-2.2/bin/ovStoreIndexer \
      -O  ./WB_SUP.ovlStore.BUILDING \
      -S ../WB_SUP.seqStore \
      -C  ./WB_SUP.ovlStore.config \
      -delete \
    > ./WB_SUP.ovlStore.BUILDING.index.err 2>&1

-- Finished on Tue Nov 16 18:31:59 2021 (like a bat out of hell) with 2047.668 GB free disk space
----------------------------------------
-- Overlap store indexer finished.
-- Checking store.
----------------------------------------
-- Starting command on Tue Nov 16 18:31:59 2021 with 2047.668 GB free disk space

    cd unitigging
    /home/jon/software/canu-2.2/bin/ovStoreDump \
     -S ../WB_SUP.seqStore \
     -O  ./WB_SUP.ovlStore \
     -counts \
     > ./WB_SUP.ovlStore/counts.dat 2> ./WB_SUP.ovlStore/counts.err

-- Finished on Tue Nov 16 18:31:59 2021 (in the blink of an eye) with 2047.667 GB free disk space
----------------------------------------
--
-- Overlap store 'unitigging/WB_SUP.ovlStore' successfully constructed.
-- Found 3804306 overlaps for 35112 reads; 84717 reads have no overlaps.
--
----------------------------------------
-- Starting command on Tue Nov 16 18:31:59 2021 with 2047.668 GB free disk space

    cd unitigging
    /home/jon/software/canu-2.2/bin/ovStoreStats \
     -C 40.31 \
     -S ../WB_SUP.seqStore \
     -O  ./WB_SUP.ovlStore \
     -o  ./WB_SUP.ovlStore \
     > ./WB_SUP.ovlStore.summary.err 2>&1

-- Finished on Tue Nov 16 18:31:59 2021 (furiously fast) with 2047.667 GB free disk space
----------------------------------------
--
-- Overlap store 'unitigging/WB_SUP.ovlStore' contains:
--
--   category            reads     %          read length        feature size or coverage  analysis
--   ----------------  -------  -------  ----------------------  ------------------------  --------------------
--   middle-missing          0    0.00        0.00 +- 0.00             0.00 +- 0.00       (bad trimming)
--   middle-hump             0    0.00        0.00 +- 0.00             0.00 +- 0.00       (bad trimming)
--   no-5-prime              0    0.00        0.00 +- 0.00             0.00 +- 0.00       (bad trimming)
--   no-3-prime              0    0.00        0.00 +- 0.00             0.00 +- 0.00       (bad trimming)
--   
--   low-coverage            1    0.00     5364.00 +- 0.00             5.50 +- 0.50       (easy to assemble, potential for lower quality consensus)
--   unique              30090   85.70    13749.04 +- 3576.59         41.57 +- 7.88       (easy to assemble, perfect, yay)
--   repeat-cont          1020    2.90    13282.42 +- 3086.47        382.77 +- 218.14     (potential for consensus errors, no impact on assembly)
--   repeat-dove            35    0.10    21060.63 +- 3333.08        299.90 +- 213.44     (hard to assemble, likely won't assemble correctly or even at all)
--   
--   span-repeat           658    1.87    14490.84 +- 3854.93       2451.88 +- 3390.76    (read spans a large repeat, usually easy to assemble)
--   uniq-repeat-cont     2615    7.45    13371.57 +- 2599.68                             (should be uniquely placed, low potential for consensus errors, no impact on assembly)
--   uniq-repeat-dove      318    0.91    19775.91 +- 4694.71                             (will end contigs, potential to misassemble)
--   uniq-anchor           375    1.07    13278.44 +- 2996.24       5270.52 +- 3768.33    (repeat read, with unique section, probable bad read)
-- Finished stage 'utg-createOverlapStore', reset canuIteration.
--
-- Loading read lengths.
-- Loading number of overlaps per read.
--
-- Configure RED for 15gb memory.
--                   Batches of at most (unlimited) reads.
--                                      500000000 bases.
--                   Expecting evidence of at most 536870912 bases per iteration.
--
--           Total                                               Reads                 Olaps Evidence
--    Job   Memory      Read Range         Reads        Bases   Memory        Olaps   Memory   Memory  (Memory in MB)
--   ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------
--      1 15360.15         1-24093          7179    100592363 12279.57       749633     8.58  1024.00
--      2 15360.24     24094-48395          7202    100592681 12279.61       754641     8.64  1024.00
--      3 15362.37     48396-72284          7168    100611300 12281.88       742084     8.49  1024.00
--      4 15361.18     72285-96811          7210    100601158 12280.64       745740     8.53  1024.00
--      5 13015.01     96812-119829         6356     81375389  9933.72       812208     9.29  1024.00
--   ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------
--                                                  483772891               3804306
-- Finished stage 'readErrorDetectionConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'red' concurrent execution on Tue Nov 16 18:31:59 2021 with 2047.666 GB free disk space (5 processes; 4 concurrently)

    cd unitigging/3-overlapErrorAdjustment
    ./red.sh 1 > ./red.000001.out 2>&1
    ./red.sh 2 > ./red.000002.out 2>&1
    ./red.sh 3 > ./red.000003.out 2>&1
    ./red.sh 4 > ./red.000004.out 2>&1
    ./red.sh 5 > ./red.000005.out 2>&1

-- Finished on Tue Nov 16 18:33:18 2021 (79 seconds) with 2047.661 GB free disk space
----------------------------------------
-- Found 5 read error detection output files.
-- Finished stage 'readErrorDetectionCheck', reset canuIteration.
--
-- Loading read lengths.
-- Loading number of overlaps per read.
--
-- Configure OEA for 8gb memory.
--                   Batches of at most (unlimited) reads.
--                                      300000000 bases.
--
--           Total                                               Reads                 Olaps  Adjusts
--    Job   Memory      Read Range         Reads        Bases   Memory        Olaps   Memory   Memory  (Memory in MB)
--   ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------
--      1  2741.30         1-71876         21423    300012631   294.77      2231796    68.11   330.42
--      2  2610.08     71877-119829        13692    183760260   183.67      1572510    47.99   330.42
--   ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------
--                                                  483772891               3804306
-- Finished stage 'overlapErrorAdjustmentConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'oea' concurrent execution on Tue Nov 16 18:33:18 2021 with 2047.661 GB free disk space (2 processes; 7 concurrently)

    cd unitigging/3-overlapErrorAdjustment
    ./oea.sh 1 > ./oea.000001.out 2>&1
    ./oea.sh 2 > ./oea.000002.out 2>&1

-- Finished on Tue Nov 16 18:34:54 2021 (96 seconds) with 2047.653 GB free disk space
----------------------------------------
-- Found 2 overlap error adjustment output files.
-- Finished stage 'overlapErrorAdjustmentCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 18:34:54 2021 with 2047.653 GB free disk space

    cd unitigging/3-overlapErrorAdjustment
    /home/jon/software/canu-2.2/bin/loadErates \
      -S ../../WB_SUP.seqStore \
      -O ../WB_SUP.ovlStore \
      -L ./oea.files \
    > ./oea.apply.err 2>&1

-- Finished on Tue Nov 16 18:34:54 2021 (in the blink of an eye) with 2047.645 GB free disk space
----------------------------------------
-- No report available.
-- Finished stage 'updateOverlapStore', reset canuIteration.
-- Finished stage 'unitig', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'bat' concurrent execution on Tue Nov 16 18:34:54 2021 with 2047.645 GB free disk space (1 processes; 1 concurrently)

    cd unitigging/4-unitigger
    ./unitigger.sh 1 > ./unitigger.000001.out 2>&1

-- Finished on Tue Nov 16 18:34:56 2021 (2 seconds) with 2047.632 GB free disk space
----------------------------------------
-- Unitigger finished successfully.
--  
--  ERROR RATES
--  -----------
--                                                   --------threshold------
--  38666                        fraction error      fraction        percent
--  samples                              (1e-5)         error          error
--                   --------------------------      --------       --------
--  command line (-eg)                           ->  12000.00       12.0000%
--  command line (-ef)                           ->  -----.--      ---.----%
--  command line (-eM)                           ->  12000.00       12.0000%
--  mean + std.dev      84.88 +-  12 *   527.42  ->   6413.92        6.4139%  (enabled)
--  median + mad         0.00 +-  12 *     0.00  ->      0.00        0.0000%
--  90th percentile                              ->     78.00        0.0780%
--  
--  BEST EDGE FILTERING
--  -------------------
--  At graph threshold 12.0000%, reads:
--    available to have edges:         3560
--    with at least one edge:          3560
--  
--  At max threshold 12.0000%, reads:  (not computed)
--    available to have edges:            0
--    with at least one edge:             0
--  
--  At tight threshold 0.0780%, reads with:
--    both edges below error threshold:      1821  (80.00% minReadsBest threshold = 2848)
--    one  edge  above error threshold:       642
--    both edges above error threshold:      1097
--    at least one edge:                     3560
--  
--  At loose threshold 6.4139%, reads with:
--    both edges below error threshold:      3489  (80.00% minReadsBest threshold = 2848)
--    one  edge  above error threshold:        67
--    both edges above error threshold:         4
--    at least one edge:                     3560
--  
--  
--  INITIAL EDGES
--  -------- ----------------------------------------
--     31487 reads are contained
--     84728 reads have no best edges (singleton)
--         4 reads have only one best edge (spur) 
--                  4 are mutual best
--      3610 reads have two best edges 
--                140 have one mutual best edge
--               3422 have two mutual best edges
--  
--  
--  FINAL EDGES
--  -------- ----------------------------------------
--     31487 reads are contained
--     84743 reads have no best edges (singleton)
--         9 reads have only one best edge (spur) 
--                  8 are mutual best
--      3590 reads have two best edges 
--                128 have one mutual best edge
--               3419 have two mutual best edges
--  
--  
--  EDGE FILTERING
--  -------- ------------------------------------------
--         0 reads are ignored
--         9 reads have a gap in overlap coverage
--        11 reads have lopsided best edges
-- Found, in version 1, after unitig construction:
--   contigs:      30 sequences, total length 11946668 bp (including 37 repeats of total length 858740 bp).
--   bubbles:      32 sequences, total length 766706 bp.
--   unassembled:  73 sequences, total length 1063238 bp.
--
-- Contig sizes based on genome size 12mbp:
--
--            NG (bp)  LG (contigs)    sum (bp)
--         ----------  ------------  ----------
--     10     2563776             1     2563776
--     20     2563776             1     2563776
--     30     2281462             2     4845238
--     40     2281462             2     4845238
--     50     1582085             3     6427323
--     60     1542069             4     7969392
--     70     1120445             5     9089837
--     80      734933             6     9824770
--     90      353744             9    11116542
--
-- Finished stage 'unitigCheck', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 18:34:56 2021 with 2047.632 GB free disk space

    cd unitigging
    /home/jon/software/canu-2.2/bin/utgcns \
      -S ../WB_SUP.seqStore \
      -T  ./WB_SUP.ctgStore 1 \
      -partition 0.8 1 0.1 \
    > ./WB_SUP.ctgStore/partitioning.log 2>&1

-- Finished on Tue Nov 16 18:34:57 2021 (one second) with 2047.399 GB free disk space
----------------------------------------
-- Using slow alignment for consensus (iteration '0').
-- Configured 4 consensus jobs.
-- Finished stage 'consensusConfigure', reset canuIteration.
--
-- Local: cns        2.875 GB    4 CPUs x   4 jobs    11.500 GB  16 CPUs  (consensus)
--
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cns' concurrent execution on Tue Nov 16 18:34:57 2021 with 2047.399 GB free disk space (4 processes; 4 concurrently)

    cd unitigging/5-consensus
    ./consensus.sh 1 > ./consensus.000001.out 2>&1
    ./consensus.sh 2 > ./consensus.000002.out 2>&1
    ./consensus.sh 3 > ./consensus.000003.out 2>&1
    ./consensus.sh 4 > ./consensus.000004.out 2>&1

-- Finished on Tue Nov 16 18:36:59 2021 (122 seconds) with 2047.369 GB free disk space
----------------------------------------
-- Using slow alignment for consensus (iteration '1').
-- Configured 4 consensus jobs.
-- All 4 consensus jobs finished successfully.
-- Finished stage 'consensusCheck', reset canuIteration.
-- Using slow alignment for consensus (iteration '0').
-- Configured 4 consensus jobs.
----------------------------------------
-- Starting command on Tue Nov 16 18:36:59 2021 with 2047.369 GB free disk space

    cd unitigging
    /home/jon/software/canu-2.2/bin/tgStoreLoad \
      -S ../WB_SUP.seqStore \
      -T  ./WB_SUP.ctgStore 2 \
      -L ./5-consensus/ctgcns.files \
    > ./5-consensus/ctgcns.files.ctgStoreLoad.err 2>&1

-- Finished on Tue Nov 16 18:36:59 2021 (in the blink of an eye) with 2047.34 GB free disk space
----------------------------------------
-- Purging consensus output after loading to ctgStore.
-- Purged 4 .cns outputs.
----------------------------------------
-- Starting command on Tue Nov 16 18:36:59 2021 with 2047.369 GB free disk space

    cd unitigging
    /home/jon/software/canu-2.2/bin/tgStoreDump \
      -S ../WB_SUP.seqStore \
      -T ./WB_SUP.ctgStore 2 \
      -sizes -s 12000000 \
    > ./WB_SUP.ctgStore/seqDB.v002.sizes.txt

-- Finished on Tue Nov 16 18:36:59 2021 (like a bat out of hell) with 2047.369 GB free disk space
----------------------------------------
-- Found, in version 2, after consensus generation:
--   contigs:      30 sequences, total length 11920415 bp (including 37 repeats of total length 852701 bp).
--   bubbles:      32 sequences, total length 763509 bp.
--   unassembled:  73 sequences, total length 1063238 bp.
--
-- Contig sizes based on genome size 12mbp:
--
--            NG (bp)  LG (contigs)    sum (bp)
--         ----------  ------------  ----------
--     10     2557672             1     2557672
--     20     2557672             1     2557672
--     30     2271239             2     4828911
--     40     2271239             2     4828911
--     50     1581255             3     6410166
--     60     1538882             4     7949048
--     70     1119075             5     9068123
--     80      734169             6     9802292
--     90      353720             9    11093630
--
-- Finished stage 'consensusLoad', reset canuIteration.
----------------------------------------
-- Starting command on Tue Nov 16 18:36:59 2021 with 2047.369 GB free disk space

    cd .
    /home/jon/software/canu-2.2/bin/tgStoreDump \
      -S ./WB_SUP.seqStore \
      -T ./unitigging/WB_SUP.ctgStore 2 \
      -o ./WB_SUP.contigs \
      -layout \
    > ./WB_SUP.contigs.layout.err 2>&1

-- Finished on Tue Nov 16 18:36:59 2021 (fast as lightning) with 2047.368 GB free disk space
----------------------------------------
----------------------------------------
-- Starting command on Tue Nov 16 18:36:59 2021 with 2047.368 GB free disk space

    cd .
    /home/jon/software/canu-2.2/bin/tgStoreDump \
      -S ./WB_SUP.seqStore \
      -T ./unitigging/WB_SUP.ctgStore 2 \
      -consensus -fasta \
      -unassembled \
    > ./WB_SUP.unassembled.fasta
    2> ./WB_SUP.unassembled.err

-- Finished on Tue Nov 16 18:36:59 2021 (in the blink of an eye) with 2047.367 GB free disk space
----------------------------------------
----------------------------------------
-- Starting command on Tue Nov 16 18:36:59 2021 with 2047.367 GB free disk space

    cd .
    /home/jon/software/canu-2.2/bin/tgStoreDump \
      -S ./WB_SUP.seqStore \
      -T ./unitigging/WB_SUP.ctgStore 2 \
      -consensus -fasta \
      -contigs \
    > ./WB_SUP.contigs.fasta
    2> ./WB_SUP.contigs.err

-- Finished on Tue Nov 16 18:36:59 2021 (furiously fast) with 2047.355 GB free disk space
----------------------------------------
-- Finished stage 'generateOutputs', reset canuIteration.
--
-- Assembly 'WB_SUP' finished in '/media/jon/BackupPlus/MinION_data/ASSEMBLY/canu/WB_SUP_canu_no_mhap'.
--
-- Summary saved in 'WB_SUP.report'.
--
-- Sequences saved:
--   Contigs       -> 'WB_SUP.contigs.fasta'
--   Unassembled   -> 'WB_SUP.unassembled.fasta'
--
-- Read layouts saved:
--   Contigs       -> 'WB_SUP.contigs.layout'.
--
-- Bye.

WB_SUP.contigs.layout.tigInfo

#tigID  tigLen  coverage    tigClass    sugRept sugBubb sugCirc numChildren
5   28250   4.61    contig  no  yes no  11
8   26693   10.05   contig  no  yes no  19
14  23519   4.99    contig  no  no  no  9
16  18262   3.56    contig  no  no  no  5
18  1538882 40.60   contig  no  no  no  4528
22  353720  27.19   contig  no  no  no  708
23  23732   16.45   contig  no  no  no  29
24  24407   4.62    contig  no  yes no  8
29  31376   4.90    contig  no  yes yes 12
30  27951   9.69    contig  no  yes no  21
34  27248   10.19   contig  no  yes no  22
35  24758   5.05    contig  yes no  no  9
36  2557672 40.43   contig  no  no  no  7412
37  22357   13.38   contig  yes no  no  23
38  23267   5.14    contig  yes no  no  10
39  20146   6.81    contig  yes no  no  10
40  13878   1.00    contig  yes yes no  1
41  28182   18.50   contig  yes yes no  35
42  31964   8.98    contig  no  no  no  20
43  2271239 40.68   contig  no  no  no  6666
44  37875   18.64   contig  yes yes no  56
45  18538   3.74    contig  yes yes no  6
46  31398   36.13   contig  yes yes no  70
47  23785   97.36   contig  yes yes no  168
48  24849   1.81    contig  yes yes no  2
49  29440   72.44   contig  yes yes no  148
50  31241   24.10   contig  yes yes yes 56
51  38221   95.90   contig  yes yes yes 264
52  19115   7.36    contig  yes no  no  11
53  95517   38.96   contig  no  no  no  262
54  734169  41.75   contig  no  no  no  2198
55  532270  40.67   contig  no  no  no  1571
56  133843  44.64   contig  no  no  no  424
57  80792   37.06   contig  no  no  no  214
58  20979   8.15    contig  yes no  no  11
59  77711   46.69   contig  no  no  no  254
60  17207   3.33    contig  yes no  no  4
61  17114   3.33    contig  yes no  no  4
62  19628   7.93    contig  yes no  no  11
63  13306   1.00    contig  yes no  no  1
64  1119075 39.95   contig  no  no  no  3244
65  51995   29.58   contig  no  no  no  109
66  1581255 41.59   contig  no  no  no  4727
67  405348  42.17   contig  no  no  no  1233
68  19444   3.10    contig  yes yes no  4
69  23813   28.62   contig  yes yes no  53
70  14538   1.00    contig  yes yes no  1
71  17429   2.48    contig  yes yes yes 3
72  34333   26.52   contig  no  yes no  69
73  22653   8.98    contig  yes yes no  14
74  79216   20.94   contig  yes no  no  140
75  12357   2.71    contig  yes no  no  3
76  26085   7.85    contig  yes yes no  18
77  27789   15.49   contig  yes yes no  33
78  13130   8.70    contig  yes yes no  11
79  19301   14.73   contig  yes yes no  24
80  16258   1.46    contig  yes yes no  2
81  13345   1.00    contig  yes yes no  1
82  24037   19.02   contig  yes yes no  36
83  16491   7.97    contig  yes yes no  11
84  11393   2.83    contig  yes yes no  3
85  20138   4.45    contig  yes yes no  7
86  13442   1.00    unassm  no  no  no  1
87  11702   1.00    unassm  no  no  no  1
88  23134   1.00    unassm  no  no  no  1
89  6425    1.00    unassm  no  no  no  1
90  16167   1.00    unassm  no  no  no  1
91  20576   1.00    unassm  no  no  no  1
92  17200   1.00    unassm  no  no  no  1
93  12863   1.00    unassm  no  no  no  1
94  11107   1.00    unassm  no  no  no  1
95  10905   1.00    unassm  no  no  no  1
96  13849   1.00    unassm  no  no  no  1
97  13652   1.00    unassm  no  no  no  1
98  17820   1.00    unassm  no  no  no  1
99  10920   1.00    unassm  no  no  no  1
100 20865   1.00    unassm  no  no  no  1
101 9914    1.00    unassm  no  no  no  1
102 12119   1.00    unassm  no  no  no  1
103 20414   1.00    unassm  no  no  no  1
104 15934   1.00    unassm  no  no  no  1
105 10676   1.00    unassm  no  no  no  1
106 12314   1.00    unassm  no  no  no  1
107 17741   1.00    unassm  no  no  no  1
108 13131   1.00    unassm  no  no  no  1
109 11646   1.00    unassm  no  no  no  1
110 17071   1.00    unassm  no  no  no  1
111 5364    1.00    unassm  no  no  no  1
112 14259   1.00    unassm  no  no  no  1
113 11765   1.00    unassm  no  no  no  1
114 13299   1.00    unassm  no  no  no  1
115 22314   1.00    unassm  no  no  no  1
116 12096   1.00    unassm  no  no  no  1
117 15677   1.00    unassm  no  no  no  1
118 17054   1.00    unassm  no  no  no  1
119 4824    1.00    unassm  no  no  no  1
120 20289   1.00    unassm  no  no  no  1
121 16092   1.00    unassm  no  no  no  1
122 11026   1.00    unassm  no  no  no  1
123 19260   1.00    unassm  no  no  no  1
124 14681   1.00    unassm  no  no  no  1
125 10688   1.00    unassm  no  no  no  1
126 13046   1.00    unassm  no  no  no  1
127 13906   1.00    unassm  no  no  no  1
128 17467   1.00    unassm  no  no  no  1
129 13354   1.00    unassm  no  no  no  1
130 15342   1.00    unassm  no  no  no  1
131 17330   1.00    unassm  no  no  no  1
132 11122   1.00    unassm  no  no  no  1
133 16699   1.00    unassm  no  no  no  1
134 20109   1.00    unassm  no  no  no  1
135 16368   1.00    unassm  no  no  no  1
136 15309   1.00    unassm  no  no  no  1
137 13657   1.00    unassm  no  no  no  1
138 11807   1.00    unassm  no  no  no  1
139 16712   1.00    unassm  no  no  no  1
140 14622   1.00    unassm  no  no  no  1
141 22726   1.00    unassm  no  no  no  1
142 15779   1.00    unassm  no  no  no  1
143 12956   1.00    unassm  no  no  no  1
144 14156   1.00    unassm  no  no  no  1
145 18106   1.00    unassm  no  no  no  1
146 18206   1.00    unassm  no  no  no  1
147 16917   1.00    unassm  no  no  no  1
148 9672    1.00    unassm  no  no  no  1
149 14964   1.00    unassm  no  no  no  1
150 20941   1.00    unassm  no  no  no  1
151 14597   1.00    unassm  no  no  no  1
152 8895    1.00    unassm  no  no  no  1
153 5691    1.00    unassm  no  no  no  1
154 13360   1.00    unassm  no  no  no  1
155 17678   1.00    unassm  no  no  no  1
156 7026    1.00    unassm  no  no  no  1
157 19300   1.00    unassm  no  no  no  1
158 15143   1.00    unassm  no  no  no  1
skoren commented 2 years ago

I would guess the issue is the default pipe output from MHAP to conversion, that's what the parameters I suggested modified. The run looks correct, the default corOutCoverage is 40x (https://canu.readthedocs.io/en/latest/faq.html#what-parameters-can-i-tweak) but it selects the best/longest reads. You can see this since the input data had read N50 of 10kb and the corrected reads have an N50 of 14kb. So I wouldn't change the default in this case. You can run the assembly w/o correction if you want as another data point.

Since the original issue is resolved by the updated parameters and your run finished correctly, I'm going to close this issue. If you encounter errors with your uncorrected run, open a new issue.