marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
654 stars 179 forks source link

Mhap overlap jobs failed, tried 2 times, giving up. #2339

Closed katievigil closed 1 week ago

katievigil commented 1 week ago

Hi I am running a large dolphin feces nanopore sequencing fastq file and I am running into memory issues I think...

How much memory should I ask for to run this sample? Thanks! Katie

Canu v2.2 Using Loni HPC system "single" node see below for details:

image image

Script:

#!/bin/bash
#SBATCH -A loni_virus24    ### Allocation name
#SBATCH -p single             ### Queue name
#SBATCH -t 7-00:00:00         ### Wall time
#SBATCH -N 1                  ### Number of nodes
#SBATCH -c 48                 ### Number of cores per task
#SBATCH --output=slurm-%j.out-%N
#SBATCH --error=slurm-%j.err-%N
#SBATCH -J dolphin            ### Job name

canu report log:

-- In sequence store './01_dolphin_feces_SQK-NBD114-24_barcode15.seqStore':
--   Found 797209 reads.
--   Found 1115751947 bases (557.87 times coverage).
--    Histogram of raw reads:
--
--    G=1115751947                       sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010         2447     35297    111576701  ||       1000-3439       789247|---------------------------------------------------------------
--    00020         1934     87274    223150932  ||       3440-5879         7374|-
--    00030         1657    149860    334726414  ||       5880-8319          488|-
--    00040         1471    221507    446302100  ||       8320-10759          75|-
--    00050         1332    301377    557877163  ||      10760-13199          13|-
--    00060         1227    388854    669451297  ||      13200-15639           2|-
--    00070         1149    482970    781026989  ||      15640-18079           2|-
--    00080         1089    582837    892601817  ||      18080-20519           2|-
--    00090         1040    687758   1004177769  ||      20520-22959           0|
--    00100         1000    797208   1115751947  ||      22960-25399           1|-
--    001.000x              797209   1115751947  ||      25400-27839           0|
--                                               ||      27840-30279           0|
--                                               ||      30280-32719           2|-
--                                               ||      32720-35159           1|-
--                                               ||      35160-37599           0|
--                                               ||      37600-40039           0|
--                                               ||      40040-42479           0|
--                                               ||      42480-44919           0|
--                                               ||      44920-47359           0|
--                                               ||      47360-49799           1|-
--                                               ||      49800-52239           0|
--                                               ||      52240-54679           0|
--                                               ||      54680-57119           0|
--                                               ||      57120-59559           0|
--                                               ||      59560-61999           0|
--                                               ||      62000-64439           0|
--                                               ||      64440-66879           0|
--                                               ||      66880-69319           0|
--                                               ||      69320-71759           0|
--                                               ||      71760-74199           0|
--                                               ||      74200-76639           0|
--                                               ||      76640-79079           0|
--                                               ||      79080-81519           0|
--                                               ||      81520-83959           0|
--                                               ||      83960-86399           0|
--                                               ||      86400-88839           0|
--                                               ||      88840-91279           0|
--                                               ||      91280-93719           0|
--                                               ||      93720-96159           0|
--                                               ||      96160-98599           0|
--                                               ||      98600-101039          0|
--                                               ||     101040-103479          0|
--                                               ||     103480-105919          0|
--                                               ||     105920-108359          0|
--                                               ||     108360-110799          0|
--                                               ||     110800-113239          0|
--                                               ||     113240-115679          0|
--                                               ||     115680-118119          0|
--                                               ||     118120-120559          0|
--                                               ||     120560-122999          1|-
--

[CORRECTION/MERS]
--
--  16-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2  84901859 ********************************************************************** 0.5295 0.1814
--       3-     4  48752975 ****************************************                               0.7251 0.2820
--       5-     7  16970145 *************                                                          0.8876 0.4026
--       8-    11   5153508 ****                                                                   0.9522 0.4767
--      12-    16   1866144 *                                                                      0.9750 0.5162
--      17-    22    857680                                                                        0.9844 0.5398
--      23-    29    477057                                                                        0.9890 0.5559
--      30-    37    306136                                                                        0.9918 0.5683
--      38-    46    209198                                                                        0.9936 0.5788
--      47-    56    153468                                                                        0.9948 0.5878
--      57-    67    118075                                                                        0.9957 0.5961
--      68-    79     90957                                                                        0.9964 0.6037
--      80-    92     73074                                                                        0.9970 0.6108
--      93-   106     58842                                                                        0.9974 0.6174
--     107-   121     47860                                                                        0.9978 0.6235
--     122-   137     39346                                                                        0.9981 0.6293
--     138-   154     32866                                                                        0.9983 0.6347
--     155-   172     27457                                                                        0.9985 0.6398
--     173-   191     22528                                                                        0.9987 0.6445
--     192-   211     19821                                                                        0.9988 0.6489
--     212-   232     16958                                                                        0.9990 0.6531
--     233-   254     14718                                                                        0.9991 0.6571
--     255-   277     12478                                                                        0.9992 0.6609
--     278-   301     10403                                                                        0.9992 0.6644
--     302-   326      8656                                                                        0.9993 0.6676
--     327-   352      7403                                                                        0.9993 0.6705
--     353-   379      6630                                                                        0.9994 0.6732
--     380-   407      5785                                                                        0.9994 0.6758
--     408-   436      5243                                                                        0.9995 0.6782
--     437-   466      4744                                                                        0.9995 0.6805
--     467-   497      4313                                                                        0.9995 0.6828
--     498-   529      3823                                                                        0.9996 0.6850
--     530-   562      3483                                                                        0.9996 0.6871
--     563-   596      3214                                                                        0.9996 0.6891
--     597-   631      2780                                                                        0.9996 0.6911
--     632-   667      2546                                                                        0.9996 0.6929
--     668-   704      2116                                                                        0.9997 0.6947
--     705-   742      1982                                                                        0.9997 0.6963
--     743-   781      1712                                                                        0.9997 0.6978
--     782-   821      1828                                                                        0.9997 0.6992
--
--           0 (max occurrences)
--   935925223 (total mers, non-unique)
--   160347383 (distinct mers, non-unique)
--           0 (unique mers)
(base) [kvigil@qbc2 01_dolphin_feces_SQK-NBD114-24_barcode15]$ cat *.err*

Found perl:
   /work/kvigil/.conda/envs/canu-env/bin/perl
   This is perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-thread-multi

Found java:
   /work/kvigil/.conda/envs/canu-env/lib/jvm/bin/java
   openjdk version "22.0.1-internal" 2024-04-16

Found canu:
   /ddnB/work/kvigil/.conda/envs/canu-env/bin/canu
   canu 2.2

Creating library '01_dolphin_feces_SQK-NBD114-24_barcode15' for Nanopore raw reads.

               reads               bases
---------- --------- ------ ------------ ------
Loaded        797209   4.2%   1115751947  10.7%  /ddnB/work/kvigil/onr.raw.data.sup.trim/rawdata/results/trimmed/sd_marinemammal/fastq/01_dolphin_feces_SQK-NBD114-24_barcode15.fastq
Short       18105370  95.8%   9267004423  89.3%

All reads processed.

               reads               bases
---------- --------- ------ ------------ ------
Loaded        797209   4.2%   1115751947  10.7%
Short       18105370  95.8%   9267004423  89.3%

Bye.

genomeSize=2m
maxinputCoverage=10000
corOutCoverage=10000
corMhapSensitivity=high
corMinCoverage=0
redMemory=32
oeaMemory=32
batMemory=200
correctedErrorRate=0.16
minInputCoverage=0
stopOnLowCoverage=0
-useGrid=false
katievigil commented 1 week ago

Error log: Below is my err log:

-- canu 2.2
--
-- CITATIONS
--
-- For 'standard' assemblies of PacBio or Nanopore reads:
--   Koren S, Walenz BP, Berlin K, Miller JR, Phillippy AM.
--   Canu: scalable and accurate long-read assembly via adaptive k-mer weighting and repeat separation.
--   Genome Res. 2017 May;27(5):722-736.
--   http://doi.org/10.1101/gr.215087.116
--
-- Read and contig alignments during correction and consensus use:
--   Šošic M, Šikic M.
--   Edlib: a C/C ++ library for fast, exact sequence alignment using edit distance.
--   Bioinformatics. 2017 May 1;33(9):1394-1395.
--   http://doi.org/10.1093/bioinformatics/btw753
--
-- Overlaps are generated using:
--   Berlin K, et al.
--   Assembling large genomes with single-molecule sequencing and locality-sensitive hashing.
--   Nat Biotechnol. 2015 Jun;33(6):623-30.
--   http://doi.org/10.1038/nbt.3238
--
--   Myers EW, et al.
--   A Whole-Genome Assembly of Drosophila.
--   Science. 2000 Mar 24;287(5461):2196-204.
--   http://doi.org/10.1126/science.287.5461.2196
--
-- Corrected read consensus sequences are generated using an algorithm derived from FALCON-sense:
--   Chin CS, et al.
--   Phased diploid genome assembly with single-molecule real-time sequencing.
--   Nat Methods. 2016 Dec;13(12):1050-1054.
--   http://doi.org/10.1038/nmeth.4035
--
-- Contig consensus sequences are generated using an algorithm derived from pbdagcon:
--   Chin CS, et al.
--   Nonhybrid, finished microbial genome assemblies from long-read SMRT sequencing data.
--   Nat Methods. 2013 Jun;10(6):563-9
--   http://doi.org/10.1038/nmeth.2474
--
-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '22.0.1-internal' (from '/work/kvigil/.conda/envs/canu-env/lib/jvm/bin/java') without -d64 support.
-- Detected gnuplot version '5.4 patchlevel 8   ' (from 'gnuplot') and image format 'png'.
--
-- Detected 48 CPUs and 189984 gigabytes of memory on the local machine.
--
-- Detected Slurm with 'sinfo' binary in /usr/bin/sinfo.
--          Slurm disabled by useGrid=false
--
-- Detected  '' with 'pbsnodes' binary in /usr/bin/pbsnodes.
--           disabled by useGrid=false
--
-- Local machine mode enabled; grid support not detected or not allowed.
--
--                                (tag)Concurrency
--                         (tag)Threads          |
--                (tag)Memory         |          |
--        (tag)             |         |          |       total usage      algorithm
--        -------  ----------  --------   --------  --------------------  -----------------------------
-- Local: meryl     12.000 GB    4 CPUs x  12 jobs   144.000 GB  48 CPUs  (k-mer counting)
-- Local: hap        8.000 GB    4 CPUs x  12 jobs    96.000 GB  48 CPUs  (read-to-haplotype assignment)
-- Local: cormhap    6.000 GB   16 CPUs x   3 jobs    18.000 GB  48 CPUs  (overlap detection with mhap)
-- Local: obtovl     4.000 GB    8 CPUs x   6 jobs    24.000 GB  48 CPUs  (overlap detection)
-- Local: utgovl     4.000 GB    8 CPUs x   6 jobs    24.000 GB  48 CPUs  (overlap detection)
-- Local: cor        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (read correction)
-- Local: ovb        4.000 GB    1 CPU  x  48 jobs   192.000 GB  48 CPUs  (overlap store bucketizer)
-- Local: ovs        8.000 GB    1 CPU  x  48 jobs   384.000 GB  48 CPUs  (overlap store sorting)
-- Local: red       32.000 GB    4 CPUs x  12 jobs   384.000 GB  48 CPUs  (read error detection)
-- Local: oea       32.000 GB    1 CPU  x  48 jobs  1536.000 GB  48 CPUs  (overlap error adjustment)
-- Local: bat      200.000 GB    4 CPUs x   1 job    200.000 GB   4 CPUs  (contig construction with bogart)
-- Local: cns        -.--- GB    4 CPUs x   - jobs     -.--- GB   - CPUs  (consensus)
--
-- Found untrimmed raw Nanopore reads in the input files.
--
-- Generating assembly '01_dolphin_feces_SQK-NBD114-24_barcode15' in '/ddnB/work/kvigil/onr.raw.data.sup.trim/rawdata/results/trimmed/sd_marinemammal/canu_out/01_dolphin_feces_SQK-NBD114-24_barcode15':
--   genomeSize:
--     2000000
--
--   Overlap Generation Limits:
--     corOvlErrorRate 0.3200 ( 32.00%)
--     obtOvlErrorRate 0.1600 ( 16.00%)
--     utgOvlErrorRate 0.1600 ( 16.00%)
--
--   Overlap Processing Limits:
--     corErrorRate    0.3000 ( 30.00%)
--     obtErrorRate    0.1600 ( 16.00%)
--     utgErrorRate    0.1600 ( 16.00%)
--     cnsErrorRate    0.1600 ( 16.00%)
--
--   Stages to run:
--     correct raw reads.
--     trim corrected reads.
--     assemble corrected and trimmed reads.
--
--
-- BEGIN CORRECTION
----------------------------------------
-- Starting command on Thu Aug 29 07:35:12 2024 with 5880470.553 GB free disk space

    cd .
    ./01_dolphin_feces_SQK-NBD114-24_barcode15.seqStore.sh \
    > ./01_dolphin_feces_SQK-NBD114-24_barcode15.seqStore.err 2>&1

-- Finished on Thu Aug 29 07:37:36 2024 (144 seconds) with 5880456.565 GB free disk space
----------------------------------------
--
-- In sequence store './01_dolphin_feces_SQK-NBD114-24_barcode15.seqStore':
--   Found 797209 reads.
--   Found 1115751947 bases (557.87 times coverage).
--    Histogram of raw reads:
--
--    G=1115751947                       sum of  ||               length     num
--    NG         length     index       lengths  ||                range    seqs
--    ----- ------------ --------- ------------  ||  ------------------- -------
--    00010         2447     35297    111576701  ||       1000-3439       789247|---------------------------------------------------------------
--    00020         1934     87274    223150932  ||       3440-5879         7374|-
--    00030         1657    149860    334726414  ||       5880-8319          488|-
--    00040         1471    221507    446302100  ||       8320-10759          75|-
--    00050         1332    301377    557877163  ||      10760-13199          13|-
--    00060         1227    388854    669451297  ||      13200-15639           2|-
--    00070         1149    482970    781026989  ||      15640-18079           2|-
--    00080         1089    582837    892601817  ||      18080-20519           2|-
--    00090         1040    687758   1004177769  ||      20520-22959           0|
--    00100         1000    797208   1115751947  ||      22960-25399           1|-
--    001.000x              797209   1115751947  ||      25400-27839           0|
--                                               ||      27840-30279           0|
--                                               ||      30280-32719           2|-
--                                               ||      32720-35159           1|-
--                                               ||      35160-37599           0|
--                                               ||      37600-40039           0|
--                                               ||      40040-42479           0|
--                                               ||      42480-44919           0|
--                                               ||      44920-47359           0|
--                                               ||      47360-49799           1|-
--                                               ||      49800-52239           0|
--                                               ||      52240-54679           0|
--                                               ||      54680-57119           0|
--                                               ||      57120-59559           0|
--                                               ||      59560-61999           0|
--                                               ||      62000-64439           0|
--                                               ||      64440-66879           0|
--                                               ||      66880-69319           0|
--                                               ||      69320-71759           0|
--                                               ||      71760-74199           0|
--                                               ||      74200-76639           0|
--                                               ||      76640-79079           0|
--                                               ||      79080-81519           0|
--                                               ||      81520-83959           0|
--                                               ||      83960-86399           0|
--                                               ||      86400-88839           0|
--                                               ||      88840-91279           0|
--                                               ||      91280-93719           0|
--                                               ||      93720-96159           0|
--                                               ||      96160-98599           0|
--                                               ||      98600-101039          0|
--                                               ||     101040-103479          0|
--                                               ||     103480-105919          0|
--                                               ||     105920-108359          0|
--                                               ||     108360-110799          0|
--                                               ||     110800-113239          0|
--                                               ||     113240-115679          0|
--                                               ||     115680-118119          0|
--                                               ||     118120-120559          0|
--                                               ||     120560-122999          1|-
--
----------------------------------------
-- Starting command on Thu Aug 29 07:37:39 2024 with 5880456.592 GB free disk space

    cd correction/0-mercounts
    ./meryl-configure.sh \
    > ./meryl-configure.err 2>&1

-- Finished on Thu Aug 29 07:37:39 2024 (lickety-split) with 5880456.592 GB free disk space
----------------------------------------
--  segments   memory batches
--  -------- -------- -------
--        01  2.35 GB       2
--        02  1.44 GB       2
--        04  0.72 GB       2
--        06  0.48 GB       2
--        08  0.36 GB       2
--
--  For 797209 reads with 1115751947 bases, limit to 11 batches.
--  Will count kmers using 01 jobs, each using 4 GB and 4 threads.
--
-- Finished stage 'merylConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Thu Aug 29 07:37:40 2024 with 5880456.592 GB free disk space (1 processes; 12 concurrently)

    cd correction/0-mercounts
    ./meryl-count.sh 1 > ./meryl-count.000001.out 2>&1

-- Finished on Thu Aug 29 07:39:08 2024 (88 seconds) with 5880445.471 GB free disk space
----------------------------------------
-- Found 1 Kmer counting (meryl) outputs.
-- Finished stage 'cor-merylCountCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'meryl' concurrent execution on Thu Aug 29 07:39:08 2024 with 5880445.471 GB free disk space (1 processes; 12 concurrently)

    cd correction/0-mercounts
    ./meryl-process.sh 1 > ./meryl-process.000001.out 2>&1

-- Finished on Thu Aug 29 07:39:18 2024 (10 seconds) with 5880444.411 GB free disk space
----------------------------------------
-- Meryl finished successfully.  Kmer frequency histogram:
--
--  16-mers                                                                                           Fraction
--    Occurrences   NumMers                                                                         Unique Total
--       1-     1         0                                                                        0.0000 0.0000
--       2-     2  84901859 ********************************************************************** 0.5295 0.1814
--       3-     4  48752975 ****************************************                               0.7251 0.2820
--       5-     7  16970145 *************                                                          0.8876 0.4026
--       8-    11   5153508 ****                                                                   0.9522 0.4767
--      12-    16   1866144 *                                                                      0.9750 0.5162
--      17-    22    857680                                                                        0.9844 0.5398
--      23-    29    477057                                                                        0.9890 0.5559
--      30-    37    306136                                                                        0.9918 0.5683
--      38-    46    209198                                                                        0.9936 0.5788
--      47-    56    153468                                                                        0.9948 0.5878
--      57-    67    118075                                                                        0.9957 0.5961
--      68-    79     90957                                                                        0.9964 0.6037
--      80-    92     73074                                                                        0.9970 0.6108
--      93-   106     58842                                                                        0.9974 0.6174
--     107-   121     47860                                                                        0.9978 0.6235
--     122-   137     39346                                                                        0.9981 0.6293
--     138-   154     32866                                                                        0.9983 0.6347
--     155-   172     27457                                                                        0.9985 0.6398
--     173-   191     22528                                                                        0.9987 0.6445
--     192-   211     19821                                                                        0.9988 0.6489
--     212-   232     16958                                                                        0.9990 0.6531
--     233-   254     14718                                                                        0.9991 0.6571
--     255-   277     12478                                                                        0.9992 0.6609
--     278-   301     10403                                                                        0.9992 0.6644
--     302-   326      8656                                                                        0.9993 0.6676
--     327-   352      7403                                                                        0.9993 0.6705
--     353-   379      6630                                                                        0.9994 0.6732
--     380-   407      5785                                                                        0.9994 0.6758
--     408-   436      5243                                                                        0.9995 0.6782
--     437-   466      4744                                                                        0.9995 0.6805
--     467-   497      4313                                                                        0.9995 0.6828
--     498-   529      3823                                                                        0.9996 0.6850
--     530-   562      3483                                                                        0.9996 0.6871
--     563-   596      3214                                                                        0.9996 0.6891
--     597-   631      2780                                                                        0.9996 0.6911
--     632-   667      2546                                                                        0.9996 0.6929
--     668-   704      2116                                                                        0.9997 0.6947
--     705-   742      1982                                                                        0.9997 0.6963
--     743-   781      1712                                                                        0.9997 0.6978
--     782-   821      1828                                                                        0.9997 0.6992
--
--           0 (max occurrences)
--   935925223 (total mers, non-unique)
--   160347383 (distinct mers, non-unique)
--           0 (unique mers)
-- Finished stage 'meryl-process', reset canuIteration.
--
-- Removing meryl database 'correction/0-mercounts/01_dolphin_feces_SQK-NBD114-24_barcode15.ms16'.
--
-- OVERLAPPER (mhap) (correction)
--
--
-- PARAMETERS: hashes=768, minMatches=2, threshold=0.73
--
-- Given 5.4 GB, can fit 8100 reads per block.
-- For 100 blocks, set stride to 25 blocks.
-- Logging partitioning to 'correction/1-overlapper/partitioning.log'.
-- Configured 99 mhap precompute jobs.
-- Configured 243 mhap overlap jobs.
-- Finished stage 'cor-mhapConfigure', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Thu Aug 29 07:39:20 2024 with 5880444.315 GB free disk space (99 processes; 3 concurrently)

    cd correction/1-overlapper
    ./precompute.sh 1 > ./precompute.000001.out 2>&1
    ./precompute.sh 2 > ./precompute.000002.out 2>&1
    ./precompute.sh 3 > ./precompute.000003.out 2>&1
    ./precompute.sh 4 > ./precompute.000004.out 2>&1
    ./precompute.sh 5 > ./precompute.000005.out 2>&1
    ./precompute.sh 6 > ./precompute.000006.out 2>&1
    ./precompute.sh 7 > ./precompute.000007.out 2>&1
    ./precompute.sh 8 > ./precompute.000008.out 2>&1
    ./precompute.sh 9 > ./precompute.000009.out 2>&1
    ./precompute.sh 10 > ./precompute.000010.out 2>&1
    ./precompute.sh 11 > ./precompute.000011.out 2>&1
    ./precompute.sh 12 > ./precompute.000012.out 2>&1
    ./precompute.sh 13 > ./precompute.000013.out 2>&1
    ./precompute.sh 14 > ./precompute.000014.out 2>&1
    ./precompute.sh 15 > ./precompute.000015.out 2>&1
    ./precompute.sh 16 > ./precompute.000016.out 2>&1
    ./precompute.sh 17 > ./precompute.000017.out 2>&1
    ./precompute.sh 18 > ./precompute.000018.out 2>&1
    ./precompute.sh 19 > ./precompute.000019.out 2>&1
    ./precompute.sh 20 > ./precompute.000020.out 2>&1
    ./precompute.sh 21 > ./precompute.000021.out 2>&1
    ./precompute.sh 22 > ./precompute.000022.out 2>&1
    ./precompute.sh 23 > ./precompute.000023.out 2>&1
    ./precompute.sh 24 > ./precompute.000024.out 2>&1
    ./precompute.sh 25 > ./precompute.000025.out 2>&1
    ./precompute.sh 26 > ./precompute.000026.out 2>&1
    ./precompute.sh 27 > ./precompute.000027.out 2>&1
    ./precompute.sh 28 > ./precompute.000028.out 2>&1
    ./precompute.sh 29 > ./precompute.000029.out 2>&1
    ./precompute.sh 30 > ./precompute.000030.out 2>&1
    ./precompute.sh 31 > ./precompute.000031.out 2>&1
    ./precompute.sh 32 > ./precompute.000032.out 2>&1
    ./precompute.sh 33 > ./precompute.000033.out 2>&1
    ./precompute.sh 34 > ./precompute.000034.out 2>&1
    ./precompute.sh 35 > ./precompute.000035.out 2>&1
    ./precompute.sh 36 > ./precompute.000036.out 2>&1
    ./precompute.sh 37 > ./precompute.000037.out 2>&1
    ./precompute.sh 38 > ./precompute.000038.out 2>&1
    ./precompute.sh 39 > ./precompute.000039.out 2>&1
    ./precompute.sh 40 > ./precompute.000040.out 2>&1
    ./precompute.sh 41 > ./precompute.000041.out 2>&1
    ./precompute.sh 42 > ./precompute.000042.out 2>&1
    ./precompute.sh 43 > ./precompute.000043.out 2>&1
    ./precompute.sh 44 > ./precompute.000044.out 2>&1
    ./precompute.sh 45 > ./precompute.000045.out 2>&1
    ./precompute.sh 46 > ./precompute.000046.out 2>&1
    ./precompute.sh 47 > ./precompute.000047.out 2>&1
    ./precompute.sh 48 > ./precompute.000048.out 2>&1
    ./precompute.sh 49 > ./precompute.000049.out 2>&1
    ./precompute.sh 50 > ./precompute.000050.out 2>&1
    ./precompute.sh 51 > ./precompute.000051.out 2>&1
    ./precompute.sh 52 > ./precompute.000052.out 2>&1
    ./precompute.sh 53 > ./precompute.000053.out 2>&1
    ./precompute.sh 54 > ./precompute.000054.out 2>&1
    ./precompute.sh 55 > ./precompute.000055.out 2>&1
    ./precompute.sh 56 > ./precompute.000056.out 2>&1
    ./precompute.sh 57 > ./precompute.000057.out 2>&1
    ./precompute.sh 58 > ./precompute.000058.out 2>&1
    ./precompute.sh 59 > ./precompute.000059.out 2>&1
    ./precompute.sh 60 > ./precompute.000060.out 2>&1
    ./precompute.sh 61 > ./precompute.000061.out 2>&1
    ./precompute.sh 62 > ./precompute.000062.out 2>&1
    ./precompute.sh 63 > ./precompute.000063.out 2>&1
    ./precompute.sh 64 > ./precompute.000064.out 2>&1
    ./precompute.sh 65 > ./precompute.000065.out 2>&1
    ./precompute.sh 66 > ./precompute.000066.out 2>&1
    ./precompute.sh 67 > ./precompute.000067.out 2>&1
    ./precompute.sh 68 > ./precompute.000068.out 2>&1
    ./precompute.sh 69 > ./precompute.000069.out 2>&1
    ./precompute.sh 70 > ./precompute.000070.out 2>&1
    ./precompute.sh 71 > ./precompute.000071.out 2>&1
    ./precompute.sh 72 > ./precompute.000072.out 2>&1
    ./precompute.sh 73 > ./precompute.000073.out 2>&1
    ./precompute.sh 74 > ./precompute.000074.out 2>&1
    ./precompute.sh 75 > ./precompute.000075.out 2>&1
    ./precompute.sh 76 > ./precompute.000076.out 2>&1
    ./precompute.sh 77 > ./precompute.000077.out 2>&1
    ./precompute.sh 78 > ./precompute.000078.out 2>&1
    ./precompute.sh 79 > ./precompute.000079.out 2>&1
    ./precompute.sh 80 > ./precompute.000080.out 2>&1
    ./precompute.sh 81 > ./precompute.000081.out 2>&1
    ./precompute.sh 82 > ./precompute.000082.out 2>&1
    ./precompute.sh 83 > ./precompute.000083.out 2>&1
    ./precompute.sh 84 > ./precompute.000084.out 2>&1
    ./precompute.sh 85 > ./precompute.000085.out 2>&1
    ./precompute.sh 86 > ./precompute.000086.out 2>&1
    ./precompute.sh 87 > ./precompute.000087.out 2>&1
    ./precompute.sh 88 > ./precompute.000088.out 2>&1
    ./precompute.sh 89 > ./precompute.000089.out 2>&1
    ./precompute.sh 90 > ./precompute.000090.out 2>&1
    ./precompute.sh 91 > ./precompute.000091.out 2>&1
    ./precompute.sh 92 > ./precompute.000092.out 2>&1
    ./precompute.sh 93 > ./precompute.000093.out 2>&1
    ./precompute.sh 94 > ./precompute.000094.out 2>&1
    ./precompute.sh 95 > ./precompute.000095.out 2>&1
    ./precompute.sh 96 > ./precompute.000096.out 2>&1
    ./precompute.sh 97 > ./precompute.000097.out 2>&1
    ./precompute.sh 98 > ./precompute.000098.out 2>&1
    ./precompute.sh 99 > ./precompute.000099.out 2>&1

-- Finished on Thu Aug 29 07:52:54 2024 (814 seconds) with 5880404.83 GB free disk space
----------------------------------------
-- All 99 mhap precompute jobs finished successfully.
-- Finished stage 'cor-mhapPrecomputeCheck', reset canuIteration.
--
-- Running jobs.  First attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Thu Aug 29 07:52:54 2024 with 5880404.83 GB free disk space (243 processes; 3 concurrently)

    cd correction/1-overlapper
    ./mhap.sh 1 > ./mhap.000001.out 2>&1
    ./mhap.sh 2 > ./mhap.000002.out 2>&1
    ./mhap.sh 3 > ./mhap.000003.out 2>&1
    ./mhap.sh 4 > ./mhap.000004.out 2>&1
    ./mhap.sh 5 > ./mhap.000005.out 2>&1
    ./mhap.sh 6 > ./mhap.000006.out 2>&1
    ./mhap.sh 7 > ./mhap.000007.out 2>&1
    ./mhap.sh 8 > ./mhap.000008.out 2>&1
    ./mhap.sh 9 > ./mhap.000009.out 2>&1
    ./mhap.sh 10 > ./mhap.000010.out 2>&1
    ./mhap.sh 11 > ./mhap.000011.out 2>&1
    ./mhap.sh 12 > ./mhap.000012.out 2>&1
    ./mhap.sh 13 > ./mhap.000013.out 2>&1
    ./mhap.sh 14 > ./mhap.000014.out 2>&1
    ./mhap.sh 15 > ./mhap.000015.out 2>&1
    ./mhap.sh 16 > ./mhap.000016.out 2>&1
    ./mhap.sh 17 > ./mhap.000017.out 2>&1
    ./mhap.sh 18 > ./mhap.000018.out 2>&1
    ./mhap.sh 19 > ./mhap.000019.out 2>&1
    ./mhap.sh 20 > ./mhap.000020.out 2>&1
    ./mhap.sh 21 > ./mhap.000021.out 2>&1
    ./mhap.sh 22 > ./mhap.000022.out 2>&1
    ./mhap.sh 23 > ./mhap.000023.out 2>&1
    ./mhap.sh 24 > ./mhap.000024.out 2>&1
    ./mhap.sh 25 > ./mhap.000025.out 2>&1
    ./mhap.sh 26 > ./mhap.000026.out 2>&1
    ./mhap.sh 27 > ./mhap.000027.out 2>&1
    ./mhap.sh 28 > ./mhap.000028.out 2>&1
    ./mhap.sh 29 > ./mhap.000029.out 2>&1
    ./mhap.sh 30 > ./mhap.000030.out 2>&1
    ./mhap.sh 31 > ./mhap.000031.out 2>&1
    ./mhap.sh 32 > ./mhap.000032.out 2>&1
    ./mhap.sh 33 > ./mhap.000033.out 2>&1
    ./mhap.sh 34 > ./mhap.000034.out 2>&1
    ./mhap.sh 35 > ./mhap.000035.out 2>&1
    ./mhap.sh 36 > ./mhap.000036.out 2>&1
    ./mhap.sh 37 > ./mhap.000037.out 2>&1
    ./mhap.sh 38 > ./mhap.000038.out 2>&1
    ./mhap.sh 39 > ./mhap.000039.out 2>&1
    ./mhap.sh 40 > ./mhap.000040.out 2>&1
    ./mhap.sh 41 > ./mhap.000041.out 2>&1
    ./mhap.sh 42 > ./mhap.000042.out 2>&1
    ./mhap.sh 43 > ./mhap.000043.out 2>&1
    ./mhap.sh 44 > ./mhap.000044.out 2>&1
    ./mhap.sh 45 > ./mhap.000045.out 2>&1
    ./mhap.sh 46 > ./mhap.000046.out 2>&1
    ./mhap.sh 47 > ./mhap.000047.out 2>&1
    ./mhap.sh 48 > ./mhap.000048.out 2>&1
    ./mhap.sh 49 > ./mhap.000049.out 2>&1
    ./mhap.sh 50 > ./mhap.000050.out 2>&1
    ./mhap.sh 51 > ./mhap.000051.out 2>&1
    ./mhap.sh 52 > ./mhap.000052.out 2>&1
    ./mhap.sh 53 > ./mhap.000053.out 2>&1
    ./mhap.sh 54 > ./mhap.000054.out 2>&1
    ./mhap.sh 55 > ./mhap.000055.out 2>&1
    ./mhap.sh 56 > ./mhap.000056.out 2>&1
    ./mhap.sh 57 > ./mhap.000057.out 2>&1
    ./mhap.sh 58 > ./mhap.000058.out 2>&1
    ./mhap.sh 59 > ./mhap.000059.out 2>&1
    ./mhap.sh 60 > ./mhap.000060.out 2>&1
    ./mhap.sh 61 > ./mhap.000061.out 2>&1
    ./mhap.sh 62 > ./mhap.000062.out 2>&1
    ./mhap.sh 63 > ./mhap.000063.out 2>&1
    ./mhap.sh 64 > ./mhap.000064.out 2>&1
    ./mhap.sh 65 > ./mhap.000065.out 2>&1
    ./mhap.sh 66 > ./mhap.000066.out 2>&1
    ./mhap.sh 67 > ./mhap.000067.out 2>&1
    ./mhap.sh 68 > ./mhap.000068.out 2>&1
    ./mhap.sh 69 > ./mhap.000069.out 2>&1
    ./mhap.sh 70 > ./mhap.000070.out 2>&1
    ./mhap.sh 71 > ./mhap.000071.out 2>&1
    ./mhap.sh 72 > ./mhap.000072.out 2>&1
    ./mhap.sh 73 > ./mhap.000073.out 2>&1
    ./mhap.sh 74 > ./mhap.000074.out 2>&1
    ./mhap.sh 75 > ./mhap.000075.out 2>&1
    ./mhap.sh 76 > ./mhap.000076.out 2>&1
    ./mhap.sh 77 > ./mhap.000077.out 2>&1
    ./mhap.sh 78 > ./mhap.000078.out 2>&1
    ./mhap.sh 79 > ./mhap.000079.out 2>&1
    ./mhap.sh 80 > ./mhap.000080.out 2>&1
    ./mhap.sh 81 > ./mhap.000081.out 2>&1
    ./mhap.sh 82 > ./mhap.000082.out 2>&1
    ./mhap.sh 83 > ./mhap.000083.out 2>&1
    ./mhap.sh 84 > ./mhap.000084.out 2>&1
    ./mhap.sh 85 > ./mhap.000085.out 2>&1
    ./mhap.sh 86 > ./mhap.000086.out 2>&1
    ./mhap.sh 87 > ./mhap.000087.out 2>&1
    ./mhap.sh 88 > ./mhap.000088.out 2>&1
    ./mhap.sh 89 > ./mhap.000089.out 2>&1
    ./mhap.sh 90 > ./mhap.000090.out 2>&1
    ./mhap.sh 91 > ./mhap.000091.out 2>&1
    ./mhap.sh 92 > ./mhap.000092.out 2>&1
    ./mhap.sh 93 > ./mhap.000093.out 2>&1
    ./mhap.sh 94 > ./mhap.000094.out 2>&1
    ./mhap.sh 95 > ./mhap.000095.out 2>&1
    ./mhap.sh 96 > ./mhap.000096.out 2>&1
    ./mhap.sh 97 > ./mhap.000097.out 2>&1
    ./mhap.sh 98 > ./mhap.000098.out 2>&1
    ./mhap.sh 99 > ./mhap.000099.out 2>&1
    ./mhap.sh 100 > ./mhap.000100.out 2>&1
    ./mhap.sh 101 > ./mhap.000101.out 2>&1
    ./mhap.sh 102 > ./mhap.000102.out 2>&1
    ./mhap.sh 103 > ./mhap.000103.out 2>&1
    ./mhap.sh 104 > ./mhap.000104.out 2>&1
    ./mhap.sh 105 > ./mhap.000105.out 2>&1
    ./mhap.sh 106 > ./mhap.000106.out 2>&1
    ./mhap.sh 107 > ./mhap.000107.out 2>&1
    ./mhap.sh 108 > ./mhap.000108.out 2>&1
    ./mhap.sh 109 > ./mhap.000109.out 2>&1
    ./mhap.sh 110 > ./mhap.000110.out 2>&1
    ./mhap.sh 111 > ./mhap.000111.out 2>&1
    ./mhap.sh 112 > ./mhap.000112.out 2>&1
    ./mhap.sh 113 > ./mhap.000113.out 2>&1
    ./mhap.sh 114 > ./mhap.000114.out 2>&1
    ./mhap.sh 115 > ./mhap.000115.out 2>&1
    ./mhap.sh 116 > ./mhap.000116.out 2>&1
    ./mhap.sh 117 > ./mhap.000117.out 2>&1
    ./mhap.sh 118 > ./mhap.000118.out 2>&1
    ./mhap.sh 119 > ./mhap.000119.out 2>&1
    ./mhap.sh 120 > ./mhap.000120.out 2>&1
    ./mhap.sh 121 > ./mhap.000121.out 2>&1
    ./mhap.sh 122 > ./mhap.000122.out 2>&1
    ./mhap.sh 123 > ./mhap.000123.out 2>&1
    ./mhap.sh 124 > ./mhap.000124.out 2>&1
    ./mhap.sh 125 > ./mhap.000125.out 2>&1
    ./mhap.sh 126 > ./mhap.000126.out 2>&1
    ./mhap.sh 127 > ./mhap.000127.out 2>&1
    ./mhap.sh 128 > ./mhap.000128.out 2>&1
    ./mhap.sh 129 > ./mhap.000129.out 2>&1
    ./mhap.sh 130 > ./mhap.000130.out 2>&1
    ./mhap.sh 131 > ./mhap.000131.out 2>&1
    ./mhap.sh 132 > ./mhap.000132.out 2>&1
    ./mhap.sh 133 > ./mhap.000133.out 2>&1
    ./mhap.sh 134 > ./mhap.000134.out 2>&1
    ./mhap.sh 135 > ./mhap.000135.out 2>&1
    ./mhap.sh 136 > ./mhap.000136.out 2>&1
    ./mhap.sh 137 > ./mhap.000137.out 2>&1
    ./mhap.sh 138 > ./mhap.000138.out 2>&1
    ./mhap.sh 139 > ./mhap.000139.out 2>&1
    ./mhap.sh 140 > ./mhap.000140.out 2>&1
    ./mhap.sh 141 > ./mhap.000141.out 2>&1
    ./mhap.sh 142 > ./mhap.000142.out 2>&1
    ./mhap.sh 143 > ./mhap.000143.out 2>&1
    ./mhap.sh 144 > ./mhap.000144.out 2>&1
    ./mhap.sh 145 > ./mhap.000145.out 2>&1
    ./mhap.sh 146 > ./mhap.000146.out 2>&1
    ./mhap.sh 147 > ./mhap.000147.out 2>&1
    ./mhap.sh 148 > ./mhap.000148.out 2>&1
    ./mhap.sh 149 > ./mhap.000149.out 2>&1
    ./mhap.sh 150 > ./mhap.000150.out 2>&1
    ./mhap.sh 151 > ./mhap.000151.out 2>&1
    ./mhap.sh 152 > ./mhap.000152.out 2>&1
    ./mhap.sh 153 > ./mhap.000153.out 2>&1
    ./mhap.sh 154 > ./mhap.000154.out 2>&1
    ./mhap.sh 155 > ./mhap.000155.out 2>&1
    ./mhap.sh 156 > ./mhap.000156.out 2>&1
    ./mhap.sh 157 > ./mhap.000157.out 2>&1
    ./mhap.sh 158 > ./mhap.000158.out 2>&1
    ./mhap.sh 159 > ./mhap.000159.out 2>&1
    ./mhap.sh 160 > ./mhap.000160.out 2>&1
    ./mhap.sh 161 > ./mhap.000161.out 2>&1
    ./mhap.sh 162 > ./mhap.000162.out 2>&1
    ./mhap.sh 163 > ./mhap.000163.out 2>&1
    ./mhap.sh 164 > ./mhap.000164.out 2>&1
    ./mhap.sh 165 > ./mhap.000165.out 2>&1
    ./mhap.sh 166 > ./mhap.000166.out 2>&1
    ./mhap.sh 167 > ./mhap.000167.out 2>&1
    ./mhap.sh 168 > ./mhap.000168.out 2>&1
    ./mhap.sh 169 > ./mhap.000169.out 2>&1
    ./mhap.sh 170 > ./mhap.000170.out 2>&1
    ./mhap.sh 171 > ./mhap.000171.out 2>&1
    ./mhap.sh 172 > ./mhap.000172.out 2>&1
    ./mhap.sh 173 > ./mhap.000173.out 2>&1
    ./mhap.sh 174 > ./mhap.000174.out 2>&1
    ./mhap.sh 175 > ./mhap.000175.out 2>&1
    ./mhap.sh 176 > ./mhap.000176.out 2>&1
    ./mhap.sh 177 > ./mhap.000177.out 2>&1
    ./mhap.sh 178 > ./mhap.000178.out 2>&1
    ./mhap.sh 179 > ./mhap.000179.out 2>&1
    ./mhap.sh 180 > ./mhap.000180.out 2>&1
    ./mhap.sh 181 > ./mhap.000181.out 2>&1
    ./mhap.sh 182 > ./mhap.000182.out 2>&1
    ./mhap.sh 183 > ./mhap.000183.out 2>&1
    ./mhap.sh 184 > ./mhap.000184.out 2>&1
    ./mhap.sh 185 > ./mhap.000185.out 2>&1
    ./mhap.sh 186 > ./mhap.000186.out 2>&1
    ./mhap.sh 187 > ./mhap.000187.out 2>&1
    ./mhap.sh 188 > ./mhap.000188.out 2>&1
    ./mhap.sh 189 > ./mhap.000189.out 2>&1
    ./mhap.sh 190 > ./mhap.000190.out 2>&1
    ./mhap.sh 191 > ./mhap.000191.out 2>&1
    ./mhap.sh 192 > ./mhap.000192.out 2>&1
    ./mhap.sh 193 > ./mhap.000193.out 2>&1
    ./mhap.sh 194 > ./mhap.000194.out 2>&1
    ./mhap.sh 195 > ./mhap.000195.out 2>&1
    ./mhap.sh 196 > ./mhap.000196.out 2>&1
    ./mhap.sh 197 > ./mhap.000197.out 2>&1
    ./mhap.sh 198 > ./mhap.000198.out 2>&1
    ./mhap.sh 199 > ./mhap.000199.out 2>&1
    ./mhap.sh 200 > ./mhap.000200.out 2>&1
    ./mhap.sh 201 > ./mhap.000201.out 2>&1
    ./mhap.sh 202 > ./mhap.000202.out 2>&1
    ./mhap.sh 203 > ./mhap.000203.out 2>&1
    ./mhap.sh 204 > ./mhap.000204.out 2>&1
    ./mhap.sh 205 > ./mhap.000205.out 2>&1
    ./mhap.sh 206 > ./mhap.000206.out 2>&1
    ./mhap.sh 207 > ./mhap.000207.out 2>&1
    ./mhap.sh 208 > ./mhap.000208.out 2>&1
    ./mhap.sh 209 > ./mhap.000209.out 2>&1
    ./mhap.sh 210 > ./mhap.000210.out 2>&1
    ./mhap.sh 211 > ./mhap.000211.out 2>&1
    ./mhap.sh 212 > ./mhap.000212.out 2>&1
    ./mhap.sh 213 > ./mhap.000213.out 2>&1
    ./mhap.sh 214 > ./mhap.000214.out 2>&1
    ./mhap.sh 215 > ./mhap.000215.out 2>&1
    ./mhap.sh 216 > ./mhap.000216.out 2>&1
    ./mhap.sh 217 > ./mhap.000217.out 2>&1
    ./mhap.sh 218 > ./mhap.000218.out 2>&1
    ./mhap.sh 219 > ./mhap.000219.out 2>&1
    ./mhap.sh 220 > ./mhap.000220.out 2>&1
    ./mhap.sh 221 > ./mhap.000221.out 2>&1
    ./mhap.sh 222 > ./mhap.000222.out 2>&1
    ./mhap.sh 223 > ./mhap.000223.out 2>&1
    ./mhap.sh 224 > ./mhap.000224.out 2>&1
    ./mhap.sh 225 > ./mhap.000225.out 2>&1
    ./mhap.sh 226 > ./mhap.000226.out 2>&1
    ./mhap.sh 227 > ./mhap.000227.out 2>&1
    ./mhap.sh 228 > ./mhap.000228.out 2>&1
    ./mhap.sh 229 > ./mhap.000229.out 2>&1
    ./mhap.sh 230 > ./mhap.000230.out 2>&1
    ./mhap.sh 231 > ./mhap.000231.out 2>&1
    ./mhap.sh 232 > ./mhap.000232.out 2>&1
    ./mhap.sh 233 > ./mhap.000233.out 2>&1
    ./mhap.sh 234 > ./mhap.000234.out 2>&1
    ./mhap.sh 235 > ./mhap.000235.out 2>&1
    ./mhap.sh 236 > ./mhap.000236.out 2>&1
    ./mhap.sh 237 > ./mhap.000237.out 2>&1
    ./mhap.sh 238 > ./mhap.000238.out 2>&1
    ./mhap.sh 239 > ./mhap.000239.out 2>&1
    ./mhap.sh 240 > ./mhap.000240.out 2>&1
    ./mhap.sh 241 > ./mhap.000241.out 2>&1
    ./mhap.sh 242 > ./mhap.000242.out 2>&1
    ./mhap.sh 243 > ./mhap.000243.out 2>&1

-- Finished on Fri Aug 30 03:29:46 2024 (70612 seconds, like watching paint dry) with 5879138.456 GB free disk space
----------------------------------------
--
-- Mhap overlap jobs failed, retry.
--   job correction/1-overlapper/results/000015.ovb FAILED.
--   job correction/1-overlapper/results/000016.ovb FAILED.
--   job correction/1-overlapper/results/000017.ovb FAILED.
--   job correction/1-overlapper/results/000018.ovb FAILED.
--   job correction/1-overlapper/results/000019.ovb FAILED.
--   job correction/1-overlapper/results/000020.ovb FAILED.
--   job correction/1-overlapper/results/000021.ovb FAILED.
--   job correction/1-overlapper/results/000022.ovb FAILED.
--   job correction/1-overlapper/results/000023.ovb FAILED.
--   job correction/1-overlapper/results/000024.ovb FAILED.
--   job correction/1-overlapper/results/000025.ovb FAILED.
--   job correction/1-overlapper/results/000026.ovb FAILED.
--   job correction/1-overlapper/results/000027.ovb FAILED.
--   job correction/1-overlapper/results/000028.ovb FAILED.
--   job correction/1-overlapper/results/000029.ovb FAILED.
--   job correction/1-overlapper/results/000030.ovb FAILED.
--   job correction/1-overlapper/results/000031.ovb FAILED.
--   job correction/1-overlapper/results/000032.ovb FAILED.
--   job correction/1-overlapper/results/000033.ovb FAILED.
--   job correction/1-overlapper/results/000034.ovb FAILED.
--   job correction/1-overlapper/results/000035.ovb FAILED.
--   job correction/1-overlapper/results/000036.ovb FAILED.
--   job correction/1-overlapper/results/000037.ovb FAILED.
--   job correction/1-overlapper/results/000038.ovb FAILED.
--   job correction/1-overlapper/results/000039.ovb FAILED.
--   job correction/1-overlapper/results/000040.ovb FAILED.
--   job correction/1-overlapper/results/000041.ovb FAILED.
--   job correction/1-overlapper/results/000042.ovb FAILED.
--   job correction/1-overlapper/results/000043.ovb FAILED.
--   job correction/1-overlapper/results/000044.ovb FAILED.
--   job correction/1-overlapper/results/000045.ovb FAILED.
--   job correction/1-overlapper/results/000046.ovb FAILED.
--   job correction/1-overlapper/results/000047.ovb FAILED.
--   job correction/1-overlapper/results/000048.ovb FAILED.
--   job correction/1-overlapper/results/000049.ovb FAILED.
--   job correction/1-overlapper/results/000050.ovb FAILED.
--   job correction/1-overlapper/results/000051.ovb FAILED.
--   job correction/1-overlapper/results/000052.ovb FAILED.
--   job correction/1-overlapper/results/000053.ovb FAILED.
--   job correction/1-overlapper/results/000054.ovb FAILED.
--   job correction/1-overlapper/results/000055.ovb FAILED.
--   job correction/1-overlapper/results/000056.ovb FAILED.
--   job correction/1-overlapper/results/000057.ovb FAILED.
--   job correction/1-overlapper/results/000058.ovb FAILED.
--   job correction/1-overlapper/results/000059.ovb FAILED.
--   job correction/1-overlapper/results/000060.ovb FAILED.
--   job correction/1-overlapper/results/000061.ovb FAILED.
--   job correction/1-overlapper/results/000062.ovb FAILED.
--   job correction/1-overlapper/results/000063.ovb FAILED.
--   job correction/1-overlapper/results/000064.ovb FAILED.
--   job correction/1-overlapper/results/000065.ovb FAILED.
--   job correction/1-overlapper/results/000066.ovb FAILED.
--   job correction/1-overlapper/results/000067.ovb FAILED.
--   job correction/1-overlapper/results/000068.ovb FAILED.
--   job correction/1-overlapper/results/000069.ovb FAILED.
--   job correction/1-overlapper/results/000070.ovb FAILED.
--   job correction/1-overlapper/results/000071.ovb FAILED.
--   job correction/1-overlapper/results/000072.ovb FAILED.
--   job correction/1-overlapper/results/000073.ovb FAILED.
--   job correction/1-overlapper/results/000074.ovb FAILED.
--   job correction/1-overlapper/results/000075.ovb FAILED.
--   job correction/1-overlapper/results/000076.ovb FAILED.
--   job correction/1-overlapper/results/000077.ovb FAILED.
--   job correction/1-overlapper/results/000078.ovb FAILED.
--   job correction/1-overlapper/results/000079.ovb FAILED.
--   job correction/1-overlapper/results/000080.ovb FAILED.
--   job correction/1-overlapper/results/000081.ovb FAILED.
--   job correction/1-overlapper/results/000082.ovb FAILED.
--   job correction/1-overlapper/results/000083.ovb FAILED.
--   job correction/1-overlapper/results/000084.ovb FAILED.
--   job correction/1-overlapper/results/000085.ovb FAILED.
--   job correction/1-overlapper/results/000086.ovb FAILED.
--   job correction/1-overlapper/results/000087.ovb FAILED.
--   job correction/1-overlapper/results/000088.ovb FAILED.
--   job correction/1-overlapper/results/000089.ovb FAILED.
--   job correction/1-overlapper/results/000090.ovb FAILED.
--   job correction/1-overlapper/results/000091.ovb FAILED.
--   job correction/1-overlapper/results/000092.ovb FAILED.
--   job correction/1-overlapper/results/000093.ovb FAILED.
--   job correction/1-overlapper/results/000094.ovb FAILED.
--   job correction/1-overlapper/results/000095.ovb FAILED.
--   job correction/1-overlapper/results/000096.ovb FAILED.
--   job correction/1-overlapper/results/000097.ovb FAILED.
--   job correction/1-overlapper/results/000098.ovb FAILED.
--   job correction/1-overlapper/results/000099.ovb FAILED.
--   job correction/1-overlapper/results/000100.ovb FAILED.
--   job correction/1-overlapper/results/000101.ovb FAILED.
--   job correction/1-overlapper/results/000102.ovb FAILED.
--   job correction/1-overlapper/results/000103.ovb FAILED.
--   job correction/1-overlapper/results/000104.ovb FAILED.
--   job correction/1-overlapper/results/000105.ovb FAILED.
--   job correction/1-overlapper/results/000106.ovb FAILED.
--   job correction/1-overlapper/results/000107.ovb FAILED.
--   job correction/1-overlapper/results/000108.ovb FAILED.
--   job correction/1-overlapper/results/000109.ovb FAILED.
--   job correction/1-overlapper/results/000110.ovb FAILED.
--   job correction/1-overlapper/results/000111.ovb FAILED.
--   job correction/1-overlapper/results/000112.ovb FAILED.
--   job correction/1-overlapper/results/000113.ovb FAILED.
--   job correction/1-overlapper/results/000114.ovb FAILED.
--   job correction/1-overlapper/results/000115.ovb FAILED.
--   job correction/1-overlapper/results/000116.ovb FAILED.
--   job correction/1-overlapper/results/000117.ovb FAILED.
--   job correction/1-overlapper/results/000118.ovb FAILED.
--   job correction/1-overlapper/results/000119.ovb FAILED.
--   job correction/1-overlapper/results/000120.ovb FAILED.
--   job correction/1-overlapper/results/000121.ovb FAILED.
--   job correction/1-overlapper/results/000122.ovb FAILED.
--   job correction/1-overlapper/results/000123.ovb FAILED.
--   job correction/1-overlapper/results/000124.ovb FAILED.
--   job correction/1-overlapper/results/000125.ovb FAILED.
--   job correction/1-overlapper/results/000126.ovb FAILED.
--   job correction/1-overlapper/results/000127.ovb FAILED.
--   job correction/1-overlapper/results/000128.ovb FAILED.
--   job correction/1-overlapper/results/000129.ovb FAILED.
--   job correction/1-overlapper/results/000130.ovb FAILED.
--   job correction/1-overlapper/results/000131.ovb FAILED.
--   job correction/1-overlapper/results/000132.ovb FAILED.
--   job correction/1-overlapper/results/000133.ovb FAILED.
--   job correction/1-overlapper/results/000134.ovb FAILED.
--   job correction/1-overlapper/results/000135.ovb FAILED.
--   job correction/1-overlapper/results/000136.ovb FAILED.
--   job correction/1-overlapper/results/000137.ovb FAILED.
--   job correction/1-overlapper/results/000138.ovb FAILED.
--   job correction/1-overlapper/results/000139.ovb FAILED.
--   job correction/1-overlapper/results/000140.ovb FAILED.
--   job correction/1-overlapper/results/000141.ovb FAILED.
--   job correction/1-overlapper/results/000142.ovb FAILED.
--   job correction/1-overlapper/results/000143.ovb FAILED.
--   job correction/1-overlapper/results/000144.ovb FAILED.
--   job correction/1-overlapper/results/000145.ovb FAILED.
--   job correction/1-overlapper/results/000146.ovb FAILED.
--   job correction/1-overlapper/results/000147.ovb FAILED.
--   job correction/1-overlapper/results/000148.ovb FAILED.
--   job correction/1-overlapper/results/000149.ovb FAILED.
--   job correction/1-overlapper/results/000150.ovb FAILED.
--   job correction/1-overlapper/results/000151.ovb FAILED.
--   job correction/1-overlapper/results/000152.ovb FAILED.
--   job correction/1-overlapper/results/000153.ovb FAILED.
--   job correction/1-overlapper/results/000154.ovb FAILED.
--   job correction/1-overlapper/results/000155.ovb FAILED.
--   job correction/1-overlapper/results/000156.ovb FAILED.
--   job correction/1-overlapper/results/000157.ovb FAILED.
--   job correction/1-overlapper/results/000158.ovb FAILED.
--   job correction/1-overlapper/results/000159.ovb FAILED.
--   job correction/1-overlapper/results/000160.ovb FAILED.
--   job correction/1-overlapper/results/000161.ovb FAILED.
--   job correction/1-overlapper/results/000162.ovb FAILED.
--   job correction/1-overlapper/results/000163.ovb FAILED.
--   job correction/1-overlapper/results/000164.ovb FAILED.
--   job correction/1-overlapper/results/000165.ovb FAILED.
--   job correction/1-overlapper/results/000166.ovb FAILED.
--   job correction/1-overlapper/results/000167.ovb FAILED.
--   job correction/1-overlapper/results/000168.ovb FAILED.
--   job correction/1-overlapper/results/000169.ovb FAILED.
--   job correction/1-overlapper/results/000170.ovb FAILED.
--   job correction/1-overlapper/results/000171.ovb FAILED.
--   job correction/1-overlapper/results/000172.ovb FAILED.
--   job correction/1-overlapper/results/000173.ovb FAILED.
--   job correction/1-overlapper/results/000174.ovb FAILED.
--   job correction/1-overlapper/results/000175.ovb FAILED.
--   job correction/1-overlapper/results/000176.ovb FAILED.
--   job correction/1-overlapper/results/000177.ovb FAILED.
--   job correction/1-overlapper/results/000178.ovb FAILED.
--   job correction/1-overlapper/results/000179.ovb FAILED.
--   job correction/1-overlapper/results/000180.ovb FAILED.
--   job correction/1-overlapper/results/000181.ovb FAILED.
--   job correction/1-overlapper/results/000182.ovb FAILED.
--   job correction/1-overlapper/results/000183.ovb FAILED.
--   job correction/1-overlapper/results/000184.ovb FAILED.
--   job correction/1-overlapper/results/000185.ovb FAILED.
--   job correction/1-overlapper/results/000186.ovb FAILED.
--   job correction/1-overlapper/results/000187.ovb FAILED.
--   job correction/1-overlapper/results/000188.ovb FAILED.
--   job correction/1-overlapper/results/000189.ovb FAILED.
--   job correction/1-overlapper/results/000190.ovb FAILED.
--   job correction/1-overlapper/results/000191.ovb FAILED.
--   job correction/1-overlapper/results/000192.ovb FAILED.
--   job correction/1-overlapper/results/000193.ovb FAILED.
--   job correction/1-overlapper/results/000194.ovb FAILED.
--   job correction/1-overlapper/results/000195.ovb FAILED.
--   job correction/1-overlapper/results/000196.ovb FAILED.
--   job correction/1-overlapper/results/000197.ovb FAILED.
--   job correction/1-overlapper/results/000198.ovb FAILED.
--   job correction/1-overlapper/results/000199.ovb FAILED.
--   job correction/1-overlapper/results/000200.ovb FAILED.
--   job correction/1-overlapper/results/000201.ovb FAILED.
--   job correction/1-overlapper/results/000202.ovb FAILED.
--   job correction/1-overlapper/results/000203.ovb FAILED.
--   job correction/1-overlapper/results/000204.ovb FAILED.
--   job correction/1-overlapper/results/000205.ovb FAILED.
--   job correction/1-overlapper/results/000206.ovb FAILED.
--   job correction/1-overlapper/results/000207.ovb FAILED.
--   job correction/1-overlapper/results/000208.ovb FAILED.
--   job correction/1-overlapper/results/000209.ovb FAILED.
--   job correction/1-overlapper/results/000210.ovb FAILED.
--   job correction/1-overlapper/results/000211.ovb FAILED.
--   job correction/1-overlapper/results/000212.ovb FAILED.
--   job correction/1-overlapper/results/000213.ovb FAILED.
--   job correction/1-overlapper/results/000214.ovb FAILED.
--   job correction/1-overlapper/results/000215.ovb FAILED.
--   job correction/1-overlapper/results/000216.ovb FAILED.
--   job correction/1-overlapper/results/000217.ovb FAILED.
--   job correction/1-overlapper/results/000218.ovb FAILED.
--   job correction/1-overlapper/results/000219.ovb FAILED.
--   job correction/1-overlapper/results/000220.ovb FAILED.
--   job correction/1-overlapper/results/000221.ovb FAILED.
--   job correction/1-overlapper/results/000222.ovb FAILED.
--   job correction/1-overlapper/results/000223.ovb FAILED.
--   job correction/1-overlapper/results/000224.ovb FAILED.
--   job correction/1-overlapper/results/000225.ovb FAILED.
--   job correction/1-overlapper/results/000226.ovb FAILED.
--   job correction/1-overlapper/results/000227.ovb FAILED.
--   job correction/1-overlapper/results/000228.ovb FAILED.
--   job correction/1-overlapper/results/000229.ovb FAILED.
--   job correction/1-overlapper/results/000230.ovb FAILED.
--   job correction/1-overlapper/results/000231.ovb FAILED.
--   job correction/1-overlapper/results/000232.ovb FAILED.
--   job correction/1-overlapper/results/000233.ovb FAILED.
--   job correction/1-overlapper/results/000234.ovb FAILED.
--   job correction/1-overlapper/results/000235.ovb FAILED.
--   job correction/1-overlapper/results/000236.ovb FAILED.
--   job correction/1-overlapper/results/000237.ovb FAILED.
--   job correction/1-overlapper/results/000238.ovb FAILED.
--   job correction/1-overlapper/results/000239.ovb FAILED.
--   job correction/1-overlapper/results/000240.ovb FAILED.
--   job correction/1-overlapper/results/000241.ovb FAILED.
--   job correction/1-overlapper/results/000242.ovb FAILED.
--   job correction/1-overlapper/results/000243.ovb FAILED.
--
--
-- Running jobs.  Second attempt out of 2.
----------------------------------------
-- Starting 'cormhap' concurrent execution on Fri Aug 30 03:29:46 2024 with 5879138.456 GB free disk space (229 processes; 3 concurrently)

    cd correction/1-overlapper
    ./mhap.sh 15 > ./mhap.000015.out 2>&1
    ./mhap.sh 16 > ./mhap.000016.out 2>&1
    ./mhap.sh 17 > ./mhap.000017.out 2>&1
    ./mhap.sh 18 > ./mhap.000018.out 2>&1
    ./mhap.sh 19 > ./mhap.000019.out 2>&1
    ./mhap.sh 20 > ./mhap.000020.out 2>&1
    ./mhap.sh 21 > ./mhap.000021.out 2>&1
    ./mhap.sh 22 > ./mhap.000022.out 2>&1
    ./mhap.sh 23 > ./mhap.000023.out 2>&1
    ./mhap.sh 24 > ./mhap.000024.out 2>&1
    ./mhap.sh 25 > ./mhap.000025.out 2>&1
    ./mhap.sh 26 > ./mhap.000026.out 2>&1
    ./mhap.sh 27 > ./mhap.000027.out 2>&1
    ./mhap.sh 28 > ./mhap.000028.out 2>&1
    ./mhap.sh 29 > ./mhap.000029.out 2>&1
    ./mhap.sh 30 > ./mhap.000030.out 2>&1
    ./mhap.sh 31 > ./mhap.000031.out 2>&1
    ./mhap.sh 32 > ./mhap.000032.out 2>&1
    ./mhap.sh 33 > ./mhap.000033.out 2>&1
    ./mhap.sh 34 > ./mhap.000034.out 2>&1
    ./mhap.sh 35 > ./mhap.000035.out 2>&1
    ./mhap.sh 36 > ./mhap.000036.out 2>&1
    ./mhap.sh 37 > ./mhap.000037.out 2>&1
    ./mhap.sh 38 > ./mhap.000038.out 2>&1
    ./mhap.sh 39 > ./mhap.000039.out 2>&1
    ./mhap.sh 40 > ./mhap.000040.out 2>&1
    ./mhap.sh 41 > ./mhap.000041.out 2>&1
    ./mhap.sh 42 > ./mhap.000042.out 2>&1
    ./mhap.sh 43 > ./mhap.000043.out 2>&1
    ./mhap.sh 44 > ./mhap.000044.out 2>&1
    ./mhap.sh 45 > ./mhap.000045.out 2>&1
    ./mhap.sh 46 > ./mhap.000046.out 2>&1
    ./mhap.sh 47 > ./mhap.000047.out 2>&1
    ./mhap.sh 48 > ./mhap.000048.out 2>&1
    ./mhap.sh 49 > ./mhap.000049.out 2>&1
    ./mhap.sh 50 > ./mhap.000050.out 2>&1
    ./mhap.sh 51 > ./mhap.000051.out 2>&1
    ./mhap.sh 52 > ./mhap.000052.out 2>&1
    ./mhap.sh 53 > ./mhap.000053.out 2>&1
    ./mhap.sh 54 > ./mhap.000054.out 2>&1
    ./mhap.sh 55 > ./mhap.000055.out 2>&1
    ./mhap.sh 56 > ./mhap.000056.out 2>&1
    ./mhap.sh 57 > ./mhap.000057.out 2>&1
    ./mhap.sh 58 > ./mhap.000058.out 2>&1
    ./mhap.sh 59 > ./mhap.000059.out 2>&1
    ./mhap.sh 60 > ./mhap.000060.out 2>&1
    ./mhap.sh 61 > ./mhap.000061.out 2>&1
    ./mhap.sh 62 > ./mhap.000062.out 2>&1
    ./mhap.sh 63 > ./mhap.000063.out 2>&1
    ./mhap.sh 64 > ./mhap.000064.out 2>&1
    ./mhap.sh 65 > ./mhap.000065.out 2>&1
    ./mhap.sh 66 > ./mhap.000066.out 2>&1
    ./mhap.sh 67 > ./mhap.000067.out 2>&1
    ./mhap.sh 68 > ./mhap.000068.out 2>&1
    ./mhap.sh 69 > ./mhap.000069.out 2>&1
    ./mhap.sh 70 > ./mhap.000070.out 2>&1
    ./mhap.sh 71 > ./mhap.000071.out 2>&1
    ./mhap.sh 72 > ./mhap.000072.out 2>&1
    ./mhap.sh 73 > ./mhap.000073.out 2>&1
    ./mhap.sh 74 > ./mhap.000074.out 2>&1
    ./mhap.sh 75 > ./mhap.000075.out 2>&1
    ./mhap.sh 76 > ./mhap.000076.out 2>&1
    ./mhap.sh 77 > ./mhap.000077.out 2>&1
    ./mhap.sh 78 > ./mhap.000078.out 2>&1
    ./mhap.sh 79 > ./mhap.000079.out 2>&1
    ./mhap.sh 80 > ./mhap.000080.out 2>&1
    ./mhap.sh 81 > ./mhap.000081.out 2>&1
    ./mhap.sh 82 > ./mhap.000082.out 2>&1
    ./mhap.sh 83 > ./mhap.000083.out 2>&1
    ./mhap.sh 84 > ./mhap.000084.out 2>&1
    ./mhap.sh 85 > ./mhap.000085.out 2>&1
    ./mhap.sh 86 > ./mhap.000086.out 2>&1
    ./mhap.sh 87 > ./mhap.000087.out 2>&1
    ./mhap.sh 88 > ./mhap.000088.out 2>&1
    ./mhap.sh 89 > ./mhap.000089.out 2>&1
    ./mhap.sh 90 > ./mhap.000090.out 2>&1
    ./mhap.sh 91 > ./mhap.000091.out 2>&1
    ./mhap.sh 92 > ./mhap.000092.out 2>&1
    ./mhap.sh 93 > ./mhap.000093.out 2>&1
    ./mhap.sh 94 > ./mhap.000094.out 2>&1
    ./mhap.sh 95 > ./mhap.000095.out 2>&1
    ./mhap.sh 96 > ./mhap.000096.out 2>&1
    ./mhap.sh 97 > ./mhap.000097.out 2>&1
    ./mhap.sh 98 > ./mhap.000098.out 2>&1
    ./mhap.sh 99 > ./mhap.000099.out 2>&1
    ./mhap.sh 100 > ./mhap.000100.out 2>&1
    ./mhap.sh 101 > ./mhap.000101.out 2>&1
    ./mhap.sh 102 > ./mhap.000102.out 2>&1
    ./mhap.sh 103 > ./mhap.000103.out 2>&1
    ./mhap.sh 104 > ./mhap.000104.out 2>&1
    ./mhap.sh 105 > ./mhap.000105.out 2>&1
    ./mhap.sh 106 > ./mhap.000106.out 2>&1
    ./mhap.sh 107 > ./mhap.000107.out 2>&1
    ./mhap.sh 108 > ./mhap.000108.out 2>&1
    ./mhap.sh 109 > ./mhap.000109.out 2>&1
    ./mhap.sh 110 > ./mhap.000110.out 2>&1
    ./mhap.sh 111 > ./mhap.000111.out 2>&1
    ./mhap.sh 112 > ./mhap.000112.out 2>&1
    ./mhap.sh 113 > ./mhap.000113.out 2>&1
    ./mhap.sh 114 > ./mhap.000114.out 2>&1
    ./mhap.sh 115 > ./mhap.000115.out 2>&1
    ./mhap.sh 116 > ./mhap.000116.out 2>&1
    ./mhap.sh 117 > ./mhap.000117.out 2>&1
    ./mhap.sh 118 > ./mhap.000118.out 2>&1
    ./mhap.sh 119 > ./mhap.000119.out 2>&1
    ./mhap.sh 120 > ./mhap.000120.out 2>&1
    ./mhap.sh 121 > ./mhap.000121.out 2>&1
    ./mhap.sh 122 > ./mhap.000122.out 2>&1
    ./mhap.sh 123 > ./mhap.000123.out 2>&1
    ./mhap.sh 124 > ./mhap.000124.out 2>&1
    ./mhap.sh 125 > ./mhap.000125.out 2>&1
    ./mhap.sh 126 > ./mhap.000126.out 2>&1
    ./mhap.sh 127 > ./mhap.000127.out 2>&1
    ./mhap.sh 128 > ./mhap.000128.out 2>&1
    ./mhap.sh 129 > ./mhap.000129.out 2>&1
    ./mhap.sh 130 > ./mhap.000130.out 2>&1
    ./mhap.sh 131 > ./mhap.000131.out 2>&1
    ./mhap.sh 132 > ./mhap.000132.out 2>&1
    ./mhap.sh 133 > ./mhap.000133.out 2>&1
    ./mhap.sh 134 > ./mhap.000134.out 2>&1
    ./mhap.sh 135 > ./mhap.000135.out 2>&1
    ./mhap.sh 136 > ./mhap.000136.out 2>&1
    ./mhap.sh 137 > ./mhap.000137.out 2>&1
    ./mhap.sh 138 > ./mhap.000138.out 2>&1
    ./mhap.sh 139 > ./mhap.000139.out 2>&1
    ./mhap.sh 140 > ./mhap.000140.out 2>&1
    ./mhap.sh 141 > ./mhap.000141.out 2>&1
    ./mhap.sh 142 > ./mhap.000142.out 2>&1
    ./mhap.sh 143 > ./mhap.000143.out 2>&1
    ./mhap.sh 144 > ./mhap.000144.out 2>&1
    ./mhap.sh 145 > ./mhap.000145.out 2>&1
    ./mhap.sh 146 > ./mhap.000146.out 2>&1
    ./mhap.sh 147 > ./mhap.000147.out 2>&1
    ./mhap.sh 148 > ./mhap.000148.out 2>&1
    ./mhap.sh 149 > ./mhap.000149.out 2>&1
    ./mhap.sh 150 > ./mhap.000150.out 2>&1
    ./mhap.sh 151 > ./mhap.000151.out 2>&1
    ./mhap.sh 152 > ./mhap.000152.out 2>&1
    ./mhap.sh 153 > ./mhap.000153.out 2>&1
    ./mhap.sh 154 > ./mhap.000154.out 2>&1
    ./mhap.sh 155 > ./mhap.000155.out 2>&1
    ./mhap.sh 156 > ./mhap.000156.out 2>&1
    ./mhap.sh 157 > ./mhap.000157.out 2>&1
    ./mhap.sh 158 > ./mhap.000158.out 2>&1
    ./mhap.sh 159 > ./mhap.000159.out 2>&1
    ./mhap.sh 160 > ./mhap.000160.out 2>&1
    ./mhap.sh 161 > ./mhap.000161.out 2>&1
    ./mhap.sh 162 > ./mhap.000162.out 2>&1
    ./mhap.sh 163 > ./mhap.000163.out 2>&1
    ./mhap.sh 164 > ./mhap.000164.out 2>&1
    ./mhap.sh 165 > ./mhap.000165.out 2>&1
    ./mhap.sh 166 > ./mhap.000166.out 2>&1
    ./mhap.sh 167 > ./mhap.000167.out 2>&1
    ./mhap.sh 168 > ./mhap.000168.out 2>&1
    ./mhap.sh 169 > ./mhap.000169.out 2>&1
    ./mhap.sh 170 > ./mhap.000170.out 2>&1
    ./mhap.sh 171 > ./mhap.000171.out 2>&1
    ./mhap.sh 172 > ./mhap.000172.out 2>&1
    ./mhap.sh 173 > ./mhap.000173.out 2>&1
    ./mhap.sh 174 > ./mhap.000174.out 2>&1
    ./mhap.sh 175 > ./mhap.000175.out 2>&1
    ./mhap.sh 176 > ./mhap.000176.out 2>&1
    ./mhap.sh 177 > ./mhap.000177.out 2>&1
    ./mhap.sh 178 > ./mhap.000178.out 2>&1
    ./mhap.sh 179 > ./mhap.000179.out 2>&1
    ./mhap.sh 180 > ./mhap.000180.out 2>&1
    ./mhap.sh 181 > ./mhap.000181.out 2>&1
    ./mhap.sh 182 > ./mhap.000182.out 2>&1
    ./mhap.sh 183 > ./mhap.000183.out 2>&1
    ./mhap.sh 184 > ./mhap.000184.out 2>&1
    ./mhap.sh 185 > ./mhap.000185.out 2>&1
    ./mhap.sh 186 > ./mhap.000186.out 2>&1
    ./mhap.sh 187 > ./mhap.000187.out 2>&1
    ./mhap.sh 188 > ./mhap.000188.out 2>&1
    ./mhap.sh 189 > ./mhap.000189.out 2>&1
    ./mhap.sh 190 > ./mhap.000190.out 2>&1
    ./mhap.sh 191 > ./mhap.000191.out 2>&1
    ./mhap.sh 192 > ./mhap.000192.out 2>&1
    ./mhap.sh 193 > ./mhap.000193.out 2>&1
    ./mhap.sh 194 > ./mhap.000194.out 2>&1
    ./mhap.sh 195 > ./mhap.000195.out 2>&1
    ./mhap.sh 196 > ./mhap.000196.out 2>&1
    ./mhap.sh 197 > ./mhap.000197.out 2>&1
    ./mhap.sh 198 > ./mhap.000198.out 2>&1
    ./mhap.sh 199 > ./mhap.000199.out 2>&1
    ./mhap.sh 200 > ./mhap.000200.out 2>&1
    ./mhap.sh 201 > ./mhap.000201.out 2>&1
    ./mhap.sh 202 > ./mhap.000202.out 2>&1
    ./mhap.sh 203 > ./mhap.000203.out 2>&1
    ./mhap.sh 204 > ./mhap.000204.out 2>&1
    ./mhap.sh 205 > ./mhap.000205.out 2>&1
    ./mhap.sh 206 > ./mhap.000206.out 2>&1
    ./mhap.sh 207 > ./mhap.000207.out 2>&1
    ./mhap.sh 208 > ./mhap.000208.out 2>&1
    ./mhap.sh 209 > ./mhap.000209.out 2>&1
    ./mhap.sh 210 > ./mhap.000210.out 2>&1
    ./mhap.sh 211 > ./mhap.000211.out 2>&1
    ./mhap.sh 212 > ./mhap.000212.out 2>&1
    ./mhap.sh 213 > ./mhap.000213.out 2>&1
    ./mhap.sh 214 > ./mhap.000214.out 2>&1
    ./mhap.sh 215 > ./mhap.000215.out 2>&1
    ./mhap.sh 216 > ./mhap.000216.out 2>&1
    ./mhap.sh 217 > ./mhap.000217.out 2>&1
    ./mhap.sh 218 > ./mhap.000218.out 2>&1
    ./mhap.sh 219 > ./mhap.000219.out 2>&1
    ./mhap.sh 220 > ./mhap.000220.out 2>&1
    ./mhap.sh 221 > ./mhap.000221.out 2>&1
    ./mhap.sh 222 > ./mhap.000222.out 2>&1
    ./mhap.sh 223 > ./mhap.000223.out 2>&1
    ./mhap.sh 224 > ./mhap.000224.out 2>&1
    ./mhap.sh 225 > ./mhap.000225.out 2>&1
    ./mhap.sh 226 > ./mhap.000226.out 2>&1
    ./mhap.sh 227 > ./mhap.000227.out 2>&1
    ./mhap.sh 228 > ./mhap.000228.out 2>&1
    ./mhap.sh 229 > ./mhap.000229.out 2>&1
    ./mhap.sh 230 > ./mhap.000230.out 2>&1
    ./mhap.sh 231 > ./mhap.000231.out 2>&1
    ./mhap.sh 232 > ./mhap.000232.out 2>&1
    ./mhap.sh 233 > ./mhap.000233.out 2>&1
    ./mhap.sh 234 > ./mhap.000234.out 2>&1
    ./mhap.sh 235 > ./mhap.000235.out 2>&1
    ./mhap.sh 236 > ./mhap.000236.out 2>&1
    ./mhap.sh 237 > ./mhap.000237.out 2>&1
    ./mhap.sh 238 > ./mhap.000238.out 2>&1
    ./mhap.sh 239 > ./mhap.000239.out 2>&1
    ./mhap.sh 240 > ./mhap.000240.out 2>&1
    ./mhap.sh 241 > ./mhap.000241.out 2>&1
    ./mhap.sh 242 > ./mhap.000242.out 2>&1
    ./mhap.sh 243 > ./mhap.000243.out 2>&1

-- Finished on Fri Aug 30 03:29:53 2024 (7 seconds) with 5879138.347 GB free disk space
----------------------------------------
--
-- Mhap overlap jobs failed, tried 2 times, giving up.
--   job correction/1-overlapper/results/000015.ovb FAILED.
--   job correction/1-overlapper/results/000016.ovb FAILED.
--   job correction/1-overlapper/results/000017.ovb FAILED.
--   job correction/1-overlapper/results/000018.ovb FAILED.
--   job correction/1-overlapper/results/000019.ovb FAILED.
--   job correction/1-overlapper/results/000020.ovb FAILED.
--   job correction/1-overlapper/results/000021.ovb FAILED.
--   job correction/1-overlapper/results/000022.ovb FAILED.
--   job correction/1-overlapper/results/000023.ovb FAILED.
--   job correction/1-overlapper/results/000024.ovb FAILED.
--   job correction/1-overlapper/results/000025.ovb FAILED.
--   job correction/1-overlapper/results/000026.ovb FAILED.
--   job correction/1-overlapper/results/000027.ovb FAILED.
--   job correction/1-overlapper/results/000028.ovb FAILED.
--   job correction/1-overlapper/results/000029.ovb FAILED.
--   job correction/1-overlapper/results/000030.ovb FAILED.
--   job correction/1-overlapper/results/000031.ovb FAILED.
--   job correction/1-overlapper/results/000032.ovb FAILED.
--   job correction/1-overlapper/results/000033.ovb FAILED.
--   job correction/1-overlapper/results/000034.ovb FAILED.
--   job correction/1-overlapper/results/000035.ovb FAILED.
--   job correction/1-overlapper/results/000036.ovb FAILED.
--   job correction/1-overlapper/results/000037.ovb FAILED.
--   job correction/1-overlapper/results/000038.ovb FAILED.
--   job correction/1-overlapper/results/000039.ovb FAILED.
--   job correction/1-overlapper/results/000040.ovb FAILED.
--   job correction/1-overlapper/results/000041.ovb FAILED.
--   job correction/1-overlapper/results/000042.ovb FAILED.
--   job correction/1-overlapper/results/000043.ovb FAILED.
--   job correction/1-overlapper/results/000044.ovb FAILED.
--   job correction/1-overlapper/results/000045.ovb FAILED.
--   job correction/1-overlapper/results/000046.ovb FAILED.
--   job correction/1-overlapper/results/000047.ovb FAILED.
--   job correction/1-overlapper/results/000048.ovb FAILED.
--   job correction/1-overlapper/results/000049.ovb FAILED.
--   job correction/1-overlapper/results/000050.ovb FAILED.
--   job correction/1-overlapper/results/000051.ovb FAILED.
--   job correction/1-overlapper/results/000052.ovb FAILED.
--   job correction/1-overlapper/results/000053.ovb FAILED.
--   job correction/1-overlapper/results/000054.ovb FAILED.
--   job correction/1-overlapper/results/000055.ovb FAILED.
--   job correction/1-overlapper/results/000056.ovb FAILED.
--   job correction/1-overlapper/results/000057.ovb FAILED.
--   job correction/1-overlapper/results/000058.ovb FAILED.
--   job correction/1-overlapper/results/000059.ovb FAILED.
--   job correction/1-overlapper/results/000060.ovb FAILED.
--   job correction/1-overlapper/results/000061.ovb FAILED.
--   job correction/1-overlapper/results/000062.ovb FAILED.
--   job correction/1-overlapper/results/000063.ovb FAILED.
--   job correction/1-overlapper/results/000064.ovb FAILED.
--   job correction/1-overlapper/results/000065.ovb FAILED.
--   job correction/1-overlapper/results/000066.ovb FAILED.
--   job correction/1-overlapper/results/000067.ovb FAILED.
--   job correction/1-overlapper/results/000068.ovb FAILED.
--   job correction/1-overlapper/results/000069.ovb FAILED.
--   job correction/1-overlapper/results/000070.ovb FAILED.
--   job correction/1-overlapper/results/000071.ovb FAILED.
--   job correction/1-overlapper/results/000072.ovb FAILED.
--   job correction/1-overlapper/results/000073.ovb FAILED.
--   job correction/1-overlapper/results/000074.ovb FAILED.
--   job correction/1-overlapper/results/000075.ovb FAILED.
--   job correction/1-overlapper/results/000076.ovb FAILED.
--   job correction/1-overlapper/results/000077.ovb FAILED.
--   job correction/1-overlapper/results/000078.ovb FAILED.
--   job correction/1-overlapper/results/000079.ovb FAILED.
--   job correction/1-overlapper/results/000080.ovb FAILED.
--   job correction/1-overlapper/results/000081.ovb FAILED.
--   job correction/1-overlapper/results/000082.ovb FAILED.
--   job correction/1-overlapper/results/000083.ovb FAILED.
--   job correction/1-overlapper/results/000084.ovb FAILED.
--   job correction/1-overlapper/results/000085.ovb FAILED.
--   job correction/1-overlapper/results/000086.ovb FAILED.
--   job correction/1-overlapper/results/000087.ovb FAILED.
--   job correction/1-overlapper/results/000088.ovb FAILED.
--   job correction/1-overlapper/results/000089.ovb FAILED.
--   job correction/1-overlapper/results/000090.ovb FAILED.
--   job correction/1-overlapper/results/000091.ovb FAILED.
--   job correction/1-overlapper/results/000092.ovb FAILED.
--   job correction/1-overlapper/results/000093.ovb FAILED.
--   job correction/1-overlapper/results/000094.ovb FAILED.
--   job correction/1-overlapper/results/000095.ovb FAILED.
--   job correction/1-overlapper/results/000096.ovb FAILED.
--   job correction/1-overlapper/results/000097.ovb FAILED.
--   job correction/1-overlapper/results/000098.ovb FAILED.
--   job correction/1-overlapper/results/000099.ovb FAILED.
--   job correction/1-overlapper/results/000100.ovb FAILED.
--   job correction/1-overlapper/results/000101.ovb FAILED.
--   job correction/1-overlapper/results/000102.ovb FAILED.
--   job correction/1-overlapper/results/000103.ovb FAILED.
--   job correction/1-overlapper/results/000104.ovb FAILED.
--   job correction/1-overlapper/results/000105.ovb FAILED.
--   job correction/1-overlapper/results/000106.ovb FAILED.
--   job correction/1-overlapper/results/000107.ovb FAILED.
--   job correction/1-overlapper/results/000108.ovb FAILED.
--   job correction/1-overlapper/results/000109.ovb FAILED.
--   job correction/1-overlapper/results/000110.ovb FAILED.
--   job correction/1-overlapper/results/000111.ovb FAILED.
--   job correction/1-overlapper/results/000112.ovb FAILED.
--   job correction/1-overlapper/results/000113.ovb FAILED.
--   job correction/1-overlapper/results/000114.ovb FAILED.
--   job correction/1-overlapper/results/000115.ovb FAILED.
--   job correction/1-overlapper/results/000116.ovb FAILED.
--   job correction/1-overlapper/results/000117.ovb FAILED.
--   job correction/1-overlapper/results/000118.ovb FAILED.
--   job correction/1-overlapper/results/000119.ovb FAILED.
--   job correction/1-overlapper/results/000120.ovb FAILED.
--   job correction/1-overlapper/results/000121.ovb FAILED.
--   job correction/1-overlapper/results/000122.ovb FAILED.
--   job correction/1-overlapper/results/000123.ovb FAILED.
--   job correction/1-overlapper/results/000124.ovb FAILED.
--   job correction/1-overlapper/results/000125.ovb FAILED.
--   job correction/1-overlapper/results/000126.ovb FAILED.
--   job correction/1-overlapper/results/000127.ovb FAILED.
--   job correction/1-overlapper/results/000128.ovb FAILED.
--   job correction/1-overlapper/results/000129.ovb FAILED.
--   job correction/1-overlapper/results/000130.ovb FAILED.
--   job correction/1-overlapper/results/000131.ovb FAILED.
--   job correction/1-overlapper/results/000132.ovb FAILED.
--   job correction/1-overlapper/results/000133.ovb FAILED.
--   job correction/1-overlapper/results/000134.ovb FAILED.
--   job correction/1-overlapper/results/000135.ovb FAILED.
--   job correction/1-overlapper/results/000136.ovb FAILED.
--   job correction/1-overlapper/results/000137.ovb FAILED.
--   job correction/1-overlapper/results/000138.ovb FAILED.
--   job correction/1-overlapper/results/000139.ovb FAILED.
--   job correction/1-overlapper/results/000140.ovb FAILED.
--   job correction/1-overlapper/results/000141.ovb FAILED.
--   job correction/1-overlapper/results/000142.ovb FAILED.
--   job correction/1-overlapper/results/000143.ovb FAILED.
--   job correction/1-overlapper/results/000144.ovb FAILED.
--   job correction/1-overlapper/results/000145.ovb FAILED.
--   job correction/1-overlapper/results/000146.ovb FAILED.
--   job correction/1-overlapper/results/000147.ovb FAILED.
--   job correction/1-overlapper/results/000148.ovb FAILED.
--   job correction/1-overlapper/results/000149.ovb FAILED.
--   job correction/1-overlapper/results/000150.ovb FAILED.
--   job correction/1-overlapper/results/000151.ovb FAILED.
--   job correction/1-overlapper/results/000152.ovb FAILED.
--   job correction/1-overlapper/results/000153.ovb FAILED.
--   job correction/1-overlapper/results/000154.ovb FAILED.
--   job correction/1-overlapper/results/000155.ovb FAILED.
--   job correction/1-overlapper/results/000156.ovb FAILED.
--   job correction/1-overlapper/results/000157.ovb FAILED.
--   job correction/1-overlapper/results/000158.ovb FAILED.
--   job correction/1-overlapper/results/000159.ovb FAILED.
--   job correction/1-overlapper/results/000160.ovb FAILED.
--   job correction/1-overlapper/results/000161.ovb FAILED.
--   job correction/1-overlapper/results/000162.ovb FAILED.
--   job correction/1-overlapper/results/000163.ovb FAILED.
--   job correction/1-overlapper/results/000164.ovb FAILED.
--   job correction/1-overlapper/results/000165.ovb FAILED.
--   job correction/1-overlapper/results/000166.ovb FAILED.
--   job correction/1-overlapper/results/000167.ovb FAILED.
--   job correction/1-overlapper/results/000168.ovb FAILED.
--   job correction/1-overlapper/results/000169.ovb FAILED.
--   job correction/1-overlapper/results/000170.ovb FAILED.
--   job correction/1-overlapper/results/000171.ovb FAILED.
--   job correction/1-overlapper/results/000172.ovb FAILED.
--   job correction/1-overlapper/results/000173.ovb FAILED.
--   job correction/1-overlapper/results/000174.ovb FAILED.
--   job correction/1-overlapper/results/000175.ovb FAILED.
--   job correction/1-overlapper/results/000176.ovb FAILED.
--   job correction/1-overlapper/results/000177.ovb FAILED.
--   job correction/1-overlapper/results/000178.ovb FAILED.
--   job correction/1-overlapper/results/000179.ovb FAILED.
--   job correction/1-overlapper/results/000180.ovb FAILED.
--   job correction/1-overlapper/results/000181.ovb FAILED.
--   job correction/1-overlapper/results/000182.ovb FAILED.
--   job correction/1-overlapper/results/000183.ovb FAILED.
--   job correction/1-overlapper/results/000184.ovb FAILED.
--   job correction/1-overlapper/results/000185.ovb FAILED.
--   job correction/1-overlapper/results/000186.ovb FAILED.
--   job correction/1-overlapper/results/000187.ovb FAILED.
--   job correction/1-overlapper/results/000188.ovb FAILED.
--   job correction/1-overlapper/results/000189.ovb FAILED.
--   job correction/1-overlapper/results/000190.ovb FAILED.
--   job correction/1-overlapper/results/000191.ovb FAILED.
--   job correction/1-overlapper/results/000192.ovb FAILED.
--   job correction/1-overlapper/results/000193.ovb FAILED.
--   job correction/1-overlapper/results/000194.ovb FAILED.
--   job correction/1-overlapper/results/000195.ovb FAILED.
--   job correction/1-overlapper/results/000196.ovb FAILED.
--   job correction/1-overlapper/results/000197.ovb FAILED.
--   job correction/1-overlapper/results/000198.ovb FAILED.
--   job correction/1-overlapper/results/000199.ovb FAILED.
--   job correction/1-overlapper/results/000200.ovb FAILED.
--   job correction/1-overlapper/results/000201.ovb FAILED.
--   job correction/1-overlapper/results/000202.ovb FAILED.
--   job correction/1-overlapper/results/000203.ovb FAILED.
--   job correction/1-overlapper/results/000204.ovb FAILED.
--   job correction/1-overlapper/results/000205.ovb FAILED.
--   job correction/1-overlapper/results/000206.ovb FAILED.
--   job correction/1-overlapper/results/000207.ovb FAILED.
--   job correction/1-overlapper/results/000208.ovb FAILED.
--   job correction/1-overlapper/results/000209.ovb FAILED.
--   job correction/1-overlapper/results/000210.ovb FAILED.
--   job correction/1-overlapper/results/000211.ovb FAILED.
--   job correction/1-overlapper/results/000212.ovb FAILED.
--   job correction/1-overlapper/results/000213.ovb FAILED.
--   job correction/1-overlapper/results/000214.ovb FAILED.
--   job correction/1-overlapper/results/000215.ovb FAILED.
--   job correction/1-overlapper/results/000216.ovb FAILED.
--   job correction/1-overlapper/results/000217.ovb FAILED.
--   job correction/1-overlapper/results/000218.ovb FAILED.
--   job correction/1-overlapper/results/000219.ovb FAILED.
--   job correction/1-overlapper/results/000220.ovb FAILED.
--   job correction/1-overlapper/results/000221.ovb FAILED.
--   job correction/1-overlapper/results/000222.ovb FAILED.
--   job correction/1-overlapper/results/000223.ovb FAILED.
--   job correction/1-overlapper/results/000224.ovb FAILED.
--   job correction/1-overlapper/results/000225.ovb FAILED.
--   job correction/1-overlapper/results/000226.ovb FAILED.
--   job correction/1-overlapper/results/000227.ovb FAILED.
--   job correction/1-overlapper/results/000228.ovb FAILED.
--   job correction/1-overlapper/results/000229.ovb FAILED.
--   job correction/1-overlapper/results/000230.ovb FAILED.
--   job correction/1-overlapper/results/000231.ovb FAILED.
--   job correction/1-overlapper/results/000232.ovb FAILED.
--   job correction/1-overlapper/results/000233.ovb FAILED.
--   job correction/1-overlapper/results/000234.ovb FAILED.
--   job correction/1-overlapper/results/000235.ovb FAILED.
--   job correction/1-overlapper/results/000236.ovb FAILED.
--   job correction/1-overlapper/results/000237.ovb FAILED.
--   job correction/1-overlapper/results/000238.ovb FAILED.
--   job correction/1-overlapper/results/000239.ovb FAILED.
--   job correction/1-overlapper/results/000240.ovb FAILED.
--   job correction/1-overlapper/results/000241.ovb FAILED.
--   job correction/1-overlapper/results/000242.ovb FAILED.
--   job correction/1-overlapper/results/000243.ovb FAILED.
--

ABORT:
ABORT: canu 2.2
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.
ABORT: Try restarting.  If that doesn't work, ask for help.

THANKS for your help and all you do!

skoren commented 1 week ago

I don't think memory is an issue as the jobs are only running 3 at a time using a total of 18gb and you're reserving 192 by cores. I suspect it's more likely you hit a disk quota since the initial precompute ran without issue and the first few jobs ran but everything from 15 onwards failed. What does one of the logs like correction/1-overlapper/000015.err say?

katievigil commented 1 week ago

Hi thanks for your response, but I could not find that error log.

I only found one .err log and this is what it says:

-rw-r--r-- 1 kvigil tulaneusers 9662 Aug 29 07:39 01_dolphin_feces_SQK-NBD114-24_barcode15.report
drwxr-xr-x 3 kvigil tulaneusers 4096 Aug 29 07:37 01_dolphin_feces_SQK-NBD114-24_barcode15.seqStore
-rw-r--r-- 1 kvigil tulaneusers  962 Aug 29 07:37 01_dolphin_feces_SQK-NBD114-24_barcode15.seqStore.err
-rwxr-xr-x 1 kvigil tulaneusers 1234 Aug 29 07:35 01_dolphin_feces_SQK-NBD114-24_barcode15.seqStore.sh
drwxr-xr-x 2 kvigil tulaneusers 4096 Aug 29 07:35 canu-logs
drwxr-xr-x 2 kvigil tulaneusers 4096 Aug 29 07:35 canu-scripts
drwxr-xr-x 4 kvigil tulaneusers 4096 Aug 29 07:39 correction
Found perl:
   /work/kvigil/.conda/envs/canu-env/bin/perl
   This is perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-thread-multi

Found java:
   /work/kvigil/.conda/envs/canu-env/lib/jvm/bin/java
   openjdk version "22.0.1-internal" 2024-04-16

Found canu:
   /ddnB/work/kvigil/.conda/envs/canu-env/bin/canu
   canu 2.2

Creating library '01_dolphin_feces_SQK-NBD114-24_barcode15' for Nanopore raw reads.

               reads               bases
---------- --------- ------ ------------ ------
Loaded        797209   4.2%   1115751947  10.7%  /ddnB/work/kvigil/onr.raw.data.sup.trim/rawdata/results/trimmed/sd_marinemammal/fastq/01_dolphin_feces_SQK-NBD114-24_barcode15.fastq
Short       18105370  95.8%   9267004423  89.3%

All reads processed.

               reads               bases
---------- --------- ------ ------------ ------
Loaded        797209   4.2%   1115751947  10.7%
Short       18105370  95.8%   9267004423  89.3%

Bye.
katievigil commented 1 week ago

I will move files off this drive on the HPC and rerun and let you know if I run into the same issue thanks!

skoren commented 1 week ago

The log file ends in .out not err, sorry. It should be in the correction/1-overlapper folder.

Also from this log it looks about 90% of your reads are shorter than 1kb and are being discarded. Do you expect your data to be this short? You probably need to adjust the overlap size and read length minimums to use more of the input sequences. Also take a look at the FAQ for metagenomic parameters to include if you aren't already.

katievigil commented 1 week ago

Got it here you go:

(base) [kvigil@qbc1 1-overlapper]$ cat  precompute.000015.out

Found perl:
   /work/kvigil/.conda/envs/canu-env/bin/perl
   This is perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-thread-multi

Found java:
   /work/kvigil/.conda/envs/canu-env/lib/jvm/bin/java
   openjdk version "22.0.1-internal" 2024-04-16

Found canu:
   /ddnB/work/kvigil/.conda/envs/canu-env/bin/canu
   canu 2.2

Running job 15 based on command line options.
/ddnB/work/kvigil/onr.raw.data.sup.trim/rawdata/results/trimmed/sd_marinemammal/canu_out/01_dolphin_feces_SQK-NBD114-24_barcode15/correction/1-overlapper
Opened seqStore '../../01_dolphin_feces_SQK-NBD114-24_barcode15.seqStore' for 'raw' reads.
Dumping raw reads from 113401 to 121500 (inclusive).

Starting mhap precompute.

Running with these settings:
--filter-threshold = 1.0E-7
--help = false
--max-shift = 0.2
--min-olap-length = 500
--min-store-length = 0
--no-rc = false
--no-self = false
--no-tf = false
--num-hashes = 768
--num-min-matches = 2
--num-threads = 16
--ordered-kmer-size = 12
--ordered-sketch-size = 1536
--repeat-idf-scale = 10.0
--repeat-weight = 0.9
--settings = 0
--store-full-id = true
--supress-noise = 0
--threshold = 0.73
--version = false
-f = ../../0-mercounts/01_dolphin_feces_SQK-NBD114-24_barcode15.ms16.ignore.gz
-h = false
-k = 16
-p = ./000015.input.fasta
-q = .
-s =

Reading in filter file ../../0-mercounts/01_dolphin_feces_SQK-NBD114-24_barcode15.ms16.ignore.gz.
Read in values for repeat 0 and 0
Warning, k-mer filter file has zero elements.
Initializing
Initialized
Time (s) to read filter file: 0.049834497000000005
Read in k-mer filter for sizes: []
Processing FASTA files for binary compression...
Current # sequences loaded and processed from file: 5000...
Current # sequences loaded and processed from file: 10000...
Current # sequences loaded and processed from file: 15000...
Processed 16200 sequences (fwd and rev).
Read, hashed, and stored file ./000015.input.fasta to ./000015.input.dat.
Time (s): 23.850577403000003
Total time (s): 23.851308347
katievigil commented 1 week ago

I think my reads will have low read length, because I did metagenomic sequencing of viruses from marine mammal stool and serum and oysters, mussels and seawater (~100 samples). I ended up re-basecalling everything using Nanopore's Dorado 0.7.0 super high accuracy basecaller and I am only having issues with the marine mammal dolphin and sea lion stool samples, but these samples have big fastq files so I was not sure if it was a memory thing or like you suggested a storage issue. @skoren Thanks for helping! How do you suggest I change my Canu script?

skoren commented 1 week ago

That's the previous steps, log, we want mhap.000015.out rather than precompute.000015.out

What's your current canu command for this assembly?

katievigil commented 1 week ago
(base) [kvigil@qbc1 1-overlapper]$ cat mhap.000015.out

Found perl:
   /work/kvigil/.conda/envs/canu-env/bin/perl
perl: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory

Found java:
   /work/kvigil/.conda/envs/canu-env/lib/jvm/bin/java
   Error occurred during initialization of VM

Found canu:
   /ddnB/work/kvigil/.conda/envs/canu-env/bin/canu
perl: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory

Running job 15 based on command line options.
Fetch blocks/000055.dat
Fetch blocks/000056.dat
Fetch blocks/000057.dat
Fetch blocks/000058.dat
Fetch blocks/000059.dat
Fetch blocks/000060.dat
Fetch blocks/000061.dat
Fetch blocks/000062.dat
Fetch blocks/000063.dat
Fetch blocks/000064.dat
Fetch blocks/000065.dat
Fetch blocks/000066.dat
Fetch blocks/000067.dat
Fetch blocks/000068.dat
Fetch blocks/000069.dat
Fetch blocks/000070.dat
Fetch blocks/000071.dat
Fetch blocks/000072.dat
Fetch blocks/000073.dat
Fetch blocks/000074.dat
Fetch blocks/000075.dat
Fetch blocks/000076.dat
Fetch blocks/000077.dat
Fetch blocks/000078.dat
Fetch blocks/000079.dat

Running block 000004 in query 000015
skoren commented 1 week ago

It looks like a issue with your compute nodes. The precompute shows everything was working fine (java, perl, etc) but now neither is able to run and if failing. That is compare this:

Found perl:
   /work/kvigil/.conda/envs/canu-env/bin/perl
   This is perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-thread-multi

Found java:
   /work/kvigil/.conda/envs/canu-env/lib/jvm/bin/java
   openjdk version "22.0.1-internal" 2024-04-16

Found canu:
   /ddnB/work/kvigil/.conda/envs/canu-env/bin/canu
   canu 2.2

to this

Found perl:
/work/kvigil/.conda/envs/canu-env/bin/perl
perl: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory

Found java:
/work/kvigil/.conda/envs/canu-env/lib/jvm/bin/java
Error occurred during initialization of VM

Found canu:
/ddnB/work/kvigil/.conda/envs/canu-env/bin/canu
perl: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory

The fact that the same system ran jobs before implies a configuration change mid-run. Nothing in canu can change or fix that, you'd have to ask your system admins what happened.

katievigil commented 1 week ago

Thank you for troubleshooting this I will consult them and re-run the samples.