sunnycqcn commented 7 years ago

Hi, When I run canu1.4, my job stoped at overlapStoreBucketize. My command is

/home/fu115/DIRECTORY/canu/canu-1.4/Linux-amd64/bin/canu \
 -p asm -d strigaA \
 genomeSize=1638.1m \
 errorRate=0.035 \
useGrid=0 \
 -pacbio-raw /scratch/snyder/f/fu115/Genome_assembly/fastq/seq/filtered_subreads.fastq

-- Detected Java(TM) Runtime Environment '1.8.0_111' (from '/group/bioinfo/apps/apps/jdk1.8.0_111/bin/java').
-- Detected gnuplot version '4.6 patchlevel 6' (from 'gnuplot') and image format 'png'.
-- Detected 20 CPUs and 505 gigabytes of memory.
-- Detected PBS/Torque '5.0.1' with 'pbsnodes' binary in /usr/pbs/bin/pbsnodes.
-- Grid engine disabled per useGrid=false option.
--
-- Allowed to run   2 jobs concurrently, and use up to  10 compute threads and  252 GB memory for stage 'bogart (unitigger)'.
-- Allowed to run   2 jobs concurrently, and use up to  10 compute threads and   32 GB memory for stage 'mhap (overlapper)'.
-- Allowed to run   2 jobs concurrently, and use up to  10 compute threads and   32 GB memory for stage 'mhap (overlapper)'.
-- Allowed to run   2 jobs concurrently, and use up to  10 compute threads and   32 GB memory for stage 'mhap (overlapper)'.
-- Allowed to run   4 jobs concurrently, and use up to   5 compute threads and    8 GB memory for stage 'read error detection (overlap error adjustment)'.
-- Allowed to run  20 jobs concurrently, and use up to   1 compute thread  and    2 GB memory for stage 'overlap error adjustment'.
-- Allowed to run   4 jobs concurrently, and use up to   5 compute threads and  126 GB memory for stage 'utgcns (consensus)'.
-- Allowed to run  20 jobs concurrently, and use up to   1 compute thread  and    4 GB memory for stage 'overlap store parallel bucketizer'.
-- Allowed to run  20 jobs concurrently, and use up to   1 compute thread  and   32 GB memory for stage 'overlap store parallel sorting'.
-- Allowed to run  20 jobs concurrently, and use up to   1 compute thread  and    8 GB memory for stage 'overlapper'.
-- Allowed to run   4 jobs concurrently, and use up to   5 compute threads and   12 GB memory for stage 'overlapper'.
-- Allowed to run   4 jobs concurrently, and use up to   5 compute threads and   12 GB memory for stage 'overlapper'.
-- Allowed to run   1 job  concurrently, and use up to  20 compute threads and  256 GB memory for stage 'meryl (k-mer counting)'.
-- Allowed to run   5 jobs concurrently, and use up to   4 compute threads and   32 GB memory for stage 'falcon_sense (read correction)'.
-- Allowed to run   2 jobs concurrently, and use up to  10 compute threads and   32 GB memory for stage 'minimap (overlapper)'.
-- Allowed to run   2 jobs concurrently, and use up to  10 compute threads and   32 GB memory for stage 'minimap (overlapper)'.
-- Allowed to run   2 jobs concurrently, and use up to  10 compute threads and   32 GB memory for stage 'minimap (overlapper)'.
--
-- This is canu parallel iteration #1, out of a maximum of 2 attempts.
--
-- Final error rates before starting pipeline:
--   
--   genomeSize          -- 1638100000
--   errorRate           -- 0.035
--   
--   corOvlErrorRate     -- 0.105
--   obtOvlErrorRate     -- 0.105
--   utgOvlErrorRate     -- 0.105
--   
--   obtErrorRate        -- 0.105
--   
--   cnsErrorRate        -- 0.105
--
--
-- BEGIN CORRECTION
--
-- overlap store bucketizer attempt 1 begins with 0 finished, and 77 to compute.
----------------------------------------
-- Starting concurrent execution on Wed Dec 28 07:21:55 2016 with 735555.484 GB free disk space (77 processes; 20 concurrently)

    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 1 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000001.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 2 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000002.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 3 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000003.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 4 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000004.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 5 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000005.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 6 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000006.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 7 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000007.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 8 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000008.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 9 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000009.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 10 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000010.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 11 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000011.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 12 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000012.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 13 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000013.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 14 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000014.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 15 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000015.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 16 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000016.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 17 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000017.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 18 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000018.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 19 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000019.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 20 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000020.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 21 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000021.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 22 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000022.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 23 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000023.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 24 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000024.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 25 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000025.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 26 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000026.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 27 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000027.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 28 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000028.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 29 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000029.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 30 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000030.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 31 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000031.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 32 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000032.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 33 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000033.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 34 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000034.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 35 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000035.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 36 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000036.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 37 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000037.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 38 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000038.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 39 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000039.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 40 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000040.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 41 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000041.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 42 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000042.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 43 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000043.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 44 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000044.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 45 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000045.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 46 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000046.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 47 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000047.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 48 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000048.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 49 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000049.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 50 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000050.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 51 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000051.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 52 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000052.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 53 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000053.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 54 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000054.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 55 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000055.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 56 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000056.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 57 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000057.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 58 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000058.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 59 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000059.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 60 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000060.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 61 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000061.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 62 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000062.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 63 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000063.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 64 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000064.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 65 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000065.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 66 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000066.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 67 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000067.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 68 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000068.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 69 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000069.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 70 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000070.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 71 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000071.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 72 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000072.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 73 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000073.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 74 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000074.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 75 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000075.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 76 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000076.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 77 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000077.out 2>&1

-- Finished on Wed Dec 28 07:21:56 2016 (1 second) with 735554.928 GB free disk space
----------------------------------------
--
-- 77 overlap store bucketizer jobs failed:
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0001 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0002 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0003 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0004 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0005 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0006 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0007 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0008 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0009 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0010 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0011 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0012 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0013 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0014 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0015 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0016 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0017 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0018 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0019 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0020 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0021 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0022 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0023 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0024 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0025 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0026 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0027 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0028 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0029 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0030 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0031 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0032 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0033 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0034 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0035 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0036 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0037 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0038 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0039 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0040 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0041 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0042 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0043 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0044 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0045 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0046 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0047 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0048 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0049 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0050 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0051 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0052 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0053 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0054 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0055 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0056 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0057 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0058 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0059 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0060 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0061 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0062 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0063 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0064 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0065 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0066 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0067 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0068 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0069 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0070 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0071 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0072 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0073 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0074 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0075 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0076 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0077 FAILED.
--
-- overlap store bucketizer attempt 2 begins with 0 finished, and 77 to compute.
----------------------------------------
-- Starting concurrent execution on Wed Dec 28 07:21:56 2016 with 735554.928 GB free disk space (77 processes; 20 concurrently)

    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 1 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000001.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 2 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000002.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 3 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000003.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 4 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000004.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 5 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000005.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 6 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000006.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 7 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000007.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 8 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000008.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 9 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000009.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 10 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000010.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 11 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000011.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 12 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000012.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 13 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000013.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 14 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000014.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 15 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000015.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 16 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000016.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 17 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000017.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 18 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000018.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 19 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000019.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 20 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000020.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 21 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000021.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 22 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000022.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 23 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000023.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 24 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000024.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 25 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000025.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 26 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000026.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 27 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000027.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 28 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000028.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 29 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000029.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 30 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000030.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 31 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000031.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 32 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000032.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 33 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000033.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 34 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000034.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 35 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000035.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 36 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000036.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 37 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000037.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 38 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000038.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 39 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000039.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 40 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000040.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 41 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000041.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 42 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000042.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 43 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000043.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 44 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000044.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 45 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000045.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 46 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000046.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 47 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000047.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 48 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000048.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 49 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000049.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 50 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000050.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 51 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000051.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 52 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000052.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 53 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000053.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 54 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000054.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 55 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000055.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 56 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000056.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 57 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000057.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 58 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000058.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 59 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000059.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 60 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000060.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 61 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000061.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 62 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000062.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 63 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000063.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 64 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000064.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 65 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000065.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 66 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000066.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 67 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000067.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 68 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000068.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 69 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000069.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 70 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000070.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 71 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000071.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 72 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000072.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 73 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000073.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 74 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000074.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 75 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000075.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 76 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000076.out 2>&1
    /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 77 > /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000077.out 2>&1

-- Finished on Wed Dec 28 07:21:57 2016 (1 second) with 735554.928 GB free disk space
----------------------------------------
--
-- 77 overlap store bucketizer jobs failed:
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0001 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0002 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0003 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0004 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0005 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0006 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0007 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0008 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0009 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0010 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0011 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0012 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0013 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0014 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0015 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0016 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0017 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0018 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0019 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0020 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0021 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0022 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0023 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0024 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0025 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0026 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0027 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0028 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0029 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0030 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0031 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0032 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0033 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0034 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0035 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0036 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0037 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0038 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0039 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0040 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0041 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0042 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0043 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0044 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0045 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0046 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0047 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0048 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0049 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0050 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0051 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0052 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0053 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0054 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0055 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0056 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0057 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0058 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0059 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0060 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0061 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0062 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0063 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0064 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0065 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0066 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0067 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0068 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0069 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0070 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0071 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0072 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0073 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0074 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0075 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0076 FAILED.
--   job /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/bucket0077 FAILED.
--
================================================================================
Don't panic, but a mostly harmless error occurred and canu failed.

canu failed with 'failed to overlapStoreBucketize.  Made 2 attempts, jobs still failed'.

skoren commented 7 years ago

The fact that all the jobs fail and in <1s makes me suspect there is an issue with the input from the previous step. Is this assembly completely run with Canu 1.4? Canu 1.4 is not backwards compatible with 1.3 so if you ran part of the assembly with 1.3 you cannot switch to 1.4. What is the output in the /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.*.out files?

sunnycqcn commented 7 years ago

Hi Koren, I only used Canu 1.4 for assembly. Thanks,

Running job 1 based on command line options. Removing incomplete bucket /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/create0001 /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh: line 126: [: unlimited: integer expression expected Max processes per user limited to unlimited, no increase possible. Max open files limited to 32768, no increase possible.

maxError fraction: 1.000 percent: 100.000 encoded: 4095 Bucketizing /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/1-overlapper/results/000001.ovb

Failed with 'Segmentation fault'

Backtrace (mangled):

/home/fu115/DIRECTORY/canu/canu-1.4/Linux-amd64/bin/ovStoreBucketizer(_Z17AS_UTL_catchCrashiP7siginfoPv+0x27)[0x404b97] /lib64/libpthread.so.0[0x34e5c0f7e0] /home/fu115/DIRECTORY/canu/canu-1.4/Linux-amd64/bin/ovStoreBucketizer[0x40afae] /home/fu115/DIRECTORY/canu/canu-1.4/Linux-amd64/bin/ovStoreBucketizer[0x4027ed] /lib64/libc.so.6(__libc_start_main+0xfd)[0x34e501ed1d] /home/fu115/DIRECTORY/canu/canu-1.4/Linux-amd64/bin/ovStoreBucketizer[0x402159]

Backtrace (demangled):

[0] /home/fu115/DIRECTORY/canu/canu-1.4/Linux-amd64/bin/ovStoreBucketizer::AS_UTL_catchCrash(int, siginfo, void) + 0x27 [0x404b97] [1] /lib64/libpthread.so.0() [0x34e5c0f7e0] [2] /home/fu115/DIRECTORY/canu/canu-1.4/Linux-amd64/bin/ovStoreBucketizer() [0x40afae] [3] /home/fu115/DIRECTORY/canu/canu-1.4/Linux-amd64/bin/ovStoreBucketizer() [0x4027ed] [4] /lib64/libc.so.6::(null) + 0xfd [0x34e501ed1d] [5] /home/fu115/DIRECTORY/canu/canu-1.4/Linux-amd64/bin/ovStoreBucketizer() [0x402159]

GDB:

/scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh: line 173: 30974 Segmentation fault (core dumped) $bin/ovStoreBucketizer -O /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING -G /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.gkpStore -C /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.ovlStore.BUILDING/config -job $jobid -i $jn

On Wed, Dec 28, 2016 at 7:27 AM, Sergey Koren notifications@github.com wrote:

The fact that all the jobs fail and in <1s makes me suspect there is an issue with the input from the previous step. Is this assembly completely run with Canu 1.4? Canu 1.4 is not backwards compatible with 1.3 so if you ran part of the assembly with 1.3 you cannot switch to 1.4. What is the output in the /scratch/snyder/f/fu115/Genome_assembly/PBonly/ canutest/strigaA/correction/asm.ovlStore.BUILDING/logs/1-bucketize.*.out files?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/marbl/canu/issues/319#issuecomment-269476962, or mute the thread https://github.com/notifications/unsubscribe-auth/AXaRKJAUU9y0x5cw0HJCLwA2Tmh4sRuXks5rMmPbgaJpZM4LW45G .

-- Fuyou Fu, Ph.D. Department of Botany and Plant Pathology Purdue University USA

frogsicle commented 7 years ago

I have a very similar problem. I've seen it with 1.4, but not with the same data and 1.3 (no, I'm not switching mid-run between the two).

With gdb I got to:

0 gkStore_getRead (this=0xbcf5d0, foverlap=..., roverlap=...) at stores/gkStore.H:502

1 a_end (this=0xbcf5d0, foverlap=..., roverlap=...) at stores/ovOverlap.H:236

2 isOverlapLong (this=0xbcf5d0, foverlap=..., roverlap=...) at stores/ovStoreFilter.C:127

3 ovStoreFilter::filterOverlap (this=0xbcf5d0, foverlap=..., roverlap=...) at stores/ovStoreFilter.C:195

4 0x0000000000402810 in main (argc=, argv=) at stores/ovStoreBucketizer.C:254

with: gkStore_getRead (this=0xbcf5d0, foverlap=..., roverlap=...) at stores/gkStore.H:502 502 (_readIDtoPartitionID[id] != _partitionID)) {

the _readIDtoPartitionID was indeed not a valid address.

Hoping that narrows the search some. Thanks!

skoren commented 7 years ago

@frogsicle is that the line GDB reported for the crash? _readIDtoPartitionID should be NULL (or 0) because partitions are not used here. Is it a different value?

@sunnycqcn are you able to share your /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/asm.gkpStore and /scratch/snyder/f/fu115/Genome_assembly/PBonly/canutest/strigaA/correction/1-overlapper/results/000001.ovb files? You can upload them to our ftp site here:ftp://ftp.cbcb.umd.edu/incoming/sergek/

frogsicle commented 7 years ago

So far as I understand gdb, yes.

(gdb) print _readsPerPartition
$3 = (uint32 *) 0x676f6c2f474e4944
(gdb) print *_readsPerPartition
Cannot access memory at address 0x676f6c2f474e4944

And in case it slipped a line (due to me adding a print statement or such):

500   gkRead      *gkStore_getRead(uint32 id)          {
501     if ((_readIDtoPartitionID)   &&
502         (_readIDtoPartitionID[id] != _partitionID)) {

skoren commented 7 years ago

_readsPerPartition should not be used in line 502. What's the value of _readIDtoPartitionID?

Can you share your asm.gkpStore and any of the ovb/counts files to our FTP site so I can see if I can reproduce the error locally?

frogsicle commented 7 years ago

oops, sorry grabbed the wrong one, but same idea:

(gdb) print _readIDtoPartitionID
$1 = (uint32 *) 0x36312e657a697465
(gdb) print *_readIDtoPartitionID
Cannot access memory at address 0x36312e657a697465

where's the ftp site?

skoren commented 7 years ago

Hmm very interesting, I confirmed this should be initialized to NULL, what does ls -lha correction/asm.gkpStore show? The ftp site is: tp://ftp.cbcb.umd.edu/incoming/sergek/

frogsicle commented 7 years ago

$ ls -lha correction/Sp40d_02.gkpStore
total 12G
drwxrwxr-x. 2 alisandra alisandra   15 23. Dez 17:10 .
drwxrwxr-x. 6 alisandra alisandra    8 23. Dez 22:47 ..
-rw-rw-r--. 1 alisandra alisandra  11G 23. Dez 17:09 blobs
-rw-rw-r--. 1 alisandra alisandra  65M 23. Dez 17:09 errorLog
-rw-rw-r--. 1 alisandra alisandra   56 23. Dez 17:09 info
-rw-rw-r--. 1 alisandra alisandra  280 23. Dez 17:09 info.txt
-rw-rw-r--. 1 alisandra alisandra 5,3K 23. Dez 17:09 libraries
-rw-rw-r--. 1 alisandra alisandra 2,5K 23. Dez 17:10 libraries.txt
-rw-rw-r--. 1 alisandra alisandra 8,9K 23. Dez 17:09 load.dat
-rw-rw-r--. 1 alisandra alisandra 2,4K 23. Dez 17:10 readlengthhistogram.txt
-rw-rw-r--. 1 alisandra alisandra  748 23. Dez 17:10 readlengths.gp
-rw-rw-r--. 1 alisandra alisandra  19M 23. Dez 17:10 readlengths.txt
-rw-rw-r--. 1 alisandra alisandra 241M 23. Dez 17:09 readNames.txt
-rw-rw-r--. 1 alisandra alisandra  54M 23. Dez 17:09 reads
-rw-rw-r--. 1 alisandra alisandra  97M 23. Dez 17:10 reads.txt

and thanks, I'll go upload stuff.

skoren commented 7 years ago

Thanks, while it's uploading, can you set a breakpoint at gkStore.C:650 and print the values of path, partID, and gkStore_mode, and partID==UINT32_MAX

frogsicle commented 7 years ago

(gdb) print partID
$1 = 4294967295
(gdb) print path
$2 = 0x7fff3cd578f3 "/mnt/data/alisandra/Lost/canu/Sp40d_02/correction/Sp40d_02.gkpStore"
(gdb) print mode
$3 = gkStore_readOnly
(gdb) print UINT32_MAX
No symbol "UINT32_MAX" in current context.

Via the (uncommented) logs, it ends up inside the first if statement, though, so I guess that's a True.

[gkStore()-- opening '/mnt/data/alisandra/Lost/canu/Sp40d_02/correction/Sp40d_02.gkpStore' for read-only access.]

sunnycqcn commented 7 years ago

I tried some times. I got the same problem.

On Wed, Dec 28, 2016 at 12:15 PM, frogsicle notifications@github.com wrote:

(gdb) print partID $1 = 4294967295 (gdb) print path $2 = 0x7fff3cd578f3 "/mnt/data/alisandra/Lost/canu/Sp40d_02/correction/ Sp40d_02.gkpStore" (gdb) print mode $3 = gkStore_readOnly (gdb) print UINT32_MAX No symbol "UINT32_MAX" in current context.

Via the (uncommented) logs, it ends up inside the first if statement, though, so I guess that's a True.

[gkStore()-- opening '/mnt/data/alisandra/Lost/canu/Sp40d_02/correction/Sp40d_02.gkpStore' for read-only access.]

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marbl/canu/issues/319#issuecomment-269516811, or mute the thread https://github.com/notifications/unsubscribe-auth/AXaRKEAeR6XtRT2H9GYiIPNVcvujIlcpks5rMqcrgaJpZM4LW45G .

-- Fuyou Fu, Ph.D. Department of Botany and Plant Pathology Purdue University USA

skoren commented 7 years ago

I can confirm with the exact same parameters the overlap bucketizing runs without error on our systems. Are you using Canu 1.4 compiled from source or a pre-compiled binary? If it is pre-compiled, try building it locally from source in debug mode (make BUILDDEBUG=1) and see if it runs then.

@frogsicle can you post the output from uname -a and gcc --version. Also for the crash, print the values of _partitionID and _numberOfPartitions. I want to see exactly why your store seems to think it should be partitioned when it is not.

Finally, can you try running the attached binary, it has (a lot) of debugging information added so should hopefully better capture when the crash is happening. ovStoreBucketizer.gz

frogsicle commented 7 years ago

So, an update: For completeness: $ uname -a Linux kerry.plabipd.de 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux $ gcc --version gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-11)

Further examination indicates that _readIDtoPartitionID is indeed NULL for the first gkStore_getNumReads() + 1 iterations. Then it changes to not NULL but not valid, and the seg fault occurs. Similarly _partitionID and _numberOfPartitions start at 0; but right before the seg fault they get an invalid memory address. Indeed the whole gkStore instance seemed to go from valid -> invalid

While attemtping to figure out what was going on, I added a print statement to track addresses of various gkStore references in ovStoreBucketizer.C; and to my surprise it ran through. Playing with this further indicated the print statements were changing the address where things were stored (ASLR dissabled for trouble shooting purposes). But I'm in a little over my head. I pushed the print statements in question to a fork of canu under my account so you can at least see what I'm talking about.

Let me how I can help further, and thanks again!

brianwalenz commented 7 years ago

This should be fixed now. Thanks for the print statements. Reopen if needed.

sunnycqcn commented 7 years ago

Thanks, I will update my results. I think I can get the results today. Fuyou

sunnycqcn commented 7 years ago

Hi Brian, I did not finish my run and stop at 4-unitigger. I get the error files as following: unitigger.err:

Graph error threshold = 0.105 (10.500%) Max error threshold = 0.105 (10.500%)

Minimum overlap length = 500 bases

number of threads = 10 (command line)

==> LOADING AND FILTERING OVERLAPS.

ReadInfo()-- Using 943770 reads, no minimum read length used.

OverlapCache()-- limited to 258048MB memory (user supplied). OverlapCache()-- 3MB for read data. OverlapCache()-- 14MB for best edges. OverlapCache()-- 25MB for unitig layouts. OverlapCache()-- 0MB for tigs. OverlapCache()-- 7MB for id maps. OverlapCache()-- 57MB for error profiles. OverlapCache()-- 10MB for overlap cache pointers. OverlapCache()-- 48MB for overlap cache initial bucket. OverlapCache()-- 160MB for overlap cache thread data. OverlapCache()-- 3MB for number of overlaps per read. OverlapCache()-- 0MB for other processes. OverlapCache()-- --------- OverlapCache()-- 273MB for data structures (sum of above). OverlapCache()-- 257774MB available for overlaps.

OverlapCache()-- Loading number of overlaps per read. OverlapCache()-- Retain at least 18 overlaps/read, based on 3.04x coverage. OverlapCache()-- Initial guess at 17900 overlaps/read (maximum 19702 overlaps/read). OverlapCache()-- 17900 overlaps/read - load all for 943759 reads, some for 11 reads - 947722812 overlaps to load - 14461MB OverlapCache()-- 19702 overlaps/read - load all for 943770 reads, some for 0 reads - 947729700 overlaps to load - 14461MB

OverlapCache()-- minPer = 18 overlaps/reads OverlapCache()-- maxPer = 19702 overlaps/reads OverlapCache()-- numBelow = 943769 reads (all overlaps loaded) OverlapCache()-- numEqual = 1 reads (all overlaps loaded) OverlapCache()-- numAbove = 0 reads (some overlaps loaded) OverlapCache()-- totalLoad = 947729700 overlaps (100.00%)

OverlapCache()-- availForOverlaps = 257774MB OverlapCache()-- totalMemory = 0MB for organization OverlapCache()-- totalMemory = 14461MB for overlaps OverlapCache()-- totalMemory = 14461MB used

OverlapCache()-- Loading: overlaps processed 189 (000.00%) loaded 189 (000.00%) droppeddupe 0 (000.00%) bogart: bogart/AS_BAT_OverlapCache.C:468: uint32 OverlapCache::filterDuplicates(uint32&): Assertion `_ovs[jj].a_iid != 0' failed.

Failed with 'Aborted'

Backtrace (mangled):

/home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart(_Z17AS_UTL_catchCrashiP7siginfoPv+0x27)[0x4f0887] /lib64/libpthread.so.0[0x313b20f7e0] /lib64/libc.so.6(gsignal+0x35)[0x313a6325e5] /lib64/libc.so.6(abort+0x175)[0x313a633dc5] /lib64/libc.so.6[0x313a62b70e] /lib64/libc.so.6(assert_perror_fail+0x0)[0x313a62b7d0] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart[0x4b139e] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart[0x4b1c05] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart[0x4b2c41] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart[0x404372] /lib64/libc.so.6(libc_start_main+0xfd)[0x313a61ed1d] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart[0x403399]

Backtrace (demangled):

[0] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart::AS_UTL_catchCrash(int, siginfo, void) + 0x27 [0x4f0887] [1] /lib64/libpthread.so.0() [0x313b20f7e0] [2] /lib64/libc.so.6::(null) + 0x35 [0x313a6325e5] [3] /lib64/libc.so.6::(null) + 0x175 [0x313a633dc5] [4] /lib64/libc.so.6() [0x313a62b70e] [5] /lib64/libc.so.6::(null) + 0 [0x313a62b7d0] [6] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart() [0x4b139e] [7] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart() [0x4b1c05] [8] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart() [0x4b2c41] [9] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart() [0x404372] [10] /lib64/libc.so.6::(null) + 0xfd [0x313a61ed1d] [11] /home/fu115/DIRECTORY/canu/canu/Linux-amd64/bin/bogart() [0x403399]

GDB:

-- Detected Java(TM) Runtime Environment '1.8.0_111' (from '/group/bioinfo/apps/apps/jdk1.8.0_111/bin/java'). -- Detected gnuplot version '4.6 patchlevel 6' (from 'gnuplot') and image format 'png'. -- Detected 20 CPUs and 505 gigabytes of memory. -- Detected PBS/Torque '5.0.1' with 'pbsnodes' binary in /usr/pbs/bin/pbsnodes. -- Grid engine disabled per useGrid=false option.

-- Allowed to run 2 jobs concurrently, and use up to 10 compute threads and 252 GB memory for stage 'bogart (unitigger)'. -- Allowed to run 2 jobs concurrently, and use up to 10 compute threads and 32 GB memory for stage 'mhap (overlapper)'. -- Allowed to run 2 jobs concurrently, and use up to 10 compute threads and 32 GB memory for stage 'mhap (overlapper)'. -- Allowed to run 2 jobs concurrently, and use up to 10 compute threads and 32 GB memory for stage 'mhap (overlapper)'. -- Allowed to run 4 jobs concurrently, and use up to 5 compute threads and 8 GB memory for stage 'read error detection (overlap error adjustment)'. -- Allowed to run 20 jobs concurrently, and use up to 1 compute thread and 2 GB memory for stage 'overlap error adjustment'. -- Allowed to run 4 jobs concurrently, and use up to 5 compute threads and 126 GB memory for stage 'utgcns (consensus)'. -- Allowed to run 20 jobs concurrently, and use up to 1 compute thread and 4 GB memory for stage 'overlap store parallel bucketizer'. -- Allowed to run 20 jobs concurrently, and use up to 1 compute thread and 32 GB memory for stage 'overlap store parallel sorting'. -- Allowed to run 20 jobs concurrently, and use up to 1 compute thread and 8 GB memory for stage 'overlapper'. -- Allowed to run 4 jobs concurrently, and use up to 5 compute threads and 12 GB memory for stage 'overlapper'. -- Allowed to run 4 jobs concurrently, and use up to 5 compute threads and 12 GB memory for stage 'overlapper'. -- Allowed to run 1 job concurrently, and use up to 20 compute threads and 256 GB memory for stage 'meryl (k-mer counting)'. -- Allowed to run 5 jobs concurrently, and use up to 4 compute threads and 32 GB memory for stage 'falcon_sense (read correction)'. -- Allowed to run 2 jobs concurrently, and use up to 10 compute threads and 32 GB memory for stage 'minimap (overlapper)'. -- Allowed to run 2 jobs concurrently, and use up to 10 compute threads and 32 GB memory for stage 'minimap (overlapper)'. -- Allowed to run 2 jobs concurrently, and use up to 10 compute threads and 32 GB memory for stage 'minimap (overlapper)'.

-- This is canu parallel iteration #1, out of a maximum of 2 attempts.

-- Final error rates before starting pipeline:

-- genomeSize -- 1638100000 -- errorRate -- 0.035

-- corOvlErrorRate -- 0.105 -- obtOvlErrorRate -- 0.105 -- utgOvlErrorRate -- 0.105

-- obtErrorRate -- 0.105

-- cnsErrorRate -- 0.105

-- -- BEGIN CORRECTION

-- Starting command on Fri Dec 30 13:11:19 2016 with 3261.968 GB free disk space

/depot/bioinfo/apps/apps/canu-1.4/Linux-amd64/bin/gatekeeperCreate \
  -minlength 1000 \
  -o /depot/tmengist/etc/strigaB/correction/asm.gkpStore.BUILDING \
  /depot/tmengist/etc/strigaB/correction/asm.gkpStore.gkp \
> /depot/tmengist/etc/strigaB/correction/asm.gkpStore.BUILDING.err 2>&1

-- Finished on Fri Dec 30 13:13:04 2016 (105 seconds) with 3254.136 GB free disk space

-- -- In gatekeeper store '/depot/tmengist/etc/strigaB/correction/asm.gkpStore': -- Found 1532127 reads. -- Found 10957014849 bases (6.68 times coverage).

-- Read length histogram (one '*' equals 2729.27 reads): -- 0 999 0 -- 1000 1999 191049

-- 2000 2999 171449

-- 3000 3999 126606 -- 4000 4999 110168 -- 5000 5999 98600 -- 6000 6999 95901 * -- 7000 7999 115226 -- 8000 8999 138763 -- 9000 9999 118477 *** -- 10000 10999 92431 * -- 11000 11999 68662 * -- 12000 12999 50139 -- 13000 13999 37693 * -- 14000 14999 28234 -- 15000 15999 21070 * -- 16000 16999 15786 * -- 17000 17999 12144 -- 18000 18999 8949 * -- 19000 19999 6889 * -- 20000 20999 5290 -- 21000 21999 4057 -- 22000 22999 3068 -- 23000 23999 2387 -- 24000 24999 1841 -- 25000 25999 1486 -- 26000 26999 1055 -- 27000 27999 974 -- 28000 28999 744 -- 29000 29999 582 -- 30000 30999 452 -- 31000 31999 361 -- 32000 32999 309 -- 33000 33999 225 -- 34000 34999 191 -- 35000 35999 135 -- 36000 36999 121 -- 37000 37999 115 -- 38000 38999 89 -- 39000 39999 75 -- 40000 40999 54 -- 41000 41999 43 -- 42000 42999 41 -- 43000 43999 31 -- 44000 44999 27 -- 45000 45999 13 -- 46000 46999 20 -- 47000 47999 13 -- 48000 48999 16 -- 49000 49999 8 -- 50000 50999 10 -- 51000 51999 13 -- 52000 52999 7 -- 53000 53999 9 -- 54000 54999 6 -- 55000 55999 6 -- 56000 56999 3 -- 57000 57999 0 -- 58000 58999 3 -- 59000 59999 3 -- 60000 60999 3 -- 61000 61999 1 -- 62000 62999 1 -- 63000 63999 1 -- 64000 64999 0 -- 65000 65999 1 -- 66000 66999 0 -- 67000 67999 0 -- 68000 68999 1 -- Meryl attempt 1 begins.

-- Starting concurrent execution on Fri Dec 30 13:13:14 2016 with 3253.855 GB free disk space (1 processes; 1 concurrently)

/depot/tmengist/etc/strigaB/correction/0-mercounts/meryl.sh 1 >

/depot/tmengist/etc/strigaB/correction/0-mercounts/meryl.000001.out 2>&1

-- Finished on Fri Dec 30 13:31:21 2016 (1087 seconds) with 3158.441 GB free disk space

-- Meryl finished successfully.

-- Starting command on Fri Dec 30 13:31:21 2016 with 3164.578 GB free disk space

/depot/bioinfo/apps/apps/canu-1.4/Linux-amd64/bin/meryl \
  -Dh \
  -s /depot/tmengist/etc/strigaB/correction/0-mercounts/asm.ms16 \
> /depot/tmengist/etc/strigaB/correction/0-mercounts/asm.ms16.histogram

\ 2> /depot/tmengist/etc/strigaB/correction/0-mercounts/ asm.ms16.histogram.info

-- Finished on Fri Dec 30 13:31:21 2016 (lickety-split) with 3172.931 GB free disk space

-- For mhap overlapping, set repeat k-mer threshold to 109340.

-- Found 10934032944 16-mers; 1865009285 distinct and 421977945 unique. Largest count 6600395.

-- OVERLAPPER (mhap) (correction)

-- Set corMhapSensitivity=high based on read coverage of 6.

-- PARAMETERS: hashes=768, minMatches=2, threshold=0.73

-- Given 32 GB, can fit 48000 reads per block. -- For 33 blocks, set stride to 8 blocks. -- Logging partitioning to '/depot/tmengist/etc/strigaB/correction/1-overlapper/partitioning.log'. -- Configured 32 mhap precompute jobs. -- Configured 77 mhap overlap jobs. -- mhap precompute attempt 1 begins with 0 finished, and 32 to compute.

-- Starting concurrent execution on Fri Dec 30 13:32:22 2016 with 3230.638 GB free disk space (32 processes; 2 concurrently)

/depot/tmengist/etc/strigaB/correction/1-overlapper/precompute.sh 1 >

/depot/tmengist/etc/strigaB/correction/1-overlapper/precompute.000001.out 2>&1 ... /depot/tmengist/etc/strigaB/correction/1-overlapper/precompute.sh 32 > /depot/tmengist/etc/strigaB/correction/1-overlapper/precompute.000032.out 2>&1

-- Finished on Fri Dec 30 19:01:28 2016 (19746 seconds) with 3047.425 GB free disk space

-- All 32 mhap precompute jobs finished successfully. -- mhap attempt 1 begins with 0 finished, and 77 to compute.

-- Starting concurrent execution on Fri Dec 30 19:01:28 2016 with 3047.425 GB free disk space (77 processes; 2 concurrently)

/depot/tmengist/etc/strigaB/correction/1-overlapper/mhap.sh 1 >

/depot/tmengist/etc/strigaB/correction/1-overlapper/mhap.000001.out 2>&1 ... /depot/tmengist/etc/strigaB/correction/1-overlapper/mhap.sh 77 > /depot/tmengist/etc/strigaB/correction/1-overlapper/mhap.000077.out 2>&1

-- Finished on Sat Dec 31 14:24:09 2016 (69761 seconds) with 2535.035 GB free disk space

-- Found 77 mhap overlap output files.

-- Starting command on Sat Dec 31 14:24:09 2016 with 2537.82 GB free disk space

/depot/tmengist/etc/strigaB/correction/asm.ovlStore.BUILDING/scripts/0-config.sh \

/depot/tmengist/etc/strigaB/correction/asm.ovlStore.BUILDING/config.err 2>&1

-- Finished on Sat Dec 31 14:24:12 2016 (3 seconds) with 2537.796 GB free disk space

-- overlap store bucketizer attempt 1 begins with 0 finished, and 77 to compute.

-- Starting concurrent execution on Sat Dec 31 14:24:12 2016 with 2537.796 GB free disk space (77 processes; 20 concurrently)

/depot/tmengist/etc/strigaB/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 1 > /depot/tmengist/etc/strigaB/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000001.out 2>&1 ... /depot/tmengist/etc/strigaB/correction/asm.ovlStore.BUILDING/scripts/1-bucketize.sh 77 > /depot/tmengist/etc/strigaB/correction/asm.ovlStore.BUILDING/logs/1-bucketize.000077.out 2>&1

-- Finished on Sat Dec 31 14:29:23 2016 (311 seconds) with 1864.818 GB free disk space

-- Overlap store bucketizer finished. -- overlap store sorter attempt 1 begins with 0 finished, and 158 to compute.

-- Starting concurrent execution on Sat Dec 31 14:29:23 2016 with 1864.818 GB free disk space (158 processes; 20 concurrently)

/depot/tmengist/etc/strigaB/correction/asm.ovlStore.BUILDING/scripts/2-sort.sh 1 > /depot/tmengist/etc/strigaB/correction/asm.ovlStore.BUILDING/logs/2-sort.000001.out 2>&1

frogsicle commented 7 years ago

Thanks a bunch, it ran all the way through for me nicely!

marbl / canu

stoped at overlapStoreBucketize #319

0 gkStore_getRead (this=0xbcf5d0, foverlap=..., roverlap=...) at stores/gkStore.H:502

1 a_end (this=0xbcf5d0, foverlap=..., roverlap=...) at stores/ovOverlap.H:236

2 isOverlapLong (this=0xbcf5d0, foverlap=..., roverlap=...) at stores/ovStoreFilter.C:127

3 ovStoreFilter::filterOverlap (this=0xbcf5d0, foverlap=..., roverlap=...) at stores/ovStoreFilter.C:195

4 0x0000000000402810 in main (argc=, argv=) at stores/ovStoreBucketizer.C:254

-- This is canu parallel iteration #1, out of a maximum of 2 attempts.

-- Final error rates before starting pipeline:

-- genomeSize -- 1638100000 -- errorRate -- 0.035

-- corOvlErrorRate -- 0.105 -- obtOvlErrorRate -- 0.105 -- utgOvlErrorRate -- 0.105

-- obtErrorRate -- 0.105

-- cnsErrorRate -- 0.105

-- -- BEGIN CORRECTION

-- Finished on Fri Dec 30 13:13:04 2016 (105 seconds) with 3254.136 GB free disk space

-- -- In gatekeeper store '/depot/tmengist/etc/strigaB/correction/asm.gkpStore': -- Found 1532127 reads. -- Found 10957014849 bases (6.68 times coverage).

-- Finished on Fri Dec 30 13:31:21 2016 (1087 seconds) with 3158.441 GB free disk space

-- Meryl finished successfully.

-- Finished on Fri Dec 30 13:31:21 2016 (lickety-split) with 3172.931 GB free disk space

-- For mhap overlapping, set repeat k-mer threshold to 109340.

-- Found 10934032944 16-mers; 1865009285 distinct and 421977945 unique. Largest count 6600395.

-- OVERLAPPER (mhap) (correction)

-- Set corMhapSensitivity=high based on read coverage of 6.

-- PARAMETERS: hashes=768, minMatches=2, threshold=0.73

-- Finished on Fri Dec 30 19:01:28 2016 (19746 seconds) with 3047.425 GB free disk space

-- All 32 mhap precompute jobs finished successfully. -- mhap attempt 1 begins with 0 finished, and 77 to compute.

-- Finished on Sat Dec 31 14:24:09 2016 (69761 seconds) with 2535.035 GB free disk space

-- Found 77 mhap overlap output files.

-- Finished on Sat Dec 31 14:24:12 2016 (3 seconds) with 2537.796 GB free disk space

-- overlap store bucketizer attempt 1 begins with 0 finished, and 77 to compute.

-- Finished on Sat Dec 31 14:29:23 2016 (311 seconds) with 1864.818 GB free disk space

-- Overlap store bucketizer finished. -- overlap store sorter attempt 1 begins with 0 finished, and 158 to compute.