marbl / canu

A single molecule sequence assembler for genomes large and small.
http://canu.readthedocs.io/
654 stars 179 forks source link

cormhap jobs timeout #917

Closed zrlewis closed 6 years ago

zrlewis commented 6 years ago

Hi again,

I am working on assembling a 3.3 Gb invertebrate genome using about 50X PacBio coverage (160 Gb reads). With your suggestions for specifying grid options (Issue #886) I was able to get the assembly running properly on a SLURM partition with many available nodes.

Here is my submission script:

canu \
    -p physalia -d physalia_assembly_10 \
    genomeSize=3.3g \
    -pacbio-raw $READS/Physalia_concatenated_reads.fasta \
    correctedErrorRate=0.065  \
    gridOptions="--partition general" \
    batMemory=100 batThreads=20 merylMemory=90 merylThreads=20 gfaThreads=20 corMemory=6 \
    gridOptionsmeryl="-t 03:00:00" gridOptionscormhap="-t 10:00:00"

I will paste my canu.out log below.

I based my gridOptionscormhap time setting on previous runs of this same assembly that had taken only a few hours to complete, but now the jobs are starting to timeout. This step has been running for about two weeks so far and only about 25% of the jobs successfully completed, with the rest timing out eventually. I can have the times manually adjusted on our cluster to get some to complete, but I have a couple questions:

  1. If I want to stop and restart the cormhap jobs, will Canu pick up where it has left off? Would I just re-submit the original submission script? This is related to Issue #794.
  2. I had set the correctedErrorRate high, because this is a wild-caught individual with probable high heterozygosity. Perhaps it is too high and this is bogging down correction. Could this be an issue? If I stop Canu and resubmit with a lower setting, will it create problems?
  3. If I resubmit, I will simply set the time limit to 7d, Should I also increase memory or CPUs for cormhap?

Thanks in advance for any suggestions!

Here is canu.out:

-- CONFIGURE CANU
--
-- Detected Java(TM) Runtime Environment '1.8.0_121' (from '/ysm-gpfs/apps/software/Java/1.8.0_121/bin/java').
-- Detected gnuplot version '4.6 patchlevel 2' (from 'gnuplot') and image format 'png'.
-- Detected 20 CPUs and 125 gigabytes of memory.
-- Detected Slurm with 'sinfo' binary in /usr/bin/sinfo.
-- Detected Slurm with 'MaxArraySize' limited to 10000 jobs.
-- 
-- Found  21 hosts with   8 cores and  121 GB memory under Slurm control.
-- Found  14 hosts with  32 cores and  499 GB memory under Slurm control.
-- Found 124 hosts with   8 cores and   43 GB memory under Slurm control.
-- Found  34 hosts with  28 cores and  246 GB memory under Slurm control.
-- Found  59 hosts with  16 cores and  121 GB memory under Slurm control.
-- Found 120 hosts with  20 cores and  121 GB memory under Slurm control.
-- Found   2 hosts with  32 cores and 1507 GB memory under Slurm control.
-- Found   1 host  with   8 cores and   58 GB memory under Slurm control.
--
--                     (tag)Threads
--            (tag)Memory         |
--        (tag)         |         |  algorithm
--        -------  ------  --------  -----------------------------
-- Grid:  meryl     90 GB   20 CPUs  (k-mer counting)
-- Grid:  cormhap   21 GB    4 CPUs  (overlap detection with mhap)
-- Grid:  obtovl    24 GB    4 CPUs  (overlap detection)
-- Grid:  utgovl    24 GB    4 CPUs  (overlap detection)
-- Grid:  cor        6 GB    4 CPUs  (read correction)
-- Grid:  ovb        4 GB    1 CPU   (overlap store bucketizer)
-- Grid:  ovs       32 GB    1 CPU   (overlap store sorting)
-- Grid:  red       16 GB    4 CPUs  (read error detection)
-- Grid:  oea        8 GB    1 CPU   (overlap error adjustment)
-- Grid:  bat      100 GB   20 CPUs  (contig construction)
-- Grid:  gfa       32 GB   20 CPUs  (GFA alignment and processing)
--
-- In 'physalia.gkpStore', found PacBio reads:
--   Raw:        15006194
--   Corrected:  0
--   Trimmed:    0
--
-- Generating assembly 'physalia' in '/gpfs/ysm/scratch60/zrl3/physalia_canu/physalia_assembly_10'
--
-- Parameters:
--
--  genomeSize        3300000000
--
--  Overlap Generation Limits:
--    corOvlErrorRate 0.2400 ( 24.00%)
--    obtOvlErrorRate 0.0650 (  6.50%)
--    utgOvlErrorRate 0.0650 (  6.50%)
--
--  Overlap Processing Limits:
--    corErrorRate    0.3000 ( 30.00%)
--    obtErrorRate    0.0650 (  6.50%)
--    utgErrorRate    0.0650 (  6.50%)
--    cnsErrorRate    0.0650 (  6.50%)
--
--
-- BEGIN CORRECTION
--
--
-- Mhap overlap jobs failed, retry.
--   job correction/1-overlapper/results/000001.ovb FAILED.
--   job correction/1-overlapper/results/000002.ovb FAILED.
--   job correction/1-overlapper/results/000003.ovb FAILED.
--   job correction/1-overlapper/results/000004.ovb FAILED.
--   job correction/1-overlapper/results/000005.ovb FAILED.
--   job correction/1-overlapper/results/000006.ovb FAILED.
--   job correction/1-overlapper/results/000007.ovb FAILED.
--   job correction/1-overlapper/results/000008.ovb FAILED.
--   job correction/1-overlapper/results/000009.ovb FAILED.
--   job correction/1-overlapper/results/000010.ovb FAILED.
--   job correction/1-overlapper/results/000011.ovb FAILED.
--   job correction/1-overlapper/results/000012.ovb FAILED.
--   job correction/1-overlapper/results/000013.ovb FAILED.
--   job correction/1-overlapper/results/000014.ovb FAILED.
--   job correction/1-overlapper/results/000015.ovb FAILED.
--   job correction/1-overlapper/results/000016.ovb FAILED.
--   job correction/1-overlapper/results/000017.ovb FAILED.
--   job correction/1-overlapper/results/000018.ovb FAILED.
--   job correction/1-overlapper/results/000019.ovb FAILED.
--   job correction/1-overlapper/results/000020.ovb FAILED.
--   job correction/1-overlapper/results/000021.ovb FAILED.
--   job correction/1-overlapper/results/000022.ovb FAILED.
--   job correction/1-overlapper/results/000023.ovb FAILED.
--   job correction/1-overlapper/results/000024.ovb FAILED.
--   job correction/1-overlapper/results/000025.ovb FAILED.
--   job correction/1-overlapper/results/000026.ovb FAILED.
--   job correction/1-overlapper/results/000027.ovb FAILED.
--   job correction/1-overlapper/results/000028.ovb FAILED.
--   job correction/1-overlapper/results/000029.ovb FAILED.
--   job correction/1-overlapper/results/000030.ovb FAILED.
--   job correction/1-overlapper/results/000031.ovb FAILED.
--   job correction/1-overlapper/results/000032.ovb FAILED.
--   job correction/1-overlapper/results/000033.ovb FAILED.
--   job correction/1-overlapper/results/000034.ovb FAILED.
--   job correction/1-overlapper/results/000035.ovb FAILED.
--   job correction/1-overlapper/results/000036.ovb FAILED.
--   job correction/1-overlapper/results/000037.ovb FAILED.
--   job correction/1-overlapper/results/000038.ovb FAILED.
--   job correction/1-overlapper/results/000039.ovb FAILED.
--   job correction/1-overlapper/results/000040.ovb FAILED.
--   job correction/1-overlapper/results/000041.ovb FAILED.
--   job correction/1-overlapper/results/000042.ovb FAILED.
--   job correction/1-overlapper/results/000043.ovb FAILED.
--   job correction/1-overlapper/results/000044.ovb FAILED.
--   job correction/1-overlapper/results/000045.ovb FAILED.
--   job correction/1-overlapper/results/000046.ovb FAILED.
--   job correction/1-overlapper/results/000047.ovb FAILED.
--   job correction/1-overlapper/results/000048.ovb FAILED.
--   job correction/1-overlapper/results/000049.ovb FAILED.
--   job correction/1-overlapper/results/000050.ovb FAILED.
--   job correction/1-overlapper/results/000051.ovb FAILED.
--   job correction/1-overlapper/results/000052.ovb FAILED.
--   job correction/1-overlapper/results/000053.ovb FAILED.
--   job correction/1-overlapper/results/000054.ovb FAILED.
--   job correction/1-overlapper/results/000055.ovb FAILED.
--   job correction/1-overlapper/results/000056.ovb FAILED.
--   job correction/1-overlapper/results/000057.ovb FAILED.
--   job correction/1-overlapper/results/000058.ovb FAILED.
--   job correction/1-overlapper/results/000059.ovb FAILED.
--   job correction/1-overlapper/results/000060.ovb FAILED.
--   job correction/1-overlapper/results/000061.ovb FAILED.
--   job correction/1-overlapper/results/000062.ovb FAILED.
--   job correction/1-overlapper/results/000063.ovb FAILED.
--   job correction/1-overlapper/results/000064.ovb FAILED.
--   job correction/1-overlapper/results/000065.ovb FAILED.
--   job correction/1-overlapper/results/000066.ovb FAILED.
--   job correction/1-overlapper/results/000067.ovb FAILED.
--   job correction/1-overlapper/results/000068.ovb FAILED.
--   job correction/1-overlapper/results/000069.ovb FAILED.
--   job correction/1-overlapper/results/000070.ovb FAILED.
--   job correction/1-overlapper/results/000071.ovb FAILED.
--   job correction/1-overlapper/results/000072.ovb FAILED.
--   job correction/1-overlapper/results/000073.ovb FAILED.
--   job correction/1-overlapper/results/000074.ovb FAILED.
--   job correction/1-overlapper/results/000075.ovb FAILED.
--   job correction/1-overlapper/results/000076.ovb FAILED.
--   job correction/1-overlapper/results/000077.ovb FAILED.
--   job correction/1-overlapper/results/000078.ovb FAILED.
--   job correction/1-overlapper/results/000079.ovb FAILED.
--   job correction/1-overlapper/results/000080.ovb FAILED.
--   job correction/1-overlapper/results/000081.ovb FAILED.
--   job correction/1-overlapper/results/000082.ovb FAILED.
--   job correction/1-overlapper/results/000083.ovb FAILED.
--   job correction/1-overlapper/results/000085.ovb FAILED.
--   job correction/1-overlapper/results/000086.ovb FAILED.
--   job correction/1-overlapper/results/000087.ovb FAILED.
--   job correction/1-overlapper/results/000089.ovb FAILED.
--   job correction/1-overlapper/results/000090.ovb FAILED.
--   job correction/1-overlapper/results/000091.ovb FAILED.
--   job correction/1-overlapper/results/000092.ovb FAILED.
--   job correction/1-overlapper/results/000093.ovb FAILED.
--   job correction/1-overlapper/results/000094.ovb FAILED.
--   job correction/1-overlapper/results/000095.ovb FAILED.
--   job correction/1-overlapper/results/000097.ovb FAILED.
--   job correction/1-overlapper/results/000098.ovb FAILED.
--   job correction/1-overlapper/results/000099.ovb FAILED.
--   job correction/1-overlapper/results/000101.ovb FAILED.
--   job correction/1-overlapper/results/000102.ovb FAILED.
--   job correction/1-overlapper/results/000103.ovb FAILED.
--   job correction/1-overlapper/results/000105.ovb FAILED.
--   job correction/1-overlapper/results/000106.ovb FAILED.
--   job correction/1-overlapper/results/000107.ovb FAILED.
--   job correction/1-overlapper/results/000109.ovb FAILED.
--   job correction/1-overlapper/results/000110.ovb FAILED.
--   job correction/1-overlapper/results/000111.ovb FAILED.
--   job correction/1-overlapper/results/000113.ovb FAILED.
--   job correction/1-overlapper/results/000114.ovb FAILED.
--   job correction/1-overlapper/results/000115.ovb FAILED.
--   job correction/1-overlapper/results/000117.ovb FAILED.
--   job correction/1-overlapper/results/000118.ovb FAILED.
--   job correction/1-overlapper/results/000119.ovb FAILED.
--   job correction/1-overlapper/results/000121.ovb FAILED.
--   job correction/1-overlapper/results/000122.ovb FAILED.
--   job correction/1-overlapper/results/000123.ovb FAILED.
--   job correction/1-overlapper/results/000125.ovb FAILED.
--   job correction/1-overlapper/results/000126.ovb FAILED.
--   job correction/1-overlapper/results/000127.ovb FAILED.
--   job correction/1-overlapper/results/000129.ovb FAILED.
--   job correction/1-overlapper/results/000130.ovb FAILED.
--   job correction/1-overlapper/results/000131.ovb FAILED.
--   job correction/1-overlapper/results/000133.ovb FAILED.
--   job correction/1-overlapper/results/000134.ovb FAILED.
--   job correction/1-overlapper/results/000135.ovb FAILED.
--   job correction/1-overlapper/results/000137.ovb FAILED.
--   job correction/1-overlapper/results/000138.ovb FAILED.
--   job correction/1-overlapper/results/000139.ovb FAILED.
--   job correction/1-overlapper/results/000141.ovb FAILED.
--   job correction/1-overlapper/results/000142.ovb FAILED.
--   job correction/1-overlapper/results/000143.ovb FAILED.
--   job correction/1-overlapper/results/000145.ovb FAILED.
--   job correction/1-overlapper/results/000146.ovb FAILED.
--   job correction/1-overlapper/results/000147.ovb FAILED.
--   job correction/1-overlapper/results/000149.ovb FAILED.
--   job correction/1-overlapper/results/000150.ovb FAILED.
--   job correction/1-overlapper/results/000151.ovb FAILED.
--   job correction/1-overlapper/results/000153.ovb FAILED.
--   job correction/1-overlapper/results/000154.ovb FAILED.
--   job correction/1-overlapper/results/000155.ovb FAILED.
--   job correction/1-overlapper/results/000157.ovb FAILED.
--   job correction/1-overlapper/results/000158.ovb FAILED.
--   job correction/1-overlapper/results/000159.ovb FAILED.
--   job correction/1-overlapper/results/000161.ovb FAILED.
--   job correction/1-overlapper/results/000162.ovb FAILED.
--   job correction/1-overlapper/results/000163.ovb FAILED.
--   job correction/1-overlapper/results/000164.ovb FAILED.
--   job correction/1-overlapper/results/000165.ovb FAILED.
--   job correction/1-overlapper/results/000166.ovb FAILED.
--   job correction/1-overlapper/results/000167.ovb FAILED.
--   job correction/1-overlapper/results/000168.ovb FAILED.
--   job correction/1-overlapper/results/000169.ovb FAILED.
--   job correction/1-overlapper/results/000170.ovb FAILED.
--   job correction/1-overlapper/results/000171.ovb FAILED.
--   job correction/1-overlapper/results/000172.ovb FAILED.
--   job correction/1-overlapper/results/000173.ovb FAILED.
--   job correction/1-overlapper/results/000174.ovb FAILED.
--   job correction/1-overlapper/results/000175.ovb FAILED.
--   job correction/1-overlapper/results/000176.ovb FAILED.
--   job correction/1-overlapper/results/000177.ovb FAILED.
--   job correction/1-overlapper/results/000178.ovb FAILED.
--   job correction/1-overlapper/results/000179.ovb FAILED.
--   job correction/1-overlapper/results/000180.ovb FAILED.
--   job correction/1-overlapper/results/000181.ovb FAILED.
--   job correction/1-overlapper/results/000182.ovb FAILED.
--   job correction/1-overlapper/results/000183.ovb FAILED.
--   job correction/1-overlapper/results/000184.ovb FAILED.
--   job correction/1-overlapper/results/000185.ovb FAILED.
--   job correction/1-overlapper/results/000186.ovb FAILED.
--   job correction/1-overlapper/results/000187.ovb FAILED.
--   job correction/1-overlapper/results/000188.ovb FAILED.
--   job correction/1-overlapper/results/000189.ovb FAILED.
--   job correction/1-overlapper/results/000190.ovb FAILED.
--   job correction/1-overlapper/results/000191.ovb FAILED.
--   job correction/1-overlapper/results/000192.ovb FAILED.
--   job correction/1-overlapper/results/000193.ovb FAILED.
--   job correction/1-overlapper/results/000194.ovb FAILED.
--   job correction/1-overlapper/results/000195.ovb FAILED.
--   job correction/1-overlapper/results/000196.ovb FAILED.
--   job correction/1-overlapper/results/000197.ovb FAILED.
--   job correction/1-overlapper/results/000198.ovb FAILED.
--   job correction/1-overlapper/results/000199.ovb FAILED.
--   job correction/1-overlapper/results/000200.ovb FAILED.
--   job correction/1-overlapper/results/000201.ovb FAILED.
--   job correction/1-overlapper/results/000202.ovb FAILED.
--   job correction/1-overlapper/results/000203.ovb FAILED.
--   job correction/1-overlapper/results/000204.ovb FAILED.
--   job correction/1-overlapper/results/000205.ovb FAILED.
--   job correction/1-overlapper/results/000206.ovb FAILED.
--   job correction/1-overlapper/results/000207.ovb FAILED.
--   job correction/1-overlapper/results/000208.ovb FAILED.
--   job correction/1-overlapper/results/000209.ovb FAILED.
--   job correction/1-overlapper/results/000210.ovb FAILED.
--   job correction/1-overlapper/results/000211.ovb FAILED.
--   job correction/1-overlapper/results/000212.ovb FAILED.
--   job correction/1-overlapper/results/000213.ovb FAILED.
--   job correction/1-overlapper/results/000214.ovb FAILED.
--   job correction/1-overlapper/results/000215.ovb FAILED.
--   job correction/1-overlapper/results/000216.ovb FAILED.
--   job correction/1-overlapper/results/000217.ovb FAILED.
--   job correction/1-overlapper/results/000218.ovb FAILED.
--   job correction/1-overlapper/results/000219.ovb FAILED.
--   job correction/1-overlapper/results/000220.ovb FAILED.
--   job correction/1-overlapper/results/000221.ovb FAILED.
--   job correction/1-overlapper/results/000222.ovb FAILED.
--   job correction/1-overlapper/results/000223.ovb FAILED.
--   job correction/1-overlapper/results/000225.ovb FAILED.
--   job correction/1-overlapper/results/000226.ovb FAILED.
--   job correction/1-overlapper/results/000227.ovb FAILED.
--   job correction/1-overlapper/results/000229.ovb FAILED.
--   job correction/1-overlapper/results/000230.ovb FAILED.
--   job correction/1-overlapper/results/000231.ovb FAILED.
--   job correction/1-overlapper/results/000233.ovb FAILED.
--   job correction/1-overlapper/results/000234.ovb FAILED.
--   job correction/1-overlapper/results/000235.ovb FAILED.
--   job correction/1-overlapper/results/000236.ovb FAILED.
--   job correction/1-overlapper/results/000237.ovb FAILED.
--   job correction/1-overlapper/results/000238.ovb FAILED.
--   job correction/1-overlapper/results/000239.ovb FAILED.
--   job correction/1-overlapper/results/000240.ovb FAILED.
--   job correction/1-overlapper/results/000241.ovb FAILED.
--   job correction/1-overlapper/results/000242.ovb FAILED.
--   job correction/1-overlapper/results/000243.ovb FAILED.
--   job correction/1-overlapper/results/000244.ovb FAILED.
--   job correction/1-overlapper/results/000245.ovb FAILED.
--   job correction/1-overlapper/results/000246.ovb FAILED.
--   job correction/1-overlapper/results/000247.ovb FAILED.
--   job correction/1-overlapper/results/000248.ovb FAILED.
--   job correction/1-overlapper/results/000249.ovb FAILED.
--   job correction/1-overlapper/results/000250.ovb FAILED.
--   job correction/1-overlapper/results/000251.ovb FAILED.
--   job correction/1-overlapper/results/000252.ovb FAILED.
--   job correction/1-overlapper/results/000253.ovb FAILED.
--   job correction/1-overlapper/results/000254.ovb FAILED.
--   job correction/1-overlapper/results/000255.ovb FAILED.
--   job correction/1-overlapper/results/000256.ovb FAILED.
--   job correction/1-overlapper/results/000257.ovb FAILED.
--   job correction/1-overlapper/results/000258.ovb FAILED.
--   job correction/1-overlapper/results/000259.ovb FAILED.
--   job correction/1-overlapper/results/000260.ovb FAILED.
--   job correction/1-overlapper/results/000261.ovb FAILED.
--   job correction/1-overlapper/results/000262.ovb FAILED.
--   job correction/1-overlapper/results/000263.ovb FAILED.
--   job correction/1-overlapper/results/000264.ovb FAILED.
--   job correction/1-overlapper/results/000265.ovb FAILED.
--   job correction/1-overlapper/results/000266.ovb FAILED.
--   job correction/1-overlapper/results/000267.ovb FAILED.
--   job correction/1-overlapper/results/000268.ovb FAILED.
--   job correction/1-overlapper/results/000269.ovb FAILED.
--   job correction/1-overlapper/results/000270.ovb FAILED.
--   job correction/1-overlapper/results/000271.ovb FAILED.
--   job correction/1-overlapper/results/000272.ovb FAILED.
--   job correction/1-overlapper/results/000273.ovb FAILED.
--   job correction/1-overlapper/results/000274.ovb FAILED.
--   job correction/1-overlapper/results/000275.ovb FAILED.
--   job correction/1-overlapper/results/000276.ovb FAILED.
--   job correction/1-overlapper/results/000277.ovb FAILED.
--   job correction/1-overlapper/results/000278.ovb FAILED.
--   job correction/1-overlapper/results/000279.ovb FAILED.
--   job correction/1-overlapper/results/000280.ovb FAILED.
--   job correction/1-overlapper/results/000281.ovb FAILED.
--   job correction/1-overlapper/results/000282.ovb FAILED.
--   job correction/1-overlapper/results/000283.ovb FAILED.
--   job correction/1-overlapper/results/000284.ovb FAILED.
--   job correction/1-overlapper/results/000285.ovb FAILED.
--   job correction/1-overlapper/results/000286.ovb FAILED.
--   job correction/1-overlapper/results/000287.ovb FAILED.
--   job correction/1-overlapper/results/000288.ovb FAILED.
--   job correction/1-overlapper/results/000289.ovb FAILED.
--   job correction/1-overlapper/results/000290.ovb FAILED.
--   job correction/1-overlapper/results/000291.ovb FAILED.
--   job correction/1-overlapper/results/000292.ovb FAILED.
--   job correction/1-overlapper/results/000293.ovb FAILED.
--   job correction/1-overlapper/results/000294.ovb FAILED.
--   job correction/1-overlapper/results/000295.ovb FAILED.
--   job correction/1-overlapper/results/000296.ovb FAILED.
--   job correction/1-overlapper/results/000297.ovb FAILED.
--   job correction/1-overlapper/results/000298.ovb FAILED.
--   job correction/1-overlapper/results/000299.ovb FAILED.
--   job correction/1-overlapper/results/000300.ovb FAILED.
--   job correction/1-overlapper/results/000302.ovb FAILED.
--   job correction/1-overlapper/results/000303.ovb FAILED.
--   job correction/1-overlapper/results/000305.ovb FAILED.
--   job correction/1-overlapper/results/000306.ovb FAILED.
--   job correction/1-overlapper/results/000307.ovb FAILED.
--   job correction/1-overlapper/results/000308.ovb FAILED.
--   job correction/1-overlapper/results/000309.ovb FAILED.
--   job correction/1-overlapper/results/000311.ovb FAILED.
--   job correction/1-overlapper/results/000312.ovb FAILED.
--   job correction/1-overlapper/results/000314.ovb FAILED.
--   job correction/1-overlapper/results/000315.ovb FAILED.
--   job correction/1-overlapper/results/000317.ovb FAILED.
--   job correction/1-overlapper/results/000318.ovb FAILED.
--   job correction/1-overlapper/results/000320.ovb FAILED.
--   job correction/1-overlapper/results/000321.ovb FAILED.
--   job correction/1-overlapper/results/000323.ovb FAILED.
--   job correction/1-overlapper/results/000324.ovb FAILED.
--   job correction/1-overlapper/results/000326.ovb FAILED.
--   job correction/1-overlapper/results/000327.ovb FAILED.
--   job correction/1-overlapper/results/000329.ovb FAILED.
--   job correction/1-overlapper/results/000330.ovb FAILED.
--   job correction/1-overlapper/results/000332.ovb FAILED.
--   job correction/1-overlapper/results/000333.ovb FAILED.
--   job correction/1-overlapper/results/000335.ovb FAILED.
--   job correction/1-overlapper/results/000336.ovb FAILED.
--   job correction/1-overlapper/results/000338.ovb FAILED.
--   job correction/1-overlapper/results/000339.ovb FAILED.
--   job correction/1-overlapper/results/000341.ovb FAILED.
--   job correction/1-overlapper/results/000342.ovb FAILED.
--   job correction/1-overlapper/results/000344.ovb FAILED.
--   job correction/1-overlapper/results/000345.ovb FAILED.
--   job correction/1-overlapper/results/000347.ovb FAILED.
--   job correction/1-overlapper/results/000348.ovb FAILED.
--   job correction/1-overlapper/results/000350.ovb FAILED.
--   job correction/1-overlapper/results/000351.ovb FAILED.
--   job correction/1-overlapper/results/000353.ovb FAILED.
--   job correction/1-overlapper/results/000354.ovb FAILED.
--   job correction/1-overlapper/results/000356.ovb FAILED.
--   job correction/1-overlapper/results/000357.ovb FAILED.
--   job correction/1-overlapper/results/000359.ovb FAILED.
--   job correction/1-overlapper/results/000360.ovb FAILED.
--   job correction/1-overlapper/results/000362.ovb FAILED.
--   job correction/1-overlapper/results/000363.ovb FAILED.
--   job correction/1-overlapper/results/000365.ovb FAILED.
--   job correction/1-overlapper/results/000366.ovb FAILED.
--   job correction/1-overlapper/results/000368.ovb FAILED.
--   job correction/1-overlapper/results/000369.ovb FAILED.
--   job correction/1-overlapper/results/000371.ovb FAILED.
--   job correction/1-overlapper/results/000372.ovb FAILED.
--   job correction/1-overlapper/results/000373.ovb FAILED.
--   job correction/1-overlapper/results/000374.ovb FAILED.
--   job correction/1-overlapper/results/000375.ovb FAILED.
--   job correction/1-overlapper/results/000376.ovb FAILED.
--   job correction/1-overlapper/results/000377.ovb FAILED.
--   job correction/1-overlapper/results/000378.ovb FAILED.
--   job correction/1-overlapper/results/000379.ovb FAILED.
--   job correction/1-overlapper/results/000380.ovb FAILED.
--   job correction/1-overlapper/results/000381.ovb FAILED.
--   job correction/1-overlapper/results/000382.ovb FAILED.
--   job correction/1-overlapper/results/000383.ovb FAILED.
--   job correction/1-overlapper/results/000384.ovb FAILED.
--   job correction/1-overlapper/results/000386.ovb FAILED.
--   job correction/1-overlapper/results/000387.ovb FAILED.
--   job correction/1-overlapper/results/000389.ovb FAILED.
--   job correction/1-overlapper/results/000390.ovb FAILED.
--   job correction/1-overlapper/results/000392.ovb FAILED.
--   job correction/1-overlapper/results/000395.ovb FAILED.
--   job correction/1-overlapper/results/000396.ovb FAILED.
--   job correction/1-overlapper/results/000398.ovb FAILED.
--   job correction/1-overlapper/results/000399.ovb FAILED.
--   job correction/1-overlapper/results/000401.ovb FAILED.
--   job correction/1-overlapper/results/000402.ovb FAILED.
--   job correction/1-overlapper/results/000404.ovb FAILED.
--   job correction/1-overlapper/results/000405.ovb FAILED.
--   job correction/1-overlapper/results/000407.ovb FAILED.
--   job correction/1-overlapper/results/000408.ovb FAILED.
--   job correction/1-overlapper/results/000410.ovb FAILED.
--   job correction/1-overlapper/results/000411.ovb FAILED.
--   job correction/1-overlapper/results/000413.ovb FAILED.
--   job correction/1-overlapper/results/000414.ovb FAILED.
--   job correction/1-overlapper/results/000415.ovb FAILED.
--   job correction/1-overlapper/results/000416.ovb FAILED.
--   job correction/1-overlapper/results/000417.ovb FAILED.
--   job correction/1-overlapper/results/000418.ovb FAILED.
--   job correction/1-overlapper/results/000419.ovb FAILED.
--   job correction/1-overlapper/results/000420.ovb FAILED.
--   job correction/1-overlapper/results/000421.ovb FAILED.
--   job correction/1-overlapper/results/000422.ovb FAILED.
--   job correction/1-overlapper/results/000423.ovb FAILED.
--   job correction/1-overlapper/results/000424.ovb FAILED.
--   job correction/1-overlapper/results/000425.ovb FAILED.
--   job correction/1-overlapper/results/000426.ovb FAILED.
--   job correction/1-overlapper/results/000427.ovb FAILED.
--   job correction/1-overlapper/results/000428.ovb FAILED.
--   job correction/1-overlapper/results/000429.ovb FAILED.
--   job correction/1-overlapper/results/000430.ovb FAILED.
--   job correction/1-overlapper/results/000431.ovb FAILED.
--   job correction/1-overlapper/results/000432.ovb FAILED.
--   job correction/1-overlapper/results/000433.ovb FAILED.
--   job correction/1-overlapper/results/000434.ovb FAILED.
--   job correction/1-overlapper/results/000435.ovb FAILED.
--   job correction/1-overlapper/results/000436.ovb FAILED.
--   job correction/1-overlapper/results/000437.ovb FAILED.
--   job correction/1-overlapper/results/000438.ovb FAILED.
--   job correction/1-overlapper/results/000439.ovb FAILED.
--   job correction/1-overlapper/results/000440.ovb FAILED.
--   job correction/1-overlapper/results/000441.ovb FAILED.
--   job correction/1-overlapper/results/000442.ovb FAILED.
--   job correction/1-overlapper/results/000443.ovb FAILED.
--   job correction/1-overlapper/results/000444.ovb FAILED.
--   job correction/1-overlapper/results/000445.ovb FAILED.
--   job correction/1-overlapper/results/000446.ovb FAILED.
--   job correction/1-overlapper/results/000447.ovb FAILED.
--   job correction/1-overlapper/results/000448.ovb FAILED.
--   job correction/1-overlapper/results/000449.ovb FAILED.
--   job correction/1-overlapper/results/000450.ovb FAILED.
--   job correction/1-overlapper/results/000451.ovb FAILED.
--   job correction/1-overlapper/results/000452.ovb FAILED.
--   job correction/1-overlapper/results/000453.ovb FAILED.
--   job correction/1-overlapper/results/000454.ovb FAILED.
--   job correction/1-overlapper/results/000455.ovb FAILED.
--   job correction/1-overlapper/results/000456.ovb FAILED.
--   job correction/1-overlapper/results/000457.ovb FAILED.
--   job correction/1-overlapper/results/000458.ovb FAILED.
--   job correction/1-overlapper/results/000459.ovb FAILED.
--   job correction/1-overlapper/results/000460.ovb FAILED.
--   job correction/1-overlapper/results/000461.ovb FAILED.
--   job correction/1-overlapper/results/000462.ovb FAILED.
--   job correction/1-overlapper/results/000463.ovb FAILED.
--   job correction/1-overlapper/results/000465.ovb FAILED.
--   job correction/1-overlapper/results/000466.ovb FAILED.
--   job correction/1-overlapper/results/000467.ovb FAILED.
--   job correction/1-overlapper/results/000469.ovb FAILED.
--   job correction/1-overlapper/results/000471.ovb FAILED.
--   job correction/1-overlapper/results/000473.ovb FAILED.
--   job correction/1-overlapper/results/000475.ovb FAILED.
--   job correction/1-overlapper/results/000477.ovb FAILED.
--   job correction/1-overlapper/results/000479.ovb FAILED.
--   job correction/1-overlapper/results/000481.ovb FAILED.
--   job correction/1-overlapper/results/000483.ovb FAILED.
--   job correction/1-overlapper/results/000485.ovb FAILED.
--   job correction/1-overlapper/results/000487.ovb FAILED.
--   job correction/1-overlapper/results/000489.ovb FAILED.
--   job correction/1-overlapper/results/000491.ovb FAILED.
--   job correction/1-overlapper/results/000493.ovb FAILED.
--   job correction/1-overlapper/results/000495.ovb FAILED.
--   job correction/1-overlapper/results/000497.ovb FAILED.
--   job correction/1-overlapper/results/000499.ovb FAILED.
--   job correction/1-overlapper/results/000501.ovb FAILED.
--   job correction/1-overlapper/results/000503.ovb FAILED.
--   job correction/1-overlapper/results/000505.ovb FAILED.
--   job correction/1-overlapper/results/000507.ovb FAILED.
--   job correction/1-overlapper/results/000509.ovb FAILED.
--   job correction/1-overlapper/results/000511.ovb FAILED.
--   job correction/1-overlapper/results/000513.ovb FAILED.
--   job correction/1-overlapper/results/000515.ovb FAILED.
--   job correction/1-overlapper/results/000517.ovb FAILED.
--   job correction/1-overlapper/results/000519.ovb FAILED.
--   job correction/1-overlapper/results/000521.ovb FAILED.
--   job correction/1-overlapper/results/000523.ovb FAILED.
--   job correction/1-overlapper/results/000525.ovb FAILED.
--   job correction/1-overlapper/results/000527.ovb FAILED.
--   job correction/1-overlapper/results/000529.ovb FAILED.
--   job correction/1-overlapper/results/000531.ovb FAILED.
--   job correction/1-overlapper/results/000533.ovb FAILED.
--   job correction/1-overlapper/results/000534.ovb FAILED.
--   job correction/1-overlapper/results/000535.ovb FAILED.
--   job correction/1-overlapper/results/000536.ovb FAILED.
--   job correction/1-overlapper/results/000537.ovb FAILED.
--   job correction/1-overlapper/results/000538.ovb FAILED.
--   job correction/1-overlapper/results/000539.ovb FAILED.
--   job correction/1-overlapper/results/000540.ovb FAILED.
--   job correction/1-overlapper/results/000541.ovb FAILED.
--   job correction/1-overlapper/results/000542.ovb FAILED.
--   job correction/1-overlapper/results/000543.ovb FAILED.
--   job correction/1-overlapper/results/000544.ovb FAILED.
--   job correction/1-overlapper/results/000545.ovb FAILED.
--   job correction/1-overlapper/results/000546.ovb FAILED.
--   job correction/1-overlapper/results/000547.ovb FAILED.
--
--
-- Running jobs.  Second attempt out of 2.
--
-- 'mhap.jobSubmit-01.sh' -> job 15311341 tasks 1-83.
-- 'mhap.jobSubmit-02.sh' -> job 15311342 tasks 85-87.
-- 'mhap.jobSubmit-03.sh' -> job 15311343 tasks 89-95.
-- 'mhap.jobSubmit-04.sh' -> job 15311344 tasks 97-99.
-- 'mhap.jobSubmit-05.sh' -> job 15311345 tasks 101-103.
-- 'mhap.jobSubmit-06.sh' -> job 15311346 tasks 105-107.
-- 'mhap.jobSubmit-07.sh' -> job 15311347 tasks 109-111.
-- 'mhap.jobSubmit-08.sh' -> job 15311348 tasks 113-115.
-- 'mhap.jobSubmit-09.sh' -> job 15311349 tasks 117-119.
-- 'mhap.jobSubmit-10.sh' -> job 15311350 tasks 121-123.
-- 'mhap.jobSubmit-11.sh' -> job 15311351 tasks 125-127.
-- 'mhap.jobSubmit-12.sh' -> job 15311352 tasks 129-131.
-- 'mhap.jobSubmit-13.sh' -> job 15311353 tasks 133-135.
-- 'mhap.jobSubmit-14.sh' -> job 15311354 tasks 137-139.
-- 'mhap.jobSubmit-15.sh' -> job 15311355 tasks 141-143.
-- 'mhap.jobSubmit-16.sh' -> job 15311380 tasks 145-147.
-- 'mhap.jobSubmit-17.sh' -> job 15311381 tasks 149-151.
-- 'mhap.jobSubmit-18.sh' -> job 15311382 tasks 153-155.
-- 'mhap.jobSubmit-19.sh' -> job 15311383 tasks 157-159.
-- 'mhap.jobSubmit-20.sh' -> job 15311384 tasks 161-223.
-- 'mhap.jobSubmit-21.sh' -> job 15311385 tasks 225-227.
-- 'mhap.jobSubmit-22.sh' -> job 15311386 tasks 229-231.
-- 'mhap.jobSubmit-23.sh' -> job 15311387 tasks 233-300.
-- 'mhap.jobSubmit-24.sh' -> job 15311388 tasks 302-303.
-- 'mhap.jobSubmit-25.sh' -> job 15311389 tasks 305-309.
-- 'mhap.jobSubmit-26.sh' -> job 15311390 tasks 311-312.
-- 'mhap.jobSubmit-27.sh' -> job 15311391 tasks 314-315.
-- 'mhap.jobSubmit-28.sh' -> job 15311392 tasks 317-318.
-- 'mhap.jobSubmit-29.sh' -> job 15311393 tasks 320-321.
-- 'mhap.jobSubmit-30.sh' -> job 15311394 tasks 323-324.
-- 'mhap.jobSubmit-31.sh' -> job 15311395 tasks 326-327.
-- 'mhap.jobSubmit-32.sh' -> job 15311396 tasks 329-330.
-- 'mhap.jobSubmit-33.sh' -> job 15311399 tasks 332-333.
-- 'mhap.jobSubmit-34.sh' -> job 15311400 tasks 335-336.
-- 'mhap.jobSubmit-35.sh' -> job 15311401 tasks 338-339.
-- 'mhap.jobSubmit-36.sh' -> job 15311402 tasks 341-342.
-- 'mhap.jobSubmit-37.sh' -> job 15311403 tasks 344-345.
-- 'mhap.jobSubmit-38.sh' -> job 15311404 tasks 347-348.
-- 'mhap.jobSubmit-39.sh' -> job 15311405 tasks 350-351.
-- 'mhap.jobSubmit-40.sh' -> job 15311406 tasks 353-354.
-- 'mhap.jobSubmit-41.sh' -> job 15311407 tasks 356-357.
-- 'mhap.jobSubmit-42.sh' -> job 15311408 tasks 359-360.
-- 'mhap.jobSubmit-43.sh' -> job 15311409 tasks 362-363.
-- 'mhap.jobSubmit-44.sh' -> job 15311410 tasks 365-366.
-- 'mhap.jobSubmit-45.sh' -> job 15311411 tasks 368-369.
-- 'mhap.jobSubmit-46.sh' -> job 15311412 tasks 371-384.
-- 'mhap.jobSubmit-47.sh' -> job 15311413 tasks 386-387.
-- 'mhap.jobSubmit-48.sh' -> job 15311414 tasks 389-390.
-- 'mhap.jobSubmit-49.sh' -> job 15311415 task 392.
-- 'mhap.jobSubmit-50.sh' -> job 15311416 tasks 395-396.
-- 'mhap.jobSubmit-51.sh' -> job 15311417 tasks 398-399.
-- 'mhap.jobSubmit-52.sh' -> job 15311418 tasks 401-402.
-- 'mhap.jobSubmit-53.sh' -> job 15311419 tasks 404-405.
-- 'mhap.jobSubmit-54.sh' -> job 15311420 tasks 407-408.
-- 'mhap.jobSubmit-55.sh' -> job 15311421 tasks 410-411.
-- 'mhap.jobSubmit-56.sh' -> job 15311422 tasks 413-463.
-- 'mhap.jobSubmit-57.sh' -> job 15311423 tasks 465-467.
-- 'mhap.jobSubmit-58.sh' -> job 15311424 task 469.
-- 'mhap.jobSubmit-59.sh' -> job 15311425 task 471.
-- 'mhap.jobSubmit-60.sh' -> job 15311426 task 473.
-- 'mhap.jobSubmit-61.sh' -> job 15311427 task 475.
-- 'mhap.jobSubmit-62.sh' -> job 15311428 task 477.
-- 'mhap.jobSubmit-63.sh' -> job 15311429 task 479.
-- 'mhap.jobSubmit-64.sh' -> job 15311430 task 481.
-- 'mhap.jobSubmit-65.sh' -> job 15311431 task 483.
-- 'mhap.jobSubmit-66.sh' -> job 15311432 task 485.
-- 'mhap.jobSubmit-67.sh' -> job 15311433 task 487.
-- 'mhap.jobSubmit-68.sh' -> job 15311434 task 489.
-- 'mhap.jobSubmit-69.sh' -> job 15311435 task 491.
-- 'mhap.jobSubmit-70.sh' -> job 15311436 task 493.
-- 'mhap.jobSubmit-71.sh' -> job 15311437 task 495.
-- 'mhap.jobSubmit-72.sh' -> job 15311438 task 497.
-- 'mhap.jobSubmit-73.sh' -> job 15311439 task 499.
-- 'mhap.jobSubmit-74.sh' -> job 15311441 task 501.
-- 'mhap.jobSubmit-75.sh' -> job 15311442 task 503.
-- 'mhap.jobSubmit-76.sh' -> job 15311443 task 505.
-- 'mhap.jobSubmit-77.sh' -> job 15311444 task 507.
-- 'mhap.jobSubmit-78.sh' -> job 15311445 task 509.
-- 'mhap.jobSubmit-79.sh' -> job 15311446 task 511.
-- 'mhap.jobSubmit-80.sh' -> job 15311447 task 513.
-- 'mhap.jobSubmit-81.sh' -> job 15311448 task 515.
-- 'mhap.jobSubmit-82.sh' -> job 15311449 task 517.
-- 'mhap.jobSubmit-83.sh' -> job 15311450 task 519.
-- 'mhap.jobSubmit-84.sh' -> job 15311451 task 521.
-- 'mhap.jobSubmit-85.sh' -> job 15311452 task 523.
-- 'mhap.jobSubmit-86.sh' -> job 15311453 task 525.
-- 'mhap.jobSubmit-87.sh' -> job 15311454 task 527.
-- 'mhap.jobSubmit-88.sh' -> job 15311455 task 529.
-- 'mhap.jobSubmit-89.sh' -> job 15311456 task 531.
-- 'mhap.jobSubmit-90.sh' -> job 15311457 tasks 533-547.
--
----------------------------------------
-- Starting command on Sat May 19 16:25:55 2018 with 482924.472 GB free disk space
    cd /gpfs/ysm/scratch60/zrl3/physalia_canu/physalia_assembly_10
    sbatch \
      --depend=afterany:15311341:15311342:15311343:15311344:15311345:15311346:15311347:15311348:15311349:15311350:15311351:15311352:15311353:15311354:15311355:15311380:15311381:15311382:15311383:15311384:15311385:15311386:15311387:15311388:15311389:15311390:15311391:15311392:15311393:15311394:15311395:15311396:15311399:15311400:15311401:15311402:15311403:15311404:15311405:15311406:15311407:15311408:15311409:15311410:15311411:15311412:15311413:15311414:15311415:15311416:15311417:15311418:15311419:15311420:15311421:15311422:15311423:15311424:15311425:15311426:15311427:15311428:15311429:15311430:15311431:15311432:15311433:15311434:15311435:15311436:15311437:15311438:15311439:15311441:15311442:15311443:15311444:15311445:15311446:15311447:15311448:15311449:15311450:15311451:15311452:15311453:15311454:15311455:15311456:15311457 \
      --mem-per-cpu=4g \
      --cpus-per-task=1 \
      --partition general  \
      -D `pwd` \
      -J 'canu_physalia' \
      -o canu-scripts/canu.04.out canu-scripts/canu.04.sh
Submitted batch job 15311458
-- Finished on Sat May 19 16:25:55 2018 (lickety-split) with 482924.472 GB free disk space
skoren commented 6 years ago
  1. Whatever jobs are completed won't be run again. Just run the original canu command.
  2. The correctedErrorRate won't affect the read correction. Your rate is set to 0.065 which is not that high, I'd actually raise it even more to 0.105 if this is Sequel data which is lower quality than RSII.
  3. You don't need to manually adjust the time, just tell canu to request more for those jobs (or all jobs if it's easier). Add gridOptions="--time=168:00:00 which will make all jobs ask for 7 days. You can also do gridOptionsCorMhap="--time=168:00:00 to change just this step.
zrlewis commented 6 years ago

@skoren Thanks for your response!

I will cancel and re-run the canu command, increasing walltime and correctedErrorRate.

Is it okay to delete the *.mhap.WORKING files prior to re-starting?

skoren commented 6 years ago

Yep, anything with WORKING in the name is partial output and can be erased.

zrlewis commented 6 years ago

I cancelled running jobs, deleted *.WORKING in correction/1-overlapper/results and resubmitted the following script. All seems to be running fine. Okay to close. Thank you, again!

canu \
    -p physalia -d physalia_assembly_10 \
    genomeSize=3.3g \
    -pacbio-raw $READS/Physalia_concatenated_reads.fasta \
    correctedErrorRate=0.105  \
    gridOptions="--partition general" \
    batMemory=100 batThreads=20 merylMemory=90 merylThreads=20 gfaThreads=20 corMemory=6 cormhapMemory=30 cormhapThreads=6 \
   gridOptionsmeryl="-t 03:00:00" gridOptionscormhap="-t 168:00:00"