search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
514 stars 112 forks source link

Cactus Terminates with error #725

Open Abelcanc3rhack3r opened 2 years ago

Abelcanc3rhack3r commented 2 years ago

Hi,

I just ran cactus using the docker image, on the evolvedmammals.txt example:

wget https://raw.githubusercontent.com/ComparativeGenomicsToolkit/cactus/master/examples/evolverMammals.txt docker run -v $(pwd):/data --rm -it quay.io/comparative-genomics-toolkit/cactus:v2.0.5 cactus /data/jobStore /data/evolverMammals.txt /data/evolverMammals.hal --root mr --binariesMode local

The workflow ran for a while, but it terminated abruptly. Here is the last page of the stdout:

[2022-05-24T08:10:09+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-05-24T08:10:09+0000] [MainThread] [I] [toil] Running Toil version 5.6.0-c34146a6437e4407a61e946e968bcce67 a0ebbca on host 6aefb3504cc8. [2022-05-24T08:10:09+0000] [MainThread] [I] [toil.worker] Working on job 'CactusConsolidated' kind-CactusConsol idated/instance-wwlm4ktd v1 [2022-05-24T08:10:10+0000] [MainThread] [I] [toil.worker] Loaded body Job('CactusConsolidated' kind-CactusConso lidated/instance-wwlm4ktd v1) from description 'CactusConsolidated' kind-CactusConsolidated/instance-wwlm4ktd v1 [2022-05-24T08:10:10+0000] [MainThread] [I] [toil.statsAndLogging] Alignments file: /tmp/17307ece98645d9b9e4d6a ed065d39ae/3185/4141/tmp6axkrjuj.tmp [2022-05-24T08:10:10+0000] [MainThread] [W] [root] Deprecated toil method. Please call "logging.getLevelName" directly. [2022-05-24T08:10:10+0000] [MainThread] [I] [cactus.shared.common] Running the command ['cactus_consolidated', '--sequences', 'simCow_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp53ahh074.tmp simDog_chr6 /tmp/17307ece98 645d9b9e4d6aed065d39ae/3185/4141/tmp3fa27t1w.tmp simHuman_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp74205 cf5.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmphmnerw4r.tmp simRat_chr6 /tmp/17307ece98645d9b 9e4d6aed065d39ae/3185/4141/tmpsm3yxe02.tmp', '--speciesTree', '((simHuman_chr6:0.144018,(simMousechr6:0.084509,simRat chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;', '--logLevel', 'I NFO', '--alignments', '/tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp6axkrjuj.tmp', '--params', '/tmp/17307ece9864 5d9b9e4d6aed065d39ae/3185/4141/tmpllqcmb8j.tmp', '--outputFile', '/tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp0p 3mynf3.tmp', '--outputHalFastaFile', '/tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmppyksfa9w.tmp', '--outputRefere nceFile', '/tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpxp_unip3.tmp', '--outgroupEvents', 'simHumanchr6 simDog chr6 simCow_chr6', '--referenceEvent', 'mr', '--secondaryAlignments', '/tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/ tmpc8yjjvzg.tmp', '--threads', '8'] [2022-05-24T08:10:18+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-05-24T08:10:18+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-TrimAndRecurseOnOutgroups/instance-12z4h8at/file-58cca2d70b0e4f8ba23cc046f9f092d1/tmpaxrm51fc.tmp' to path '/tmp/1 7307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp53ahh074.tmp' [2022-05-24T08:10:18+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-TrimAndRecurseOnOutgroups/instance-f8mm0nlq/file-32eff6c661fa4814a307ada4053c4b62/tmpylkfkf9e.tmp' to path '/tmp/1 7307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp3fa27t1w.tmp' [2022-05-24T08:10:18+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-TrimAndRecurseOnOutgroups/instance-gzizxyas/file-3b32eef54dda450095e08e70b26f20cf/tmp6g17xb40.tmp' to path '/tmp/1 7307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp74205cf5.tmp' [2022-05-24T08:10:18+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-CactusBlastPhase/instance-855oy_oc/cleanup/file-8df0131d02c0463ab46a59f110a14b59/tmp65g7lhnh.tmp' to path '/tmp/17 307ece98645d9b9e4d6aed065d39ae/3185/4141/tmphmnerw4r.tmp' [2022-05-24T08:10:18+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-CactusBlastPhase/instance-855oy_oc/cleanup/file-41eff30eab9b4072b4a8481f39d6584e/tmp3ujqus85.tmp' to path '/tmp/17 307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpsm3yxe02.tmp' [2022-05-24T08:10:18+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-mappingQualityRescoring/instance-2vuvseqf/file-ed187aabe6b34f37ab991fcd9acd6caf/tmpbz83ltl5.tmp' to path '/tmp/173 07ece98645d9b9e4d6aed065d39ae/3185/4141/tmp6axkrjuj.tmp' [2022-05-24T08:10:18+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-mappingQualityRescoring/instance-2vuvseqf/file-6d55433d5a94429bbca53e5fa9762910/tmpp1i0slrg.tmp' to path '/tmp/173 07ece98645d9b9e4d6aed065d39ae/3185/4141/tmpc8yjjvzg.tmp' Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/toil/worker.py", line 405, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 943, in _runner super(RoundedJob, self)._runner(*args, jobStore=jobStore, File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2399, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2317, in _run return self.run(fileStore) File "/usr/local/lib/python3.8/dist-packages/cactus/pipeline/cactus_workflow.py", line 408, in run messages = runCactusConsolidated(seqMap=seqMap, File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 197, in runCactusConsolidated masterMessages = cactus_call(check_output=True, returnStdErr=True, realtimeStderrPrefix='cactus_consolidate d({})'.format(referenceEvent), File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 877, in cactus_call raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out)) RuntimeError: Command /usr/bin/time -v cactus_consolidated --sequences 'simCow_chr6 /tmp/17307ece98645d9b9e4d6a ed065d39ae/3185/4141/tmp53ahh074.tmp simDog_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp3fa27t1w.tmp simHum an_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp74205cf5.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d 39ae/3185/4141/tmphmnerw4r.tmp simRat_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpsm3yxe02.tmp' --speciesTr ee '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.1890 8,simDog_chr6:0.16303)Anc2:0.032898)Anc0;' --logLevel INFO --alignments /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141 /tmp6axkrjuj.tmp --params /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpllqcmb8j.tmp --outputFile /tmp/17307ece986 45d9b9e4d6aed065d39ae/3185/4141/tmp0p3mynf3.tmp --outputHalFastaFile /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tm ppyksfa9w.tmp --outputReferenceFile /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpxp_unip3.tmp --outgroupEvents 's imHuman_chr6 simDog_chr6 simCow_chr6' --referenceEvent mr --secondaryAlignments /tmp/17307ece98645d9b9e4d6aed065d39ae/3 185/4141/tmpc8yjjvzg.tmp --threads 8 exited 132: stdout=, stderr=Params file: /tmp/17307ece98645d9b9e4d6aed065d39ae/318 5/4141/tmpllqcmb8j.tmp Output file string : /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp0p3mynf3.tmp Output hal fasta file string : /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmppyksfa9w.tmp Output reference fasta file string : /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpxp_unip3.tmp Sequence files and events: simCow_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp53ahh074.tmp simDog_c hr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp3fa27t1w.tmp simHuman_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae /3185/4141/tmp74205cf5.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmphmnerw4r.tmp simRat_chr6 /t mp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpsm3yxe02.tmp Alignments file: /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp6axkrjuj.tmp Secondary alignments file: /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpc8yjjvzg.tmp Constraint alignments file: (null) Species tree: ((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,( simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0; Outgroup events: simHuman_chr6 simDog_chr6 simCow_chr6 Reference event: mr Loaded the parameters files, 0 seconds have elapsed Set up the cactus disk, 0 seconds have elapsed Constructed the first flower Going to build the event tree with newick string: ((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6: 0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0; Parsed the tree Constructed the basic event tree Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp53ahh074.tmp to simCow_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp53ahh074.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp53ahh074.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp3fa27t1w.tmp to simDog_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp3fa27t1w.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp3fa27t1w.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp74205cf5.tmp to simHuman_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp74205cf5.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp74205cf5.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmphmnerw4r.tmp to simMouse_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmphmnerw4r.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmphmnerw4r.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpsm3yxe02.tmp to simRat_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpsm3yxe02.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpsm3yxe02.tmp is specified incomplete, the sequences will not be attached Constructed the initial flower with 74 sequences and 10 events with string: (((simHuman_chr6:0.144018,(simMouse _chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)An c0:9.22337e+18)ROOT:9.22337e+18; Established the first Flower in the hierarchy, 0 seconds have elapsed Converted alignment coordinates, 0 seconds have elapsed Stripped the unique IDs, 0 seconds have elapsed Starting annealing round with a minimum chain length of 64 and an alignment trim of 3 There were 53263 blocks in the sequence graph, representing 1528662 total aligned bases Block degree stats: min 1, avg 2.770460, median 3, max 49 Block support stats: min 0.000000, avg 0.851332, median 0.750000, max 2.000000 Starting melting round with a minimum chain length of 2 A melting round is destroying 4670 blocks with an average degree of 3.538116 from chains with length less than

  1. Total aligned bases lost: 16523 Starting melting round with a minimum chain length of 4 A melting round is destroying 5798 blocks with an average degree of 3.009141 from chains with length less than
  2. Total aligned bases lost: 39287 Starting melting round with a minimum chain length of 8 A melting round is destroying 12244 blocks with an average degree of 2.374142 from chains with length less than
    1. Total aligned bases lost: 150276 A melting round is destroying 8293 blocks with an average degree of 2.389123 from chains with length less than

  3. Total aligned bases lost: 209185 Destroying 350 recoverable blocks The blocks covered 7464 columns for a total of 16261 aligned bases Destroying 0 recoverable blocks The blocks covered 0 columns for a total of 0 aligned bases Ran cactus caf, 7 seconds have elapsed Ran extended flowers ready for bar, 8 seconds have elapsed Command terminated by signal 4 Command being timed: "cactus_consolidated --sequences simCow_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae /3185/4141/tmp53ahh074.tmp simDog_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp3fa27t1w.tmp simHuman_chr6 /t mp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp74205cf5.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/ 4141/tmphmnerw4r.tmp simRat_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpsm3yxe02.tmp --speciesTree ((simHum an_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr 6:0.16303)Anc2:0.032898)Anc0; --logLevel INFO --alignments /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmp6axkrjuj. tmp --params /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpllqcmb8j.tmp --outputFile /tmp/17307ece98645d9b9e4d6aed 065d39ae/3185/4141/tmp0p3mynf3.tmp --outputHalFastaFile /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmppyksfa9w.tmp --outputReferenceFile /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpxp_unip3.tmp --outgroupEvents simHuman_chr6 s imDog_chr6 simCow_chr6 --referenceEvent mr --secondaryAlignments /tmp/17307ece98645d9b9e4d6aed065d39ae/3185/4141/tmpc8y jjvzg.tmp --threads 8" User time (seconds): 7.70 System time (seconds): 0.11 Percent of CPU this job got: 96% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.09 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 76000 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 17870 Voluntary context switches: 26 Involuntary context switches: 25 Swaps: 0 File system inputs: 0 File system outputs: 5680 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

    [2022-05-24T08:10:18+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 6a efb3504cc8 <========= [2022-05-24T08:10:19+0000] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'C actusConsolidated' kind-CactusConsolidated/instance-wwlm4ktd v2 with ID kind-CactusConsolidated/instance-wwlm4ktd to 1 [2022-05-24T08:10:19+0000] [MainThread] [W] [toil.job] We have increased the default memory of the failed job 'CactusCo nsolidated' kind-CactusConsolidated/instance-wwlm4ktd v2 to 2147483648 bytes [2022-05-24T08:10:19+0000] [MainThread] [W] [toil.job] We have increased the disk of the failed job 'CactusConsolidated ' kind-CactusConsolidated/instance-wwlm4ktd v2 to the default of 2147483648 bytes [2022-05-24T08:10:19+0000] [MainThread] [I] [toil.leader] Issued job 'CactusConsolidated' kind-CactusConsolidated/insta nce-wwlm4ktd v3 with job batch system ID: 192 and cores: 8, disk: 2.0 Gi, and memory: 2.0 Gi [2022-05-24T08:10:20+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/17307ece98645d9b9e4d6aed065d39ae/ 7f83/worker_log.txt [2022-05-24T08:10:30+0000] [Thread-1 ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker CactusConsolidated file:/data/jobStore2 kind-CactusConsolidated/instance-wwlm4ktd. [2022-05-24T08:10:30+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CactusConsolidated' kind-Cactu sConsolidated/instance-wwlm4ktd v3 Exit reason: None [2022-05-24T08:10:30+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'C actusConsolidated' kind-CactusConsolidated/instance-wwlm4ktd v5 [2022-05-24T08:10:30+0000] [MainThread] [W] [toil.leader] Log from job "kind-CactusConsolidated/instance-wwlm4ktd" foll ows: =========> [2022-05-24T08:10:20+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-05-24T08:10:20+0000] [MainThread] [I] [toil] Running Toil version 5.6.0-c34146a6437e4407a61e946e968bcce67 a0ebbca on host 6aefb3504cc8. [2022-05-24T08:10:20+0000] [MainThread] [I] [toil.worker] Working on job 'CactusConsolidated' kind-CactusConsol idated/instance-wwlm4ktd v4 [2022-05-24T08:10:21+0000] [MainThread] [I] [toil.worker] Loaded body Job('CactusConsolidated' kind-CactusConso lidated/instance-wwlm4ktd v4) from description 'CactusConsolidated' kind-CactusConsolidated/instance-wwlm4ktd v4 [2022-05-24T08:10:21+0000] [MainThread] [I] [toil.statsAndLogging] Alignments file: /tmp/17307ece98645d9b9e4d6a ed065d39ae/7f83/3bbc/tmpyk1uyzgo.tmp [2022-05-24T08:10:21+0000] [MainThread] [W] [root] Deprecated toil method. Please call "logging.getLevelName" directly. [2022-05-24T08:10:21+0000] [MainThread] [I] [cactus.shared.common] Running the command ['cactus_consolidated', '--sequences', 'simCow_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp simDog_chr6 /tmp/17307ece98 645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp simHuman_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdrep h01.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp simRat_chr6 /tmp/17307ece98645d9b 9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp', '--speciesTree', '((simHuman_chr6:0.144018,(simMousechr6:0.084509,simRat chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDogchr6:0.16303)Anc2:0.032898)Anc0;', '--logLevel', 'I NFO', '--alignments', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpyk1uyzgo.tmp', '--params', '/tmp/17307ece9864 5d9b9e4d6aed065d39ae/7f83/3bbc/tmpi0061dtu.tmp', '--outputFile', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpr4 s0r53.tmp', '--outputHalFastaFile', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpzeaqu6nq.tmp', '--outputRefere nceFile', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp23htpi2x.tmp', '--outgroupEvents', 'simHumanchr6 simDog chr6 simCow_chr6', '--referenceEvent', 'mr', '--secondaryAlignments', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/ tmp6cujpzld.tmp', '--threads', '8'] [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-TrimAndRecurseOnOutgroups/instance-12z4h8at/file-58cca2d70b0e4f8ba23cc046f9f092d1/tmpaxrm51fc.tmp' to path '/tmp/1 7307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-TrimAndRecurseOnOutgroups/instance-f8mm0nlq/file-32eff6c661fa4814a307ada4053c4b62/tmpylkfkf9e.tmp' to path '/tmp/1 7307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-TrimAndRecurseOnOutgroups/instance-gzizxyas/file-3b32eef54dda450095e08e70b26f20cf/tmp6g17xb40.tmp' to path '/tmp/1 7307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-CactusBlastPhase/instance-855oy_oc/cleanup/file-8df0131d02c0463ab46a59f110a14b59/tmp65g7lhnh.tmp' to path '/tmp/17 307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-CactusBlastPhase/instance-855oy_oc/cleanup/file-41eff30eab9b4072b4a8481f39d6584e/tmp3ujqus85.tmp' to path '/tmp/17 307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-mappingQualityRescoring/instance-2vuvseqf/file-ed187aabe6b34f37ab991fcd9acd6caf/tmpbz83ltl5.tmp' to path '/tmp/173 07ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpyk1uyzgo.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/ kind-mappingQualityRescoring/instance-2vuvseqf/file-6d55433d5a94429bbca53e5fa9762910/tmpp1i0slrg.tmp' to path '/tmp/173 07ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp6cujpzld.tmp' Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/toil/worker.py", line 405, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 943, in _runner super(RoundedJob, self)._runner(*args, jobStore=jobStore, File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2399, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2317, in _run return self.run(fileStore) File "/usr/local/lib/python3.8/dist-packages/cactus/pipeline/cactus_workflow.py", line 408, in run messages = runCactusConsolidated(seqMap=seqMap, File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 197, in runCactusConsolidated masterMessages = cactus_call(check_output=True, returnStdErr=True, realtimeStderrPrefix='cactus_consolidate d({})'.format(referenceEvent), File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 877, in cactus_call raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out)) RuntimeError: Command /usr/bin/time -v cactus_consolidated --sequences 'simCow_chr6 /tmp/17307ece98645d9b9e4d6a ed065d39ae/7f83/3bbc/tmpjusc5rel.tmp simDog_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp simHum an_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d 39ae/7f83/3bbc/tmp4gq0w_l0.tmp simRat_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp' --speciesTr ee '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.1890 8,simDogchr6:0.16303)Anc2:0.032898)Anc0;' --logLevel INFO --alignments /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc /tmpyk1uyzgo.tmp --params /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpi0061dtu.tmp --outputFile /tmp/17307ece986 45d9b9e4d6aed065d39ae/7f83/3bbc/tmpr4s0r53.tmp --outputHalFastaFile /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tm pzeaqu6nq.tmp --outputReferenceFile /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp23htpi2x.tmp --outgroupEvents 's imHuman_chr6 simDog_chr6 simCowchr6' --referenceEvent mr --secondaryAlignments /tmp/17307ece98645d9b9e4d6aed065d39ae/7 f83/3bbc/tmp6cujpzld.tmp --threads 8 exited 132: stdout=, stderr=Params file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f8 3/3bbc/tmpi0061dtu.tmp Output file string : /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpr4s0r53.tmp Output hal fasta file string : /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpzeaqu6nq.tmp Output reference fasta file string : /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp23htpi2x.tmp Sequence files and events: simCow_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp simDog_c hr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp simHuman_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae /7f83/3bbc/tmpvdreph01.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp simRat_chr6 /t mp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp Alignments file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpyk1uyzgo.tmp Secondary alignments file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp6cujpzld.tmp Constraint alignments file: (null) Species tree: ((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,( simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0; Outgroup events: simHuman_chr6 simDog_chr6 simCow_chr6 Reference event: mr Loaded the parameters files, 0 seconds have elapsed Set up the cactus disk, 0 seconds have elapsed Constructed the first flower Going to build the event tree with newick string: ((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6: 0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0; Parsed the tree Constructed the basic event tree Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp to simCow_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp to simDog_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp to simHuman_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp to simMouse_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp to simRat_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp is specified incomplete, the sequences will not be attached Constructed the initial flower with 74 sequences and 10 events with string: (((simHuman_chr6:0.144018,(simMouse _chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)An c0:9.22337e+18)ROOT:9.22337e+18; Established the first Flower in the hierarchy, 0 seconds have elapsed Converted alignment coordinates, 0 seconds have elapsed Stripped the unique IDs, 0 seconds have elapsed Starting annealing round with a minimum chain length of 64 and an alignment trim of 3 There were 53263 blocks in the sequence graph, representing 1528662 total aligned bases Block degree stats: min 1, avg 2.770460, median 3, max 49 Block support stats: min 0.000000, avg 0.851332, median 0.750000, max 2.000000 Starting melting round with a minimum chain length of 2 A melting round is destroying 4670 blocks with an average degree of 3.538116 from chains with length less than

  4. Total aligned bases lost: 16523 Starting melting round with a minimum chain length of 4 A melting round is destroying 5798 blocks with an average degree of 3.009141 from chains with length less than
  5. Total aligned bases lost: 39287 Starting melting round with a minimum chain length of 8 A melting round is destroying 12244 blocks with an average degree of 2.374142 from chains with length less than
    1. Total aligned bases lost: 150276 A melting round is destroying 8293 blocks with an average degree of 2.389123 from chains with length less than

  6. Total aligned bases lost: 209185 Destroying 350 recoverable blocks The blocks covered 7464 columns for a total of 16261 aligned bases Destroying 0 recoverable blocks The blocks covered 0 columns for a total of 0 aligned bases Ran cactus caf, 7 seconds have elapsed Ran extended flowers ready for bar, 7 seconds have elapsed Command terminated by signal 4 Command being timed: "cactus_consolidated --sequences simCow_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae /7f83/3bbc/tmpjusc5rel.tmp simDog_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp simHuman_chr6 /t mp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/ 3bbc/tmp4gq0w_l0.tmp simRat_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp --speciesTree ((simHum an_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDogchr 6:0.16303)Anc2:0.032898)Anc0; --logLevel INFO --alignments /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpyk1uyzgo. tmp --params /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpi0061dtu.tmp --outputFile /tmp/17307ece98645d9b9e4d6aed 065d39ae/7f83/3bbc/tmpr4s0r53.tmp --outputHalFastaFile /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpzeaqu6nq.tmp --outputReferenceFile /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp23htpi2x.tmp --outgroupEvents simHuman_chr6 s imDog_chr6 simCow_chr6 --referenceEvent mr --secondaryAlignments /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp6cu jpzld.tmp --threads 8" User time (seconds): 7.69 System time (seconds): 0.08 Percent of CPU this job got: 96% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.05 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 76456 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 17990 Voluntary context switches: 23 Involuntary context switches: 30 Swaps: 0 File system inputs: 0 File system outputs: 5680 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

    [2022-05-24T08:10:29+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 6a efb3504cc8 <========= [2022-05-24T08:10:30+0000] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'C actusConsolidated' kind-CactusConsolidated/instance-wwlm4ktd v5 with ID kind-CactusConsolidated/instance-wwlm4ktd to 0 [2022-05-24T08:10:30+0000] [MainThread] [W] [toil.leader] Job 'CactusConsolidated' kind-CactusConsolidated/instance-wwl m4ktd v6 is completely failed [2022-05-24T08:10:45+0000] [MainThread] [I] [toil.leader] Finished toil run with 7 failed jobs. [2022-05-24T08:10:45+0000] [MainThread] [I] [toil.leader] Failed jobs at end of the run: 'CactusBlastPhase' kind-Cactus BlastPhase/instance-855oy_oc v2 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDow n/instance-5qk_yih1 v2 'ProgressiveNext' kind-ProgressiveNext/instance-d3xwbd5v v2 'CactusConsolidated' kind-CactusCons olidated/instance-wwlm4ktd v6 'RunCactusPreprocessorThenProgressiveDown2' kind-RunCactusPreprocessorThenProgressiveDown 2/instance-o6imd61e v3 'ProgressiveDown' kind-ProgressiveDown/instance-pvee9c36 v2 'ProgressiveUp' kind-ProgressiveUp/i nstance-sg_3_nnm v2

Workflow Progress 100%|██████████████████████████████████████████████| 193/193 (2 failures) [04:00<00:00, 0.80 jobs/s] Traceback (most recent call last): File "/usr/local/bin/cactus", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/dist-packages/cactus/progressive/cactus_progressive.py", line 406, in main runCactusProgressive(options) File "/usr/local/lib/python3.8/dist-packages/cactus/progressive/cactus_progressive.py", line 456, in runCactusProgressive halID = toil.start(RunCactusPreprocessorThenProgressiveDown(options, project, memory=configWrapper.getDefaultMemory())) File "/usr/local/lib/python3.8/dist-packages/toil/common.py", line 951, in start return self._runMainLoop(rootJobDescription) File "/usr/local/lib/python3.8/dist-packages/toil/common.py", line 1273, in _runMainLoop return Leader(config=self.config, File "/usr/local/lib/python3.8/dist-packages/toil/leader.py", line 289, in run raise FailedJobsException(self.jobStore, failed_jobs, exit_code=self.recommended_fail_exit_code) toil.leader.FailedJobsException: The job store '/data/jobStore2' contains 7 failed jobs: 'CactusBlastPhase' kind-CactusBlastPhase/instance-855oy_oc v2, 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instance-5qk_yih1 v2, 'ProgressiveNext' kind-ProgressiveNext/instance-d3xwbd5v v2, 'CactusConsolidated' kind-CactusConsolidated/instance-wwlm4ktd v6, 'RunCactusPreprocessorThenProgressiveDown2' kind-RunCactusPreprocessorThenProgressiveDown2/instance-o6imd61e v3, 'ProgressiveDown' kind-ProgressiveDown/instance-pvee9c36 v2, 'ProgressiveUp' kind-ProgressiveUp/instance-sg_3_nnm v2 Log from job "'CactusConsolidated' kind-CactusConsolidated/instance-wwlm4ktd v6" follows: =========> [2022-05-24T08:10:20+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-05-24T08:10:20+0000] [MainThread] [I] [toil] Running Toil version 5.6.0-c34146a6437e4407a61e946e968bcce67a0ebbca on host 6aefb3504cc8. [2022-05-24T08:10:20+0000] [MainThread] [I] [toil.worker] Working on job 'CactusConsolidated' kind-CactusConsolidated/instance-wwlm4ktd v4 [2022-05-24T08:10:21+0000] [MainThread] [I] [toil.worker] Loaded body Job('CactusConsolidated' kind-CactusConsolidated/instance-wwlm4ktd v4) from description 'CactusConsolidated' kind-CactusConsolidated/instance-wwlm4ktd v4 [2022-05-24T08:10:21+0000] [MainThread] [I] [toil.statsAndLogging] Alignments file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpyk1uyzgo.tmp [2022-05-24T08:10:21+0000] [MainThread] [W] [root] Deprecated toil method. Please call "logging.getLevelName" directly. [2022-05-24T08:10:21+0000] [MainThread] [I] [cactus.shared.common] Running the command ['cactus_consolidated', '--sequences', 'simCow_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp simDog_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp simHuman_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp simRat_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp', '--speciesTree', '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDogchr6:0.16303)Anc2:0.032898)Anc0;', '--logLevel', 'INFO', '--alignments', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpyk1uyzgo.tmp', '--params', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpi0061dtu.tmp', '--outputFile', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpr4s0r53.tmp', '--outputHalFastaFile', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpzeaqu6nq.tmp', '--outputReferenceFile', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp23htpi2x.tmp', '--outgroupEvents', 'simHuman_chr6 simDog_chr6 simCow_chr6', '--referenceEvent', 'mr', '--secondaryAlignments', '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp6cujpzld.tmp', '--threads', '8'] [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-TrimAndRecurseOnOutgroups/instance-12z4h8at/file-58cca2d70b0e4f8ba23cc046f9f092d1/tmpaxrm51fc.tmp' to path '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-TrimAndRecurseOnOutgroups/instance-f8mm0nlq/file-32eff6c661fa4814a307ada4053c4b62/tmpylkfkf9e.tmp' to path '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-TrimAndRecurseOnOutgroups/instance-gzizxyas/file-3b32eef54dda450095e08e70b26f20cf/tmp6g17xb40.tmp' to path '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CactusBlastPhase/instance-855oy_oc/cleanup/file-8df0131d02c0463ab46a59f110a14b59/tmp65g7lhnh.tmp' to path '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CactusBlastPhase/instance-855oy_oc/cleanup/file-41eff30eab9b4072b4a8481f39d6584e/tmp3ujqus85.tmp' to path '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-mappingQualityRescoring/instance-2vuvseqf/file-ed187aabe6b34f37ab991fcd9acd6caf/tmpbz83ltl5.tmp' to path '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpyk1uyzgo.tmp' [2022-05-24T08:10:29+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-mappingQualityRescoring/instance-2vuvseqf/file-6d55433d5a94429bbca53e5fa9762910/tmpp1i0slrg.tmp' to path '/tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp6cujpzld.tmp' Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/toil/worker.py", line 405, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 943, in _runner super(RoundedJob, self)._runner(*args, jobStore=jobStore, File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2399, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2317, in _run return self.run(fileStore) File "/usr/local/lib/python3.8/dist-packages/cactus/pipeline/cactus_workflow.py", line 408, in run messages = runCactusConsolidated(seqMap=seqMap, File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 197, in runCactusConsolidated masterMessages = cactus_call(check_output=True, returnStdErr=True, realtimeStderrPrefix='cactus_consolidated({})'.format(referenceEvent), File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 877, in cactus_call raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out)) RuntimeError: Command /usr/bin/time -v cactus_consolidated --sequences 'simCow_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp simDog_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp simHuman_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp simRat_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp' --speciesTree '((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDogchr6:0.16303)Anc2:0.032898)Anc0;' --logLevel INFO --alignments /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpyk1uyzgo.tmp --params /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpi0061dtu.tmp --outputFile /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpr4s0r53.tmp --outputHalFastaFile /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpzeaqu6nq.tmp --outputReferenceFile /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp23htpi2x.tmp --outgroupEvents 'simHuman_chr6 simDog_chr6 simCowchr6' --referenceEvent mr --secondaryAlignments /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp6cujpzld.tmp --threads 8 exited 132: stdout=, stderr=Params file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpi0061dtu.tmp Output file string : /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpr4s0r53.tmp Output hal fasta file string : /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpzeaqu6nq.tmp Output reference fasta file string : /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp23htpi2x.tmp Sequence files and events: simCow_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp simDog_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp simHuman_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp simRat_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp Alignments file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpyk1uyzgo.tmp Secondary alignments file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp6cujpzld.tmp Constraint alignments file: (null) Species tree: ((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0; Outgroup events: simHuman_chr6 simDog_chr6 simCow_chr6 Reference event: mr Loaded the parameters files, 0 seconds have elapsed Set up the cactus disk, 0 seconds have elapsed Constructed the first flower Going to build the event tree with newick string: ((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0; Parsed the tree Constructed the basic event tree Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp to simCow_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp to simDog_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp to simHuman_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp to simMouse_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp is specified incomplete, the sequences will not be attached Assigning sequence /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp to simRat_chr6 Processing file: /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp The file /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp is specified incomplete, the sequences will not be attached Constructed the initial flower with 74 sequences and 10 events with string: (((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0:9.22337e+18)ROOT:9.22337e+18; Established the first Flower in the hierarchy, 0 seconds have elapsed Converted alignment coordinates, 0 seconds have elapsed Stripped the unique IDs, 0 seconds have elapsed Starting annealing round with a minimum chain length of 64 and an alignment trim of 3 There were 53263 blocks in the sequence graph, representing 1528662 total aligned bases Block degree stats: min 1, avg 2.770460, median 3, max 49 Block support stats: min 0.000000, avg 0.851332, median 0.750000, max 2.000000 Starting melting round with a minimum chain length of 2 A melting round is destroying 4670 blocks with an average degree of 3.538116 from chains with length less than 2. Total aligned bases lost: 16523 Starting melting round with a minimum chain length of 4 A melting round is destroying 5798 blocks with an average degree of 3.009141 from chains with length less than 4. Total aligned bases lost: 39287 Starting melting round with a minimum chain length of 8 A melting round is destroying 12244 blocks with an average degree of 2.374142 from chains with length less than 8. Total aligned bases lost: 150276 A melting round is destroying 8293 blocks with an average degree of 2.389123 from chains with length less than 64. Total aligned bases lost: 209185 Destroying 350 recoverable blocks The blocks covered 7464 columns for a total of 16261 aligned bases Destroying 0 recoverable blocks The blocks covered 0 columns for a total of 0 aligned bases Ran cactus caf, 7 seconds have elapsed Ran extended flowers ready for bar, 7 seconds have elapsed Command terminated by signal 4 Command being timed: "cactus_consolidated --sequences simCow_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpjusc5rel.tmp simDog_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpuyib6a3x.tmp simHuman_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpvdreph01.tmp simMouse_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp4gq0w_l0.tmp simRat_chr6 /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp5d6cskw2.tmp --speciesTree ((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDogchr6:0.16303)Anc2:0.032898)Anc0; --logLevel INFO --alignments /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpyk1uyzgo.tmp --params /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpi0061dtu.tmp --outputFile /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpr4s0r53.tmp --outputHalFastaFile /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmpzeaqu6nq.tmp --outputReferenceFile /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp23htpi2x.tmp --outgroupEvents simHuman_chr6 simDog_chr6 simCow_chr6 --referenceEvent mr --secondaryAlignments /tmp/17307ece98645d9b9e4d6aed065d39ae/7f83/3bbc/tmp6cujpzld.tmp --threads 8" User time (seconds): 7.69 System time (seconds): 0.08 Percent of CPU this job got: 96% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.05 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 76456 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 17990 Voluntary context switches: 23 Involuntary context switches: 30 Swaps: 0 File system inputs: 0 File system outputs: 5680 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

[2022-05-24T08:10:29+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 6aefb3504cc8

Might I know what went wrong?

Thanks

Im using ubuntu linux OS

glennhickey commented 2 years ago

I think this is the same issue as #574

eskutkaan commented 2 years ago

Getting the same error. I managed to get cactus work for other datasets before with the same machine so I don't think that the error points to the CPU.