search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
503 stars 111 forks source link

Failed when using --restart #816

Open haoyanioz opened 1 year ago

haoyanioz commented 1 year ago

Dear developers, Thanks so much for this very useful tool. For unknown reasons, the cactus command is broken on our server via docker. So I restarted the task using --restart. However, the following errors were generated. I'm not sure why it happened. The log file is shown below. Could you give me some suggestions? Thank you so much.

[2022-10-21T14:13:39+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/8d83/worker_log.txt [2022-10-21T14:13:39+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/worker_log.txt [2022-10-21T14:13:41+0000] [MainThread] [I] [toil-rt] 2022-10-21 14:13:41.251924: Running the command: "bash -c set -eo pipefail && cat /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmp6hrkxtca.tmp | cactus_mirrorAndOrientAlignments INFO | sort -T/tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmp_rv_onpo -k6,6 -k7,7n -k8,8n | uniq | cactus_splitAlignmentOverlaps INFO | cactuscalculateMappingQualities INFO 5 0.0 0.001 /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmpj9l90xrq.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmp5r1vj182.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmpszkz9oi.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmpahm2wlak.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmppb326ugj.tmp" [2022-10-21T14:18:51+0000] [MainThread] [I] [toil-rt] 2022-10-21 14:18:51.922838: Successfully ran: "bash -c 'set -eo pipefail && cat /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmp6hrkxtca.tmp | cactus_mirrorAndOrientAlignments INFO | sort -T/tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmp_rv_onpo -k6,6 -k7,7n -k8,8n | uniq | cactus_splitAlignmentOverlaps INFO | cactuscalculateMappingQualities INFO 5 0.0 0.001 /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmpj9l90xrq.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmp5r1vj182.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmpszkz9oi.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmpahm2wlak.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmppb326ugj.tmp'" in 310.6526 seconds and 8.7 Mi memory [2022-10-21T14:18:51+0000] [MainThread] [I] [toil-rt] 2022-10-21 14:18:51.924067: Running the command: "bash -c set -eo pipefail && cat /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmp5r1vj182.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmpszkz9oi.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmpahm2wlak.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmppb326ugj.tmp" [2022-10-21T14:18:53+0000] [MainThread] [I] [toil-rt] 2022-10-21 14:18:53.286667: Successfully ran: "bash -c 'set -eo pipefail && cat /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmp5r1vj182.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmpszkz9oi.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmpahm2wlak.tmp /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d8c4/d666/tmppb326ugj.tmp'" in 1.35 seconds and 1.6 Mi memory [2022-10-21T14:19:01+0000] [Thread-4 ] [I] [toil.statsAndLogging] Got message from job at time 10-21-2022 14:19:01: Input cigar file has 1733028 lines [2022-10-21T14:19:01+0000] [Thread-4 ] [I] [toil.statsAndLogging] Got message from job at time 10-21-2022 14:19:01: Filtered, non-overlapping primary cigar file has 3696690 lines [2022-10-21T14:19:01+0000] [Thread-4 ] [I] [toil.statsAndLogging] Got message from job at time 10-21-2022 14:19:01: Filtered, non-overlapping secondary cigar file has 11017292 lines [2022-10-21T14:19:01+0000] [Thread-4 ] [W] [toil.statsAndLogging] Got message from job at time 10-21-2022 14:19:01: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-mappingQualityRescoring/instance-g448h9we/cleanup/file-c6711fe9163b4616a5454b7ce890ee65/stream used 161.77% disk (3.2 GiB [3474059264B] used, 2.0 GiB [2147483648B] requested). [2022-10-21T14:19:02+0000] [MainThread] [I] [toil.leader] Issued job 'Job' kind-Job/instance-jimnnjqx v2 with job batch system ID: 3868 and cores: 0, disk: 95.4 Mi, and memory: 488.3 Mi [2022-10-21T14:19:03+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/068c/worker_log.txt [2022-10-21T14:19:04+0000] [MainThread] [I] [toil.leader] Issued job 'EncapsulatedJob' kind-EncapsulatedJob/instance-pq7y0ndw v2 with job batch system ID: 3869 and cores: 1, disk: 2.0 Gi, and memory: 2.0 Gi [2022-10-21T14:19:04+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/a315/worker_log.txt [2022-10-21T14:19:05+0000] [MainThread] [I] [toil.leader] Issued job 'CactusConsolidated' kind-CactusConsolidated/instance-a3ms7ry2 v1 with job batch system ID: 3870 and cores: 160, disk: 12.3 Gi, and memory: 73.8 Gi [2022-10-21T14:19:06+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/worker_log.txt [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] 2022-10-21 14:19:11.414198: Running the command: "cactus_consolidated --sequences Anc02 /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp8j3jqwfv.tmp Anc03 /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpo91ybiui.tmp GalGal /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp4l98a8gq.tmp GarGla /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpa0t8k4gt.tmp --speciesTree (((Anc03:1.0,GarGla:1.0)Anc01:1.0,Anc02:1.0)Anc00:1.0,GalGal:1.0)mr; --logLevel INFO --alignments /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp7907d37d.tmp --params /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpgg54q8el.tmp --outputFile /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpyx_5d4ad.tmp --outputHalFastaFile /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpdppy4nbb.tmp --outputReferenceFile /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp7s4c1k0d.tmp --outgroupEvents Anc02 GalGal --referenceEvent Anc01 --secondaryAlignments /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpcblr6nmc.tmp --threads 160" [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Params file: /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpgg54q8el.tmp [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Output file string : /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpyx_5d4ad.tmp [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Output hal fasta file string : /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpdppy4nbb.tmp [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Output reference fasta file string : /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp7s4c1k0d.tmp [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Sequence files and events: Anc02 /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp8j3jqwfv.tmp Anc03 /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpo91ybiui.tmp GalGal /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp4l98a8gq.tmp GarGla /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpa0t8k4gt.tmp [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Alignments file: /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp7907d37d.tmp [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Secondary alignments file: /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpcblr6nmc.tmp [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Constraint alignments file: (null) [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Species tree: (((Anc03:1.0,GarGla:1.0)Anc01:1.0,Anc02:1.0)Anc00:1.0,GalGal:1.0)mr; [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Outgroup events: Anc02 GalGal [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Reference event: Anc01 [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Loaded the parameters files, 0 seconds have elapsed [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Set up the cactus disk, 0 seconds have elapsed [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Constructed the first flower [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Going to build the event tree with newick string: (((Anc03:1.0,GarGla:1.0)Anc01:1.0,Anc02:1.0)Anc00:1.0,GalGal:1.0)mr; [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Parsed the tree [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Constructed the basic event tree [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Assigning sequence /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp8j3jqwfv.tmp to Anc02 [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Processing file: /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp8j3jqwfv.tmp [2022-10-21T14:19:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): The file /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp8j3jqwfv.tmp is specified incomplete, the sequences will not be attached [2022-10-21T14:19:20+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Assigning sequence /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpo91ybiui.tmp to Anc03 [2022-10-21T14:19:20+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Processing file: /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpo91ybiui.tmp [2022-10-21T14:19:20+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): The file /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpo91ybiui.tmp is specified incomplete, the sequences will not be attached [2022-10-21T14:19:29+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Assigning sequence /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp4l98a8gq.tmp to GalGal [2022-10-21T14:19:29+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Processing file: /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp4l98a8gq.tmp [2022-10-21T14:19:29+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): The file /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp4l98a8gq.tmp is specified incomplete, the sequences will not be attached [2022-10-21T14:19:30+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Assigning sequence /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpa0t8k4gt.tmp to GarGla [2022-10-21T14:19:30+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Processing file: /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpa0t8k4gt.tmp [2022-10-21T14:19:30+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): The file /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpa0t8k4gt.tmp is specified incomplete, the sequences will not be attached [2022-10-21T14:19:40+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Constructed the initial flower with 29804 sequences and 8 events with string: ((((Anc03:1,GarGla:1)Anc01:1,Anc02:1)Anc00:1,GalGal:1)mr:9.22337e+18)ROOT:9.22337e+18; [2022-10-21T14:19:40+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Established the first Flower in the hierarchy, 29 seconds have elapsed [2022-10-21T14:21:04+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Converted alignment coordinates, 113 seconds have elapsed [2022-10-21T14:21:04+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Stripped the unique IDs, 113 seconds have elapsed [2022-10-21T14:21:04+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Starting annealing round with a minimum chain length of 64 and an alignment trim of 3 [2022-10-21T14:25:54+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): There were 22778020 blocks in the sequence graph, representing 3048850403 total aligned bases [2022-10-21T14:25:59+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Block degree stats: min 1, avg 3.220747, median 3, max 482202 [2022-10-21T14:25:59+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Block support stats: min 0.000000, avg 0.816365, median 0.750000, max 4.000000 [2022-10-21T14:26:11+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Starting melting round with a minimum chain length of 2 [2022-10-21T14:33:59+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run [2022-10-21T14:39:10+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): A melting round is destroying 1406952 blocks with an average degree of 5.856043 from chains with length less than 2. Total aligned bases lost: 8239172 [2022-10-21T14:41:05+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Starting melting round with a minimum chain length of 4 [2022-10-21T14:52:04+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): A melting round is destroying 1468761 blocks with an average degree of 3.884034 from chains with length less than 4. Total aligned bases lost: 12793189 [2022-10-21T14:53:50+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Starting melting round with a minimum chain length of 8 [2022-10-21T15:04:08+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): A melting round is destroying 5569055 blocks with an average degree of 2.360124 from chains with length less than 8. Total aligned bases lost: 73744695 [2022-10-21T15:15:36+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): A melting round is destroying 3053712 blocks with an average degree of 2.333889 from chains with length less than 64. Total aligned bases lost: 79905703 [2022-10-21T15:26:28+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Destroying 134335 recoverable blocks [2022-10-21T15:26:28+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): The blocks covered 8328537 columns for a total of 26822638 aligned bases [2022-10-21T15:34:01+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run [2022-10-21T15:34:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Destroying 2633 recoverable blocks [2022-10-21T15:34:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): The blocks covered 144692 columns for a total of 628679 aligned bases [2022-10-21T15:43:18+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Destroying 129 recoverable blocks [2022-10-21T15:43:18+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): The blocks covered 4303 columns for a total of 31903 aligned bases [2022-10-21T15:51:36+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Destroying 47 recoverable blocks [2022-10-21T15:51:36+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): The blocks covered 2365 columns for a total of 4730 aligned bases [2022-10-21T15:59:58+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Destroying 16 recoverable blocks [2022-10-21T15:59:58+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): The blocks covered 684 columns for a total of 1368 aligned bases [2022-10-21T16:01:59+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Pinch graph component with 6825 nodes and 7176 edges is being split up by breaking 442 edges to reduce size to less than 836 max, but found 200 pointless edges [2022-10-21T16:21:42+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Ran cactus caf, 7351 seconds have elapsed [2022-10-21T16:25:12+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Ran extended flowers ready for bar, 7561 seconds have elapsed [2022-10-21T16:34:02+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run [2022-10-21T17:34:04+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run [2022-10-21T18:02:24+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Ran cactus bar (use poa:1), 13393 seconds have elapsed [2022-10-21T18:04:05+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): There are 9 layers in the flowers hierarchy [2022-10-21T18:04:05+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): In the 0 layer there are 1 flowers in the flowers hierarchy [2022-10-21T18:10:32+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): In the 1 layer there are 8558898 flowers in the flowers hierarchy [2022-10-21T18:17:42+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): In the 2 layer there are 5283152 flowers in the flowers hierarchy [2022-10-21T18:34:06+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run [2022-10-21T18:59:50+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): In the 3 layer there are 236334 flowers in the flowers hierarchy [2022-10-21T19:01:47+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): In the 4 layer there are 18128 flowers in the flowers hierarchy [2022-10-21T19:01:56+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): In the 5 layer there are 1355 flowers in the flowers hierarchy [2022-10-21T19:01:57+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): In the 6 layer there are 124 flowers in the flowers hierarchy [2022-10-21T19:01:57+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): In the 7 layer there are 106 flowers in the flowers hierarchy [2022-10-21T19:01:57+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): In the 8 layer there are 10 flowers in the flowers hierarchy [2022-10-21T19:01:57+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Ran cactus make reference, 16966 seconds have elapsed [2022-10-21T19:34:06+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run [2022-10-21T20:28:58+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Ran cactus make reference bottom up coordinates, 22187 seconds have elapsed [2022-10-21T20:31:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Ran cactus make reference top down coordinates, 22332 seconds have elapsed [2022-10-21T20:34:06+0000] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run [2022-10-21T21:15:27+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Ran cactus to hal stage, 24976 seconds have elapsed [2022-10-21T21:15:55+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Dumped sequences for hal file, 25004 seconds have elapsed [2022-10-21T21:16:01+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Dumped reference sequences, 25010 seconds have elapsed [2022-10-21T21:16:10+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Cactus consolidated is done!, 25019 seconds have elapsed [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Command being timed: "cactus_consolidated --sequences Anc02 /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp8j3jqwfv.tmp Anc03 /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpo91ybiui.tmp GalGal /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp4l98a8gq.tmp GarGla /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpa0t8k4gt.tmp --speciesTree (((Anc03:1.0,GarGla:1.0)Anc01:1.0,Anc02:1.0)Anc00:1.0,GalGal:1.0)mr; --logLevel INFO --alignments /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp7907d37d.tmp --params /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpgg54q8el.tmp --outputFile /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpyx_5d4ad.tmp --outputHalFastaFile /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpdppy4nbb.tmp --outputReferenceFile /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp7s4c1k0d.tmp --outgroupEvents Anc02 GalGal --referenceEvent Anc01 --secondaryAlignments /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpcblr6nmc.tmp --threads 160" [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): User time (seconds): 492941.44 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): System time (seconds): 18534.83 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Percent of CPU this job got: 2043% [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Elapsed (wall clock) time (h:mm:ss or m:ss): 6:57:11 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Average shared text size (kbytes): 0 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Average unshared data size (kbytes): 0 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Average stack size (kbytes): 0 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Average total size (kbytes): 0 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Maximum resident set size (kbytes): 82186000 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Average resident set size (kbytes): 0 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Major (requiring I/O) page faults: 28 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Voluntary context switches: 142739538 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Involuntary context switches: 45489935 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Swaps: 0 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): File system inputs: 8064 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Minor (reclaiming a frame) page faults: 2664725396 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): File system outputs: 19010728 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Socket messages sent: 0 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Socket messages received: 0 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Signals delivered: 0 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Page size (bytes): 4096 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc01): Exit status: 0 [2022-10-21T21:16:23+0000] [MainThread] [I] [toil-rt] 2022-10-21 21:16:23.316299: Successfully ran: "cactus_consolidated --sequences 'Anc02 /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp8j3jqwfv.tmp Anc03 /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpo91ybiui.tmp GalGal /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp4l98a8gq.tmp GarGla /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpa0t8k4gt.tmp' --speciesTree '(((Anc03:1.0,GarGla:1.0)Anc01:1.0,Anc02:1.0)Anc00:1.0,GalGal:1.0)mr;' --logLevel INFO --alignments /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp7907d37d.tmp --params /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpgg54q8el.tmp --outputFile /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpyx_5d4ad.tmp --outputHalFastaFile /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpdppy4nbb.tmp --outputReferenceFile /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmp7s4c1k0d.tmp --outgroupEvents 'Anc02 GalGal' --referenceEvent Anc01 --secondaryAlignments /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/d078/43ae/tmpcblr6nmc.tmp --threads 160" in 25031.8862 seconds [2022-10-21T21:16:30+0000] [Thread-4 ] [W] [toil.statsAndLogging] Got message from job at time 10-21-2022 21:16:30: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-CactusConsolidated/instance-a3ms7ry2/cleanup/file-0a96db220a264d35b1a92e297d00caac/stream used 102.80% disk (12.6 GiB [13534269440B] used, 12.3 GiB [13165530129B] requested). [2022-10-21T21:16:34+0000] [MainThread] [I] [toil.leader] Issued job 'CactusBlastPhase' kind-CactusBlastPhase/instance-5h2tez1q v2 with job batch system ID: 3871 and cores: 1, disk: 2.0 Gi, and memory: 3.3 Gi [2022-10-21T21:16:35+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/498b/worker_log.txt [2022-10-21T21:16:36+0000] [MainThread] [I] [toil.leader] Issued job 'ProgressiveUp' kind-ProgressiveUp/instance-39mo2me3 v2 with job batch system ID: 3872 and cores: 1, disk: 2.0 Gi, and memory: 3.3 Gi [2022-10-21T21:16:37+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/b7d4/worker_log.txt [2022-10-21T21:16:39+0000] [MainThread] [I] [toil.leader] Issued job 'ProgressiveOut' kind-ProgressiveOut/instance-0paq0fy9 v1 with job batch system ID: 3873 and cores: 1, disk: 2.0 Gi, and memory: 3.3 Gi [2022-10-21T21:16:40+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/b6a0/worker_log.txt [2022-10-21T21:16:42+0000] [MainThread] [I] [toil.leader] Issued job 'ProgressiveNext' kind-ProgressiveNext/instance-6uqy3saa v3 with job batch system ID: 3874 and cores: 1, disk: 2.0 Gi, and memory: 3.3 Gi [2022-10-21T21:16:43+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/334a/worker_log.txt [2022-10-21T21:16:44+0000] [MainThread] [I] [toil.leader] Issued job 'ProgressiveDown' kind-ProgressiveDown/instance-zb4xdv2j v3 with job batch system ID: 3875 and cores: 1, disk: 2.0 Gi, and memory: 2.0 Gi [2022-10-21T21:16:45+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/0a78/worker_log.txt [2022-10-21T21:16:45+0000] [MainThread] [I] [toil.leader] Issued job 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v1 with job batch system ID: 3876 and cores: 1, disk: 2.0 Gi, and memory: 3.3 Gi [2022-10-21T21:16:46+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/3d99/worker_log.txt [2022-10-21T21:16:48+0000] [Thread-1 ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker ProgressiveNext file:/data/Podoces_cactus11/jobStore kind-ProgressiveNext/instance-4n0_k9zd. [2022-10-21T21:16:48+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v1 Exit reason: None [2022-10-21T21:16:48+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v2 [2022-10-21T21:16:48+0000] [MainThread] [W] [toil.leader] Log from job "kind-ProgressiveNext/instance-4n0_k9zd" follows: =========> [2022-10-21T21:16:46+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-10-21T21:16:46+0000] [MainThread] [I] [toil] Running Toil version 5.6.0-c34146a6437e4407a61e946e968bcce67a0ebbca on host 2487ff50ef89. [2022-10-21T21:16:46+0000] [MainThread] [I] [toil.worker] Working on job 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v1 [2022-10-21T21:16:47+0000] [MainThread] [I] [toil.worker] Loaded body Job('ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v1) from description 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v1 [2022-10-21T21:16:47+0000] [MainThread] [I] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Project has 1 dependencies [2022-10-21T21:16:47+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-10-21T21:16:47+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-2a3f9221965a4a2ba475c873bf70405c/cactus_progressive_config.xml' to path '/tmp/3820f2b81bed5d0d8e40fc5e2ae44989/3d99/06b6/tmprt4d5p_a.tmp' Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/toil/worker.py", line 405, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 943, in _runner super(RoundedJob, self)._runner(args, jobStore=jobStore, File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2399, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2317, in _run return self.run(fileStore) File "/usr/local/lib/python3.8/dist-packages/cactus/progressive/cactus_progressive.py", line 102, in run experiment = ExperimentWrapper(ET.parse(fileStore.readGlobalFile(expID)).getroot()) File "/usr/local/lib/python3.8/dist-packages/toil/fileStores/nonCachingFileStore.py", line 111, in readGlobalFile self.jobStore.read_file(fileStoreID, localFilePath, symlink=symlink) File "/usr/local/lib/python3.8/dist-packages/toil/jobStores/fileJobStore.py", line 444, in read_file self._check_job_store_file_id(file_id) File "/usr/local/lib/python3.8/dist-packages/toil/jobStores/fileJobStore.py", line 779, in _check_job_store_file_id raise NoSuchFileException(jobStoreFileID) toil.jobStores.abstractJobStore.NoSuchFileException: File 'files/no-job/file-3d9c2aa40a1d4a89ae265a9f194141b8/mr_experiment.xml' does not exist. [2022-10-21T21:16:47+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 2487ff50ef89 <========= [2022-10-21T21:16:48+0000] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v2 with ID kind-ProgressiveNext/instance-4n0_k9zd to 1 [2022-10-21T21:16:48+0000] [MainThread] [I] [toil.leader] Issued job 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v3 with job batch system ID: 3877 and cores: 1, disk: 2.0 Gi, and memory: 3.3 Gi [2022-10-21T21:16:49+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3820f2b81bed5d0d8e40fc5e2ae44989/6e5b/worker_log.txt [2022-10-21T21:16:50+0000] [Thread-1 ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker ProgressiveNext file:/data/Podoces_cactus11/jobStore kind-ProgressiveNext/instance-4n0_k9zd. [2022-10-21T21:16:50+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v3 Exit reason: None [2022-10-21T21:16:50+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v5 [2022-10-21T21:16:50+0000] [MainThread] [W] [toil.leader] Log from job "kind-ProgressiveNext/instance-4n0_k9zd" follows: =========> [2022-10-21T21:16:49+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-10-21T21:16:49+0000] [MainThread] [I] [toil] Running Toil version 5.6.0-c34146a6437e4407a61e946e968bcce67a0ebbca on host 2487ff50ef89. [2022-10-21T21:16:49+0000] [MainThread] [I] [toil.worker] Working on job 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v4 [2022-10-21T21:16:49+0000] [MainThread] [I] [toil.worker] Loaded body Job('ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v4) from description 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v4 [2022-10-21T21:16:49+0000] [MainThread] [I] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Project has 1 dependencies [2022-10-21T21:16:49+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-10-21T21:16:49+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-2a3f9221965a4a2ba475c873bf70405c/cactus_progressive_config.xml' to path '/tmp/3820f2b81bed5d0d8e40fc5e2ae44989/6e5b/65d0/tmpq5ll8rsf.tmp' Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/toil/worker.py", line 405, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 943, in _runner super(RoundedJob, self)._runner(args, jobStore=jobStore, File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2399, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2317, in _run return self.run(fileStore) File "/usr/local/lib/python3.8/dist-packages/cactus/progressive/cactus_progressive.py", line 102, in run experiment = ExperimentWrapper(ET.parse(fileStore.readGlobalFile(expID)).getroot()) File "/usr/local/lib/python3.8/dist-packages/toil/fileStores/nonCachingFileStore.py", line 111, in readGlobalFile self.jobStore.read_file(fileStoreID, localFilePath, symlink=symlink) File "/usr/local/lib/python3.8/dist-packages/toil/jobStores/fileJobStore.py", line 444, in read_file self._check_job_store_file_id(file_id) File "/usr/local/lib/python3.8/dist-packages/toil/jobStores/fileJobStore.py", line 779, in _check_job_store_file_id raise NoSuchFileException(jobStoreFileID) toil.jobStores.abstractJobStore.NoSuchFileException: File 'files/no-job/file-3d9c2aa40a1d4a89ae265a9f194141b8/mr_experiment.xml' does not exist. [2022-10-21T21:16:49+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 2487ff50ef89 <========= [2022-10-21T21:16:50+0000] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v5 with ID kind-ProgressiveNext/instance-4n0_k9zd to 0 [2022-10-21T21:16:50+0000] [MainThread] [W] [toil.leader] Job 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v6 is completely failed [2022-10-21T21:17:10+0000] [MainThread] [I] [toil.leader] Finished toil run with 5 failed jobs. [2022-10-21T21:17:10+0000] [MainThread] [I] [toil.leader] Failed jobs at end of the run: 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v6 'ProgressiveDown' kind-ProgressiveDown/instance-_s3c9_hz v2 'RunCactusPreprocessorThenProgressiveDown2' kind-RunCactusPreprocessorThenProgressiveDown2/instance-5_ns60qw v4 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instance-zcvc2ndw v3 'ProgressiveDown' kind-ProgressiveDown/instance-qyrd3osf v2

Workflow Progress 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3878/3878 (2 failures) [16h 43:13<00:00, 0.06 jobs/s] [2022-10-21T21:17:10+0000] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server. [2022-10-21T21:17:11+0000] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread. Traceback (most recent call last): File "/usr/local/bin/cactus", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/dist-packages/cactus/progressive/cactus_progressive.py", line 406, in main runCactusProgressive(options) File "/usr/local/lib/python3.8/dist-packages/cactus/progressive/cactus_progressive.py", line 416, in runCactusProgressive halID = toil.restart() File "/usr/local/lib/python3.8/dist-packages/toil/common.py", line 984, in restart return self._runMainLoop(rootJobDescription) File "/usr/local/lib/python3.8/dist-packages/toil/common.py", line 1273, in _runMainLoop return Leader(config=self.config, File "/usr/local/lib/python3.8/dist-packages/toil/leader.py", line 289, in run raise FailedJobsException(self.jobStore, failed_jobs, exit_code=self.recommended_fail_exit_code) toil.leader.FailedJobsException: The job store '/data/Podoces_cactus11/jobStore' contains 5 failed jobs: 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v6, 'ProgressiveDown' kind-ProgressiveDown/instance-_s3c9_hz v2, 'RunCactusPreprocessorThenProgressiveDown2' kind-RunCactusPreprocessorThenProgressiveDown2/instance-5_ns60qw v4, 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instance-zcvc2ndw v3, 'ProgressiveDown' kind-ProgressiveDown/instance-qyrd3osf v2 Log from job "'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v6" follows: =========> [2022-10-21T21:16:49+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-10-21T21:16:49+0000] [MainThread] [I] [toil] Running Toil version 5.6.0-c34146a6437e4407a61e946e968bcce67a0ebbca on host 2487ff50ef89. [2022-10-21T21:16:49+0000] [MainThread] [I] [toil.worker] Working on job 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v4 [2022-10-21T21:16:49+0000] [MainThread] [I] [toil.worker] Loaded body Job('ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v4) from description 'ProgressiveNext' kind-ProgressiveNext/instance-4n0_k9zd v4 [2022-10-21T21:16:49+0000] [MainThread] [I] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Project has 1 dependencies [2022-10-21T21:16:49+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-10-21T21:16:49+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-2a3f9221965a4a2ba475c873bf70405c/cactus_progressive_config.xml' to path '/tmp/3820f2b81bed5d0d8e40fc5e2ae44989/6e5b/65d0/tmpq5ll8rsf.tmp' Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/toil/worker.py", line 405, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 943, in _runner super(RoundedJob, self)._runner(*args, jobStore=jobStore, File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2399, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2317, in _run return self.run(fileStore) File "/usr/local/lib/python3.8/dist-packages/cactus/progressive/cactus_progressive.py", line 102, in run experiment = ExperimentWrapper(ET.parse(fileStore.readGlobalFile(expID)).getroot()) File "/usr/local/lib/python3.8/dist-packages/toil/fileStores/nonCachingFileStore.py", line 111, in readGlobalFile self.jobStore.read_file(fileStoreID, localFilePath, symlink=symlink) File "/usr/local/lib/python3.8/dist-packages/toil/jobStores/fileJobStore.py", line 444, in read_file self._check_job_store_file_id(file_id) File "/usr/local/lib/python3.8/dist-packages/toil/jobStores/fileJobStore.py", line 779, in _check_job_store_file_id raise NoSuchFileException(jobStoreFileID) toil.jobStores.abstractJobStore.NoSuchFileException: File 'files/no-job/file-3d9c2aa40a1d4a89ae265a9f194141b8/mr_experiment.xml' does not exist. [2022-10-21T21:16:49+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 2487ff50ef89 <=========

Cheers, STAR

glennhickey commented 1 year ago 
toil.jobStores.abstractJobStore.NoSuchFileException: File 'files/no-job/file-3d9c2aa40a1d4a89ae265a9f194141b8/mr_experiment.xml' does not exist.

I've seen something like this in issues before but am not sure the exact cause. It does look like a corrupt job-store, but I don't think that's the whole story.

That said, this is fixed in the latest version of cactus since this version no longer reads and writes xml files from disk. Unfortunately you cannot --restart with a different version of the tool than was first run, so you will need to begin again from scratch.