cactus_consolidated error

[lpescitelli]

Hello,

I am having issues with running cactus on 3 fish genomes, two of reference quality and one not. I am not sure if this is simply a disk space error (which if so I am unsure where to indicate I want it to increase) or some other error (see below for error). It looks like it tried at least three times to get through this step and failed every time. Any help would be greatly appreciated.

Thank you!

Log from job "'CactusConsolidated' kind-CactusConsolidated/instance-vl35k8j6 v6" follows: =========> [2021-12-14T12:21:43-0500] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2021-12-14T12:21:43-0500] [MainThread] [I] [toil] Running Toil version 5.5.0-b0ff5be051f2fd55352e00450b7848dcf8354a3b on host system76-pc. [2021-12-14T12:21:43-0500] [MainThread] [I] [toil.worker] Working on job 'CactusConsolidated' kind-CactusConsolidated/instance-vl35k8j6 v4 [2021-12-14T12:21:43-0500] [MainThread] [I] [toil.worker] Loaded body Job('CactusConsolidated' kind-CactusConsolidated/instance-vl35k8j6 v4) from description 'CactusConsolidated' kind-CactusConsolidated/instance-vl35k8j6 v4 [2021-12-14T12:21:43-0500] [MainThread] [I] [toil.statsAndLogging] Alignments file: /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpsvvzcvll.tmp [2021-12-14T12:21:43-0500] [MainThread] [W] [root] Deprecated toil method. Please call "logging.getLevelName" directly. [2021-12-14T12:21:43-0500] [MainThread] [I] [cactus.shared.common] Running the command ['cactus_consolidated', '--sequences', 'FL_Pompano /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpak16ww9l.tmp golden_pompano /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmp_jbaa3wk.tmp greater_amberjack /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpneqsvhfc.tmp', '--speciesTree', '(golden_pompano:1.0,greater_amberjack:1.0,FL_Pompano:1.0)Anc0;', '--logLevel', 'INFO', '--alignments', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpsvvzcvll.tmp', '--params', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpjca9th52.tmp', '--outputFile', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmphfnm_0mv.tmp', '--outputHalFastaFile', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpibx8tq6t.tmp', '--outputReferenceFile', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpu8kh8rfw.tmp', '--referenceEvent', 'Anc0', '--secondaryAlignments', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpcc7xy8b9.tmp', '--threads', '32'] [2021-12-14T13:03:15-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2021-12-14T13:03:15-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CactusBlastPhase/instance-29c33t1m/cleanup/file-3ae2935258d64b848e8c3fc70db9656a/tmp4pmx7xtr.tmp' to path '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpak16ww9l.tmp' [2021-12-14T13:03:15-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CactusBlastPhase/instance-29c33t1m/cleanup/file-be3bc3ed684743fe85f9d0f4d02ade2b/tmpyr4ykh18.tmp' to path '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmp_jbaa3wk.tmp' [2021-12-14T13:03:15-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CactusBlastPhase/instance-29c33t1m/cleanup/file-e8cba6b2d0bf4be1a90a99ac6f73bd0f/tmp96d9evi3.tmp' to path '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpneqsvhfc.tmp' [2021-12-14T13:03:15-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-JobFunctionWrappingJob/instance-b75ky_8l/file-75363ef775394a6da30cc47682705f92/tmpcvrwutl5.tmp' to path '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpsvvzcvll.tmp' [2021-12-14T13:03:15-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-JobFunctionWrappingJob/instance-b75ky_8l/file-7f096e282b684cdd9b93081c1be5580c/tmpk3txh68w.tmp' to path '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpcc7xy8b9.tmp' [2021-12-14T13:03:15-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-CactusConsolidated/instance-vl35k8j6/cleanup/file-ed1326ee87bb427ba8e4f29d7f299f91/stream used 1780.10% disk (102.5 GiB [110012878848B] used, 5.8 GiB [6180166866B] requested). Traceback (most recent call last): File "/home/aquaomics/cactus_env/lib/python3.8/site-packages/toil/worker.py", line 402, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/home/aquaomics/cactus_env/lib/python3.8/site-packages/cactus/shared/common.py", line 932, in _runner super(RoundedJob, self)._runner(*args, jobStore=jobStore, File "/home/aquaomics/cactus_env/lib/python3.8/site-packages/toil/job.py", line 2362, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/home/aquaomics/cactus_env/lib/python3.8/site-packages/toil/job.py", line 2283, in _run return self.run(fileStore) File "/home/aquaomics/cactus_env/lib/python3.8/site-packages/cactus/pipeline/cactus_workflow.py", line 408, in run messages = runCactusConsolidated(seqMap=seqMap, File "/home/aquaomics/cactus_env/lib/python3.8/site-packages/cactus/shared/common.py", line 197, in runCactusConsolidated masterMessages = cactus_call(check_output=True, returnStdErr=True, realtimeStderrPrefix='cactus_consolidated({})'.format(referenceEvent), File "/home/aquaomics/cactus_env/lib/python3.8/site-packages/cactus/shared/common.py", line 868, in cactus_call raise RuntimeError("Command {} signaled {}: {}".format(call, signal.Signals(-process.returncode).name, out)) RuntimeError: Command ['cactus_consolidated', '--sequences', 'FL_Pompano /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpak16ww9l.tmp golden_pompano /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmp_jbaa3wk.tmp greater_amberjack /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpneqsvhfc.tmp', '--speciesTree', '(golden_pompano:1.0,greater_amberjack:1.0,FL_Pompano:1.0)Anc0;', '--logLevel', 'INFO', '--alignments', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpsvvzcvll.tmp', '--params', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpjca9th52.tmp', '--outputFile', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmphfnm_0mv.tmp', '--outputHalFastaFile', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpibx8tq6t.tmp', '--outputReferenceFile', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpu8kh8rfw.tmp', '--referenceEvent', 'Anc0', '--secondaryAlignments', '/home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpcc7xy8b9.tmp', '--threads', '32'] signaled SIGKILL: stdout=, stderr=Params file: /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpjca9th52.tmp Output file string : /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmphfnm_0mv.tmp Output hal fasta file string : /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpibx8tq6t.tmp Output reference fasta file string : /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpu8kh8rfw.tmp Sequence files and events: FL_Pompano /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpak16ww9l.tmp golden_pompano /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmp_jbaa3wk.tmp greater_amberjack /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpneqsvhfc.tmp Alignments file: /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpsvvzcvll.tmp Secondary alignments file: /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpcc7xy8b9.tmp Constraint alignments file: (null) Species tree: (golden_pompano:1.0,greater_amberjack:1.0,FL_Pompano:1.0)Anc0; Outgroup events: (null) Reference event: Anc0 Loaded the parameters files, 0 seconds have elapsed Set up the cactus disk, 0 seconds have elapsed Constructed the first flower Going to build the event tree with newick string: (golden_pompano:1.0,greater_amberjack:1.0,FL_Pompano:1.0)Anc0; Parsed the tree Constructed the basic event tree Assigning sequence /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpak16ww9l.tmp to FL_Pompano Processing file: /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpak16ww9l.tmp The file /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpak16ww9l.tmp is specified incomplete, the sequences will not be attached Assigning sequence /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmp_jbaa3wk.tmp to golden_pompano Processing file: /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmp_jbaa3wk.tmp The file /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmp_jbaa3wk.tmp is specified incomplete, the sequences will not be attached Assigning sequence /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpneqsvhfc.tmp to greater_amberjack Processing file: /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpneqsvhfc.tmp The file /home/aquaomics/cactus/3fish_run/e320d40a100a568eabe29e71c9c4efaf/7de3/d060/tmpneqsvhfc.tmp is specified incomplete, the sequences will not be attached Constructed the initial flower with 43109 sequences and 5 events with string: ((golden_pompano:1,greater_amberjack:1,FL_Pompano:1)Anc0:9.22337e+18)ROOT:9.22337e+18; Established the first Flower in the hierarchy, 13 seconds have elapsed Converted alignment coordinates, 1017 seconds have elapsed Stripped the unique IDs, 1017 seconds have elapsed Starting annealing round with a minimum chain length of 64 and an alignment trim of 3 There were 63279373 blocks in the sequence graph, representing 1731233415 total aligned bases Block degree stats: min 1, avg 4.658805, median 3, max 177187 Block support stats: min 0.000000, avg 0.438768, median 0.333333, max 3.500000 Starting melting round with a minimum chain length of 2

[2021-12-14T13:03:15-0500] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host system76-pc

<=========

[glennhickey]

The relevant part his signaled SIGKILL. That means the process was killed by exterior forces. The most common cause is running out of memory.

[lpescitelli]

Thank you for the quick response. I guess we just do not have enough memory on this machine to run it.