gwct
commented
2 years ago Hello, I am trying to run the GPU version of cactus (2.0.5) and am experiencing an issue similar to the one posted above. I am running cactus through a singularity image built from the provided docker container. The exact command I'm running is:
singularity exec --nv --cleanenv --bind $TMPDIR:/tmp cactus_v2.0.5-gpu.sif cactus-align $TMPDIR/Anc11-align output/genomes.txt output/Anc11.cigar output/Anc11.hal --root Anc11 --realTimeLogging --logInfo --retryCount 0 --configFile output/config-prepared.xml --maxCores 64 --defaultDisk 450GI can post the whole log if requested, but the parts that seem to be the problem are as follows:
First, there seems to be a problem with the blossom5 command:
[2022-05-14T01:57:22+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): ERROR: Something went wrong with the command: blossom5 -e /tmp/stTmpVKqOiG -w /tmp/stTmpyFaw53 > /dev/null 2>&1
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Command exited with non-zero status 1
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Command being timed: "cactus_consolidated --sequences Anc12 /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpi3o63clc.tmp Anc13 /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmp4bgqemb_.tmp Anc16 /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmp0r1bnj8z.tmp Anc18 /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpbwu_uqt_.tmp Cinsculpta /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpdzywelu1.tmp --speciesTree ((Anc16:2.0,(Anc18:1.0,Cinsculpta:1.0)Anc11:1.0)Anc05:1.0,(Anc12:1.0,Anc13:1.0)Anc06:1.0)Anc02; --logLevel INFO --alignments /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpcwoohi_l.tmp --params /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmp42m0szxc.tmp --outputFile /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpg94c09js.tmp --outputHalFastaFile /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmplu3jtjng.tmp --outputReferenceFile /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpr20rh5fg.tmp --outgroupEvents Anc16 Anc12 Anc13 --referenceEvent Anc11 --secondaryAlignments /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmp4wzi1vyu.tmp --threads 64"
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): User time (seconds): 200653.72
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): System time (seconds): 272597.58
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Percent of CPU this job got: 1057%
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Elapsed (wall clock) time (h:mm:ss or m:ss): 12:26:02
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Average shared text size (kbytes): 0
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Average unshared data size (kbytes): 0
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Average stack size (kbytes): 0
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Average total size (kbytes): 0
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Maximum resident set size (kbytes): 313456004
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Average resident set size (kbytes): 0
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Major (requiring I/O) page faults: 20
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Minor (reclaiming a frame) page faults: 20334878572
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Voluntary context switches: 297740498
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Involuntary context switches: 3068208
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Swaps: 0
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): File system inputs: 27366
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): File system outputs: 77828232
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Socket messages sent: 0
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Socket messages received: 0
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Signals delivered: 0
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Page size (bytes): 4096
[2022-05-14T01:57:53+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc11): Exit status: 1
[2022-05-14T01:57:54+0000] [Thread-1 ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker CactusConsolidated file:/tmp/Anc11-align kind-CactusConsolidated/instance-y811o9v3.
[2022-05-14T01:57:54+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CactusConsolidated' kind-CactusConsolidated/instance-y811o9v3 v1
Exit reason: None
[2022-05-14T01:57:54+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'CactusConsolidated' kind-CactusConsolidated/instance-y811o9v3 v2
[2022-05-14T01:57:54+0000] [MainThread] [W] [toil.leader] Log from job "kind-CactusConsolidated/instance-y811o9v3" follows:Then, within that log report I see this traceback relating to disk usage:
[2022-05-14T01:57:53+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-CactusConsolidated/instance-y811o9v3/cleanup/file-b679f7de38d3480ba17fd4f0817df921/stream used 495.90% disk (59.6 GiB [63981568000B] used, 12.0 GiB [12902153700B] requested).
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/toil/worker.py", line 405, in workerScript
job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 943, in _runner
super(RoundedJob, self)._runner(*args, jobStore=jobStore,
File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2399, in _runner
returnValues = self._run(jobGraph=None, fileStore=fileStore)
File "/usr/local/lib/python3.8/dist-packages/toil/job.py", line 2317, in _run
return self.run(fileStore)
File "/usr/local/lib/python3.8/dist-packages/cactus/pipeline/cactus_workflow.py", line 408, in run
messages = runCactusConsolidated(seqMap=seqMap,
File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 197, in runCactusConsolidated
masterMessages = cactus_call(check_output=True, returnStdErr=True, realtimeStderrPrefix='cactus_consolidated({})'.format(referenceEvent),
File "/usr/local/lib/python3.8/dist-packages/cactus/shared/common.py", line 877, in cactus_call
raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
RuntimeError: Command /usr/bin/time -v cactus_consolidated --sequences 'Anc12 /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpi3o63clc.tmp Anc13 /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmp4bgqemb_.tmp Anc16 /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmp0r1bnj8z.tmp Anc18 /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpbwu_uqt_.tmp Cinsculpta /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpdzywelu1.tmp' --speciesTree '((Anc16:2.0,(Anc18:1.0,Cinsculpta:1.0)Anc11:1.0)Anc05:1.0,(Anc12:1.0,Anc13:1.0)Anc06:1.0)Anc02;' --logLevel INFO --alignments /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpcwoohi_l.tmp --params /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmp42m0szxc.tmp --outputFile /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpg94c09js.tmp --outputHalFastaFile /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmplu3jtjng.tmp --outputReferenceFile /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmpr20rh5fg.tmp --outgroupEvents 'Anc16 Anc12 Anc13' --referenceEvent Anc11 --secondaryAlignments /tmp/562c17956a6055aeb425d2a7017554cc/bf2a/7552/tmp4wzi1vyu.tmp --threads 64 exited 1: stdout=Attaching the sequence to the cactus root 2152536, header ML683766.1 with length 126691712 and 35447807 total bases aligned and 0 bases aligned to other chromosome threadsI have tried increasing both the memory requested on the node I'm running and the --defaultDisk option, but nothing seems to affect this error: it always uses 59.6GiB and only requests 12.0Gib. I also do not see an option outDirMin in any of the help menus, so I'm not sure where I should set that.
This same command has completed successfully on other nodes in the same input dataset.
Any help would be appreciated!
Hi there
I have used cactus with step-by-step method, and successfully completed most steps. But there was a problem in the last few steps that I had never seen before.
cactus_consolidated(Anc02): ERROR: Something went wrong with the command: blossom5 -e /tmp/stTmpXw6xCY -w /tmp/stTmpphCBjK > /dev/null 2>&1
I've rerun this step many times, but it's always the same when things go wrong. The previous steps were successful and took about a month, so I didn't want to start again. Any useful help would be appreciated.
This is all the steps for
cactus-preparegeneration: `## Preprocessor cactus-preprocess jobstore/0 seqFile steps-output/fanchu.txt --inputNames pig whale xilu --realTimeLogging --logInfo --retryCount0 cactus-preprocess jobstore/1 seqFile steps-output/fanchu.txt --inputNames giraffe prohor blamun --realTimeLogging --logInfo --retryCount 0 cactus-preprocess jobstore/2 seqFile steps-output/fanchu.txt --inputNames muskdeer mhl milu --realTimeLogging --logInfo --retryCount 0 cactus-preprocess jobstore/3 seqFile steps-output/fanchu.txt --inputNames cattle goat sheep --realTimeLogging --logInfo --retryCount 0## Alignment
### Round 0 cactus-blast jobstore/4 steps-output/fanchu.txt steps-output/Anc04.cigar --root Anc04 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/5 steps-output/fanchu.txt steps-output/Anc04.cigar steps-output/Anc04.hal --root Anc04 --realTimeLogging --logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/Anc04.hal Anc04 --hdf5InMemory > steps-output/Anc04.fa
cactus-blast jobstore/6 steps-output/fanchu.txt steps-output/Anc08.cigar --root Anc08 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/7 steps-output/fanchu.txt steps-output/Anc08.cigar steps-output/Anc08.hal --root Anc08 --realTimeLogging --logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/Anc08.hal Anc08 --hdf5InMemory > steps-output/Anc08.fa
cactus-blast jobstore/8 steps-output/fanchu.txt steps-output/Anc10.cigar --root Anc10 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/9 steps-output/fanchu.txt steps-output/Anc10.cigar steps-output/Anc10.hal --root Anc10 --realTimeLogging --logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/Anc10.hal Anc10 --hdf5InMemory > steps-output/Anc10.fa
### Round 1 cactus-blast jobstore/10 steps-output/fanchu.txt steps-output/Anc06.cigar --root Anc06 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/11 steps-output/fanchu.txt steps-output/Anc06.cigar steps-output/Anc06.hal --root Anc06 --realTimeLogging --logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/Anc06.hal Anc06 --hdf5InMemory > steps-output/Anc06.fa
cactus-blast jobstore/12 steps-output/fanchu.txt steps-output/Anc09.cigar --root Anc09 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/13 steps-output/fanchu.txt steps-output/Anc09.cigar steps-output/Anc09.hal --root Anc09 --realTimeLogging --logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/Anc09.hal Anc09 --hdf5InMemory > steps-output/Anc09.fa
### Round 2 cactus-blast jobstore/14 steps-output/fanchu.txt steps-output/Anc07.cigar --root Anc07 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/15 steps-output/fanchu.txt steps-output/Anc07.cigar steps-output/Anc07.hal --root Anc07 --realTimeLogging --logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/Anc07.hal Anc07 --hdf5InMemory > steps-output/Anc07.fa
### Round 3 cactus-blast jobstore/16 steps-output/fanchu.txt steps-output/Anc05.cigar --root Anc05 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/17 steps-output/fanchu.txt steps-output/Anc05.cigar steps-output/Anc05.hal --root Anc05 --realTimeLogging --logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/Anc05.hal Anc05 --hdf5InMemory > steps-output/Anc05.fa
### Round 4 cactus-blast jobstore/18 steps-output/fanchu.txt steps-output/Anc03.cigar --root Anc03 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/19 steps-output/fanchu.txt steps-output/Anc03.cigar steps-output/Anc03.hal --root Anc03 --realTimeLogging --logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/Anc03.hal Anc03 --hdf5InMemory > steps-output/Anc03.fa
### Round 5 cactus-blast jobstore/20 steps-output/fanchu.txt steps-output/Anc02.cigar --root Anc02 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/21 steps-output/fanchu.txt steps-output/Anc02.cigar steps-output/Anc02.hal --root Anc02 --realTimeLogging --logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/Anc02.hal Anc02 --hdf5InMemory > steps-output/Anc02.fa
### Round 6 cactus-blast jobstore/22 steps-output/fanchu.txt steps-output/Anc01.cigar --root Anc01 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/23 steps-output/fanchu.txt steps-output/Anc01.cigar steps-output/Anc01.hal --root Anc01 --realTimeLogging --logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/Anc01.hal Anc01 --hdf5InMemory > steps-output/Anc01.fa
### Round 7 cactus-blast jobstore/24 steps-output/fanchu.txt steps-output/Anc00.cigar --root Anc00 --realTimeLogging --logInfo --retryCount 0 cactus-align jobstore/25 steps-output/fanchu.txt steps-output/Anc00.cigar steps-output/fanchu.hal --root Anc00 --realTimeLogging--logInfo --retryCount 0 --maxCores 2 hal2fasta steps-output/fanchu.hal Anc00 --hdf5InMemory > steps-output/Anc00.fa
## HAL merging halAppendSubtree steps-output/fanchu.hal steps-output/Anc01.hal Anc01 Anc01 --merge --hdf5InMemory halAppendSubtree steps-output/fanchu.hal steps-output/Anc02.hal Anc02 Anc02 --merge --hdf5InMemory halAppendSubtree steps-output/fanchu.hal steps-output/Anc03.hal Anc03 Anc03 --merge --hdf5InMemory halAppendSubtree steps-output/fanchu.hal steps-output/Anc05.hal Anc05 Anc05 --merge --hdf5InMemory halAppendSubtree steps-output/fanchu.hal steps-output/Anc07.hal Anc07 Anc07 --merge --hdf5InMemory halAppendSubtree steps-output/fanchu.hal steps-output/Anc06.hal Anc06 Anc06 --merge --hdf5InMemory halAppendSubtree steps-output/fanchu.hal steps-output/Anc09.hal Anc09 Anc09 --merge --hdf5InMemory halAppendSubtree steps-output/fanchu.hal steps-output/Anc04.hal Anc04 Anc04 --merge --hdf5InMemory halAppendSubtree steps-output/fanchu.hal steps-output/Anc10.hal Anc10 Anc10 --merge --hdf5InMemory halAppendSubtree steps-output/fanchu.hal steps-output/Anc08.hal Anc08 Anc08 --merge --hdf5InMemory`
And there is the commad with the error:
cactus-align jobstore/212 steps-output/fanchu.txt steps-output/Anc02.cigar steps-output/Anc02.hal --root Anc02 --realTimeLogging--logInfo --retryCount 0 --maxMemory 200G --maxCores 120 --deadlockWait 600 --binariesMode local --maxLocalJobs 120 --defaultDisk 200G