search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
511 stars 112 forks source link

ModuleNotFoundError: No module named 'pymesos' #832

Closed ld9866 closed 1 year ago

ld9866 commented 1 year ago

Dear developer: Here we are trying "The Minigraph-Cactus Pangenome Pipeline", but we found it is hard to work,could you help me to solve the problem? Thank you!

docker run -v 22.Minigraph_cactus:/data --rm -it quay.io/comparative-genomics-toolkit/cactus:v2.2.4 cactus-minigraph /data/ 5-cow-pg.txt cow-pg.gfa.gz --reference bosTau9 --logFile cow-pg-minigraph.log --batchSystem mesos --defaultPreemptable --nodeType r5.8xlarge:1.5 --nodeStorage 650 --maxNodes 25 --betaInertia 0 --targetTime 1 --mapCores 16 [2022-11-08T12:28:10+0000] [MainThread] [I] [toil.statsAndLogging] Enabling realtime logging in Toil [2022-11-08T12:28:10+0000] [MainThread] [I] [toil.statsAndLogging] Cactus Command: /home/cactus/cactus_env/bin/cactus-minigraph /data/ 5-cow-pg.txt cow-pg.gfa.gz --reference bosTau9 --logFile cow-pg-minigraph.log --batchSystem mesos --defaultPreemptable --nodeType r5.8xlarge:1.5 --nodeStorage 650 --maxNodes 25 --betaInertia 0 --targetTime 1 --mapCores 16 [2022-11-08T12:28:10+0000] [MainThread] [I] [toil.statsAndLogging] Cactus Commit: b699f6bfb4fe88b7876e533dede24e48bbfeb700 Traceback (most recent call last): File "/home/cactus/cactus_env/bin/cactus-minigraph", line 8, in sys.exit(main()) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/refmap/cactus_minigraph.py", line 74, in main with Toil(options) as toil: File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 869, in enter config.setOptions(self.options) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 309, in setOptions set_batchsystem_options(self.batchSystem, set_option) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/batchSystems/options.py", line 35, in set_batchsystem_options batch_system_type = BATCH_SYSTEM_FACTORY_REGISTRY[batch_system]() File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/batchSystems/registry.py", line 48, in mesos_batch_system_factory from toil.batchSystems.mesos.batchSystem import MesosBatchSystem File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/batchSystems/mesos/batchSystem.py", line 32, in from pymesos import MesosSchedulerDriver, Scheduler, decode_data, encode_data ModuleNotFoundError: No module named 'pymesos'

glennhickey commented 1 year ago

That "No module named 'pymesos' error can be fixed with pip install -U toil[all]. BUT the only way I'm familiar with to run on mesos is on aws exactly as described here.

ld9866 commented 1 year ago

Hello! Thank you very much for your timely reply! We currently can't use aws to do our tasks, we have to deploy locally, so how have you been solving this problem? It might be a problem if we keep going like this.

glennhickey commented 1 year ago

I assume you're getting these command lines from my example graphs, which were created on AWS. If you are not set up on AWS, you cannot copy and paste these command lines and run the exactly as is.

You can try removing all the AWS-specific options --batchSystem mesos --defaultPreemptable --nodeType r5.8xlarge:1.5 --nodeStorage 650 --maxNodes 25 --betaInertia 0 --targetTime 1

and setting a local jobstore (something like ./jobstore) for the first argument. This will schedule all jobs on your local machine.

Before doing this, I strongly recommend reading the README and running the quick-start example as well as the yeast example to better understand the commands.

glennhickey commented 1 year ago

I've added a note to the readme to warn about trying the aws-specific options outside aws.

ld9866 commented 1 year ago

Thanks for your patient help, we have tested the sample file according to your suggestions. There are no problems in the test file, and we have successfully obtained the result file. However, we still encountered a lot of problems when using the data of cattle. There was an error in the first step. Could you please help us to check the reason? Code “docker run -v /home/lidong/Data/05.pangenome/22.Minigraph_cactus:/data --rm -it quay.io/comparative-genomics-toolkit/cactus:v2.2.4 cactus-minigraph /data/jobstore 5-cow-pg.txt cow-pg.gfa.gz --reference bosTau9”

[2022-11-09T03:36:01+0000] [MainThread] [I] [toil.statsAndLogging] Enabling realtime logging in Toil [2022-11-09T03:36:01+0000] [MainThread] [I] [toil.statsAndLogging] Cactus Command: /home/cactus/cactus_env/bin/cactus-minigraph /data/jobstore 5-cow-pg.txt cow-pg.gfa.gz --reference bosTau9 [2022-11-09T03:36:01+0000] [MainThread] [I] [toil.statsAndLogging] Cactus Commit: b699f6bfb4fe88b7876e533dede24e48bbfeb700 [2022-11-09T03:36:01+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/bosTau9.fa.gz [2022-11-09T03:36:05+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/GCA_021234555.1_ARS-LIC_NZ_Jersey_genomic.fna.gz [2022-11-09T03:36:05+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/GCA_905123885.1_ROSLIN_BTI_ANK1_genomic.fna.gz [2022-11-09T03:36:05+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/GCA_905123515.1_ROSLIN_BTT_NDA1_genomic.fna.gz [2022-11-09T03:36:05+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/GCA_021347905.1_ARS-LIC_NZ_Holstein-Friesian_1_genomic.fna.gz [2022-11-09T03:36:05+0000] [MainThread] [I] [toil.job] Saving graph of 1 jobs, 1 new [2022-11-09T03:36:05+0000] [MainThread] [I] [toil.job] Processing job 'minigraph_construct_workflow' kind-minigraph_construct_workflow/instance-t_dengnx v0 [2022-11-09T03:36:05+0000] [MainThread] [I] [toil] Running Toil version 5.7.1-b5cae9634820d76cb6c13b2a6312895122017d54 on host 11fec40d2093. [2022-11-09T03:36:05+0000] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging. [2022-11-09T03:36:05+0000] [MainThread] [I] [toil.leader] Issued job 'minigraph_construct_workflow' kind-minigraph_construct_workflow/instance-t_dengnx v1 with job batch system ID: 0 and cores: 1, disk: 2.0 Gi, and memory: 2.0 Gi [2022-11-09T03:36:05+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3da1f568512e56e7bd567aff3626ca70/7ac7/worker_log.txt [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.leader] 0 jobs are running, 0 jobs are issued and waiting to run [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-tah0ywbz v1 with job batch system ID: 1 and cores: 1, disk: 5.8 Gi, and memory: 2.0 Gi [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-njay2bfv v1 with job batch system ID: 2 and cores: 1, disk: 5.5 Gi, and memory: 2.0 Gi [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-druecxyt v1 with job batch system ID: 3 and cores: 1, disk: 5.4 Gi, and memory: 2.0 Gi [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-ca340unt v1 with job batch system ID: 4 and cores: 1, disk: 5.8 Gi, and memory: 2.0 Gi [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v1 with job batch system ID: 5 and cores: 1, disk: 10.7 Gi, and memory: 2.0 Gi [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3da1f568512e56e7bd567aff3626ca70/63eb/worker_log.txt [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3da1f568512e56e7bd567aff3626ca70/6d1f/worker_log.txt [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3da1f568512e56e7bd567aff3626ca70/c544/worker_log.txt [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3da1f568512e56e7bd567aff3626ca70/06eb/worker_log.txt [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3da1f568512e56e7bd567aff3626ca70/aa04/worker_log.txt [2022-11-09T03:36:07+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:07.689722: Running the command: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/6d1f/0c9c/tmpjmekcf4o/bosTau9.fa | cactus_sanitizeFastaHeaders - bosTau9" [2022-11-09T03:36:07+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:07.700967: Running the command: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/63eb/62d4/tmpv5q5adb1/ROSLIN_BTI_ANK1.0.fa | cactus_sanitizeFastaHeaders - ROSLIN_BTI_ANK1.0" [2022-11-09T03:36:08+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:08.042776: Running the command: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/aa04/0dde/tmpn8qh_zhr/ARS-LIC_NZ_Jersey.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Jersey.0" [2022-11-09T03:36:08+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:08.670763: Running the command: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/06eb/585f/tmp7ga1h97m/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0" [2022-11-09T03:36:08+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:08.912183: Running the command: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/c544/0431/tmpvp07dc83/ROSLIN_BTT_NDA1.0.fa | cactus_sanitizeFastaHeaders - ROSLIN_BTT_NDA1.0" [2022-11-09T03:36:32+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:32.730901: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/aa04/0dde/tmpn8qh_zhr/ARS-LIC_NZ_Jersey.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Jersey.0'" in 24.6766 seconds and 155.0 Mi memory [2022-11-09T03:36:33+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:33.555942: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/6d1f/0c9c/tmpjmekcf4o/bosTau9.fa | cactus_sanitizeFastaHeaders - bosTau9'" in 25.8576 seconds and 155.0 Mi memory [2022-11-09T03:36:34+0000] [Thread-1 (daddy)] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker sanitize_fasta_header file:/data/jobstore kind-sanitize_fasta_header/instance-fvsws77w. [2022-11-09T03:36:34+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v1 Exit reason: None [2022-11-09T03:36:34+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v2 [2022-11-09T03:36:34+0000] [MainThread] [W] [toil.leader] Log from job "kind-sanitize_fasta_header/instance-fvsws77w" follows: =========> [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-11-09T03:36:06+0000] [MainThread] [I] [toil] Running Toil version 5.7.1-b5cae9634820d76cb6c13b2a6312895122017d54 on host 11fec40d2093. [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.worker] Working on job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v1 [2022-11-09T03:36:06+0000] [MainThread] [I] [toil.worker] Loaded body Job('sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v1) from description 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v1 [2022-11-09T03:36:08+0000] [MainThread] [I] [cactus.shared.common] Running the command ['bash', '-c', 'set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/06eb/585f/tmp7ga1h97m/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0'] [2022-11-09T03:36:08+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:08.670763: Running the command: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/06eb/585f/tmp7ga1h97m/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0" [2022-11-09T03:36:33+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-11-09T03:36:33+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-0f6f2964cf694867a8b5f45ad1bb8a16/GCA_021347905.1_ARS-LIC_NZ_Holstein-Friesian_1_genomic.fna.gz' to path '/tmp/3da1f568512e56e7bd567aff3626ca70/06eb/585f/tmp7ga1h97m/ARS-LIC_NZ_Holstein-Friesian_1.0.fa' Traceback (most recent call last): File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/worker.py", line 407, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2406, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2324, in _run return self.run(fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2547, in run rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/preprocessor/checkUniqueHeaders.py", line 61, in sanitize_fasta_header cactus_call(parameters=cmd, outfile=out_fa_path) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/shared/common.py", line 808, in cactus_call raise RuntimeError("{}Command {} exited {}: {}".format(sigill_msg, call, process.returncode, out)) RuntimeError: Command /usr/bin/time -v bash -c 'set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/06eb/585f/tmp7ga1h97m/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0' exited 2: stdout=None, stderr= gzip: /tmp/3da1f568512e56e7bd567aff3626ca70/06eb/585f/tmp7ga1h97m/ARS-LIC_NZ_Holstein-Friesian_1.0.fa: decompression OK, trailing garbage ignored Command exited with non-zero status 2 Command being timed: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/06eb/585f/tmp7ga1h97m/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0" User time (seconds): 37.18 System time (seconds): 4.36 Percent of CPU this job got: 164% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:25.25 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 158584 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 162979 Voluntary context switches: 167965 Involuntary context switches: 260 Swaps: 0 File system inputs: 6672 File system outputs: 5258464 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 2

[2022-11-09T03:36:33+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 11fec40d2093

<========= [2022-11-09T03:36:34+0000] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v2 with ID kind-sanitize_fasta_header/instance-fvsws77w to 1 [2022-11-09T03:36:34+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v3 with job batch system ID: 6 and cores: 1, disk: 10.7 Gi, and memory: 2.0 Gi [2022-11-09T03:36:35+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/worker_log.txt [2022-11-09T03:36:35+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:35.280908: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/63eb/62d4/tmpv5q5adb1/ROSLIN_BTI_ANK1.0.fa | cactus_sanitizeFastaHeaders - ROSLIN_BTI_ANK1.0'" in 27.5715 seconds and 155.0 Mi memory [2022-11-09T03:36:35+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:35.294479: Running the command: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0" [2022-11-09T03:36:35+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:35.408769: Successfully ran: "bash -c 'set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/c544/0431/tmpvp07dc83/ROSLIN_BTT_NDA1.0.fa | cactus_sanitizeFastaHeaders - ROSLIN_BTT_NDA1.0'" in 26.4951 seconds and 154.9 Mi memory [2022-11-09T03:37:13+0000] [Thread-1 (daddy)] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker sanitize_fasta_header file:/data/jobstore kind-sanitize_fasta_header/instance-fvsws77w. [2022-11-09T03:37:13+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v3 Exit reason: None [2022-11-09T03:37:13+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v5 [2022-11-09T03:37:13+0000] [MainThread] [W] [toil.leader] Log from job "kind-sanitize_fasta_header/instance-fvsws77w" follows: =========> [2022-11-09T03:36:35+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-11-09T03:36:35+0000] [MainThread] [I] [toil] Running Toil version 5.7.1-b5cae9634820d76cb6c13b2a6312895122017d54 on host 11fec40d2093. [2022-11-09T03:36:35+0000] [MainThread] [I] [toil.worker] Working on job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v4 [2022-11-09T03:36:35+0000] [MainThread] [I] [toil.worker] Loaded body Job('sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v4) from description 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v4 [2022-11-09T03:36:35+0000] [MainThread] [I] [cactus.shared.common] Running the command ['bash', '-c', 'set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0'] [2022-11-09T03:36:35+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:35.294479: Running the command: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0" [2022-11-09T03:37:12+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-11-09T03:37:12+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-0f6f2964cf694867a8b5f45ad1bb8a16/GCA_021347905.1_ARS-LIC_NZ_Holstein-Friesian_1_genomic.fna.gz' to path '/tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa' Traceback (most recent call last): File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/worker.py", line 407, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2406, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2324, in _run return self.run(fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2547, in run rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/preprocessor/checkUniqueHeaders.py", line 61, in sanitize_fasta_header cactus_call(parameters=cmd, outfile=out_fa_path) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/shared/common.py", line 808, in cactus_call raise RuntimeError("{}Command {} exited {}: {}".format(sigill_msg, call, process.returncode, out)) RuntimeError: Command /usr/bin/time -v bash -c 'set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0' exited 2: stdout=None, stderr= gzip: /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa: decompression OK, trailing garbage ignored Command exited with non-zero status 2 Command being timed: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0" User time (seconds): 37.10 System time (seconds): 3.65 Percent of CPU this job got: 108% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:37.50 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 158600 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 144184 Voluntary context switches: 159571 Involuntary context switches: 394 Swaps: 0 File system inputs: 1366616 File system outputs: 5258464 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 2

[2022-11-09T03:37:12+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 11fec40d2093

<========= [2022-11-09T03:37:13+0000] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v5 with ID kind-sanitize_fasta_header/instance-fvsws77w to 0 [2022-11-09T03:37:13+0000] [MainThread] [W] [toil.leader] Job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v6 is completely failed [2022-11-09T03:37:18+0000] [MainThread] [I] [toil.leader] Finished toil run with 2 failed jobs. [2022-11-09T03:37:18+0000] [MainThread] [I] [toil.leader] Failed jobs at end of the run: 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v6 'sanitize_fasta_headers' kind-minigraph_construct_workflow/instance-t_dengnx v2

Workflow Progress 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 (2 failures) [01:12<00:00, 0.10 jobs/s] [2022-11-09T03:37:18+0000] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server. [2022-11-09T03:37:18+0000] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread. Traceback (most recent call last): File "/home/cactus/cactus_env/bin/cactus-minigraph", line 8, in sys.exit(main()) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/refmap/cactus_minigraph.py", line 122, in main gfa_id = toil.start(Job.wrapJobFn(minigraph_construct_workflow, config_node, input_seq_id_map, input_seq_order, options.outputGFA)) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 966, in start return self._runMainLoop(rootJobDescription) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 1391, in _runMainLoop jobCache=self._jobCache).run() File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/leader.py", line 309, in run raise FailedJobsException(self.jobStore, failed_jobs, exit_code=self.recommended_fail_exit_code) toil.leader.FailedJobsException: The job store '/data/jobstore' contains 2 failed jobs: 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v6, 'sanitize_fasta_headers' kind-minigraph_construct_workflow/instance-t_dengnx v2 Log from job "'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v6" follows: =========> [2022-11-09T03:36:35+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-11-09T03:36:35+0000] [MainThread] [I] [toil] Running Toil version 5.7.1-b5cae9634820d76cb6c13b2a6312895122017d54 on host 11fec40d2093. [2022-11-09T03:36:35+0000] [MainThread] [I] [toil.worker] Working on job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v4 [2022-11-09T03:36:35+0000] [MainThread] [I] [toil.worker] Loaded body Job('sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v4) from description 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fvsws77w v4 [2022-11-09T03:36:35+0000] [MainThread] [I] [cactus.shared.common] Running the command ['bash', '-c', 'set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0'] [2022-11-09T03:36:35+0000] [MainThread] [I] [toil-rt] 2022-11-09 03:36:35.294479: Running the command: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0" [2022-11-09T03:37:12+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-11-09T03:37:12+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-0f6f2964cf694867a8b5f45ad1bb8a16/GCA_021347905.1_ARS-LIC_NZ_Holstein-Friesian_1_genomic.fna.gz' to path '/tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa' Traceback (most recent call last): File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/worker.py", line 407, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2406, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2324, in _run return self.run(fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2547, in run rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/preprocessor/checkUniqueHeaders.py", line 61, in sanitize_fasta_header cactus_call(parameters=cmd, outfile=out_fa_path) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/shared/common.py", line 808, in cactus_call raise RuntimeError("{}Command {} exited {}: {}".format(sigill_msg, call, process.returncode, out)) RuntimeError: Command /usr/bin/time -v bash -c 'set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0' exited 2: stdout=None, stderr= gzip: /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa: decompression OK, trailing garbage ignored Command exited with non-zero status 2 Command being timed: "bash -c set -eo pipefail && gzip -dc /tmp/3da1f568512e56e7bd567aff3626ca70/cf0f/ad02/tmp6uzrrpsv/ARS-LIC_NZ_Holstein-Friesian_1.0.fa | cactus_sanitizeFastaHeaders - ARS-LIC_NZ_Holstein-Friesian_1.0" User time (seconds): 37.10 System time (seconds): 3.65 Percent of CPU this job got: 108% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:37.50 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 158600 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 144184 Voluntary context switches: 159571 Involuntary context switches: 394 Swaps: 0 File system inputs: 1366616 File system outputs: 5258464 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 2

[2022-11-09T03:37:12+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 11fec40d2093

<=========

ld9866 commented 1 year ago

This is 5-cow-pg.txt bosTau9 bosTau9.fa.gz ARS-LIC_NZ_Jersey.0 GCA_021234555.1_ARS-LIC_NZ_Jersey_genomic.fna.gz ROSLIN_BTI_ANK1.0 GCA_905123885.1_ROSLIN_BTI_ANK1_genomic.fna.gz ROSLIN_BTT_NDA1.0 GCA_905123515.1_ROSLIN_BTT_NDA1_genomic.fna.gz ARS-LIC_NZ_Holstein-Friesian_1.0 GCA_021347905.1_ARS-LIC_NZ_Holstein-Friesian_1_genomic.fna.gz

glennhickey commented 1 year ago 
decompression OK, trailing garbage ignored
Command exited with non-zero status 2

is an error in gzip, meaning the archive, in this case GCA_021347905.1_ARS-LIC_NZ_Holstein-Friesian_1_genomic.fna.gz is corrupt.

ld9866 commented 1 year ago

Dear Developer, thank you for your prompt reply. We are now using our real data, but we have encountered a new error, which I see has appeared before, but has not been answered. Can you help me? Happy every day

Code: docker run -v /home/lidong/Data/05.pangenome/23.pig.Minigraph_cactus/01.fasta:/data --rm -it quay.io/comparative-genomics-toolkit/cactus:v2.2.4 cactus-graphmap --mapCores 30 --defaultMemory 40G --defaultCores 4 --defaultDisk 40G --maxMemory 200G --maxDisk 400G /data/jobstore /data/fa.list primates.gfa.gz primates.paf --reference Duroc --outputFasta primates.gfa.fa.gz

Result： [2022-11-09T14:16:07+0000] [MainThread] [I] [toil.statsAndLogging] Enabling realtime logging in Toil [2022-11-09T14:16:07+0000] [MainThread] [I] [toil.statsAndLogging] Cactus Command: /home/cactus/cactus_env/bin/cactus-graphmap --mapCores 30 --defaultMemory 40G --defaultCores 4 --defaultDisk 40G --maxMemory 200G --maxDisk 400G /data/jobstore /data/fa.list primates.gfa.gz primates.paf --reference Duroc --outputFasta primates.gfa.fa.gz [2022-11-09T14:16:07+0000] [MainThread] [I] [toil.statsAndLogging] Cactus Commit: b699f6bfb4fe88b7876e533dede24e48bbfeb700 [2022-11-09T14:16:07+0000] [MainThread] [I] [toil.statsAndLogging] Importing primates.gfa.gz [2022-11-09T14:16:11+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/Sscrofa11.1.chr1.fa [2022-11-09T14:16:11+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/Bamei.chr1.fa [2022-11-09T14:16:11+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data/Bama.chr1.fa [2022-11-09T14:16:11+0000] [MainThread] [I] [toil.job] Saving graph of 1 jobs, 1 new [2022-11-09T14:16:11+0000] [MainThread] [I] [toil.job] Processing job 'minigraph_workflow' kind-minigraph_workflow/instance-ecq2c16j v0 [2022-11-09T14:16:11+0000] [MainThread] [I] [toil] Running Toil version 5.7.1-b5cae9634820d76cb6c13b2a6312895122017d54 on host 0c80ecf4199d. [2022-11-09T14:16:11+0000] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging. [2022-11-09T14:16:11+0000] [MainThread] [I] [toil.leader] Issued job 'minigraph_workflow' kind-minigraph_workflow/instance-ecq2c16j v1 with job batch system ID: 0 and cores: 4, disk: 37.3 Gi, and memory: 37.3 Gi [2022-11-09T14:16:11+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/5895/worker_log.txt [2022-11-09T14:16:12+0000] [MainThread] [I] [toil.leader] 0 jobs are running, 0 jobs are issued and waiting to run [2022-11-09T14:16:12+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_headers' kind-sanitize_fasta_headers/instance-r77qsddz v1 with job batch system ID: 1 and cores: 4, disk: 37.3 Gi, and memory: 37.3 Gi [2022-11-09T14:16:12+0000] [MainThread] [I] [toil.leader] Issued job 'make_minigraph_fasta' kind-make_minigraph_fasta/instance-5tir57w9 v1 with job batch system ID: 2 and cores: 30, disk: 37.3 Gi, and memory: 37.3 Gi [2022-11-09T14:16:12+0000] [MainThread] [I] [toil.leader] Issued job 'unzip_gz' kind-unzip_gz/instance-kd5k3m62 v1 with job batch system ID: 3 and cores: 4, disk: 386.0 Mi, and memory: 37.3 Gi [2022-11-09T14:16:12+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/c841/worker_log.txt [2022-11-09T14:16:12+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/b176/worker_log.txt [2022-11-09T14:16:12+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/12d7/worker_log.txt [2022-11-09T14:16:12+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:12.859101: Running the command: "gzip -fd primates.gfa.gz" [2022-11-09T14:16:12+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:12.906034: Running the command: "bash -c set -eo pipefail && gfatools gfa2fa /tmp/664b8033018b57ed8eeaf89049092aea/c841/452d/tmpltwig_ca/mg.gfa | sed -e 's/^>(.)/>id=MINIGRAPH|\1/g' | bgzip --threads 30" [2022-11-09T14:16:12+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-bsis6zzu v1 with job batch system ID: 4 and cores: 4, disk: 1.8 Gi, and memory: 37.3 Gi [2022-11-09T14:16:12+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-hocpio60 v1 with job batch system ID: 5 and cores: 4, disk: 1.8 Gi, and memory: 37.3 Gi [2022-11-09T14:16:12+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-c6a46nrv v1 with job batch system ID: 6 and cores: 4, disk: 1.9 Gi, and memory: 37.3 Gi [2022-11-09T14:16:13+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/730b/worker_log.txt [2022-11-09T14:16:13+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/64cb/worker_log.txt [2022-11-09T14:16:13+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/a758/worker_log.txt [2022-11-09T14:16:13+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:13.694261: Running the command: "cactus_sanitizeFastaHeaders /tmp/664b8033018b57ed8eeaf89049092aea/730b/e0fe/tmpxkukse59/Bama.fa Bama" [2022-11-09T14:16:13+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:13.701183: Running the command: "cactus_sanitizeFastaHeaders /tmp/664b8033018b57ed8eeaf89049092aea/a758/bd17/tmpg8d2ww4r/Duroc.fa Duroc" [2022-11-09T14:16:13+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:13.710978: Running the command: "cactus_sanitizeFastaHeaders /tmp/664b8033018b57ed8eeaf89049092aea/64cb/4e8e/tmppae_rwmx/Bamei.fa Bamei" [2022-11-09T14:16:14+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:14.731255: Successfully ran: "gzip -fd primates.gfa.gz" in 1.8708 seconds and 1.8 Mi memory [2022-11-09T14:16:15+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:15.500958: Successfully ran: "bash -c 'set -eo pipefail && gfatools gfa2fa /tmp/664b8033018b57ed8eeaf89049092aea/c841/452d/tmpltwig_ca/mg.gfa | sed -e '"'"'s/^>(.)/>id=MINIGRAPH|\1/g'"'"' | bgzip --threads 30'" in 2.5937 seconds and 274.3 Mi memory [2022-11-09T14:16:16+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:16.479085: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/664b8033018b57ed8eeaf89049092aea/a758/bd17/tmpg8d2ww4r/Duroc.fa Duroc" in 2.7767 seconds and 461.1 Mi memory [2022-11-09T14:16:16+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:16.688652: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/664b8033018b57ed8eeaf89049092aea/730b/e0fe/tmpxkukse59/Bama.fa Bama" in 2.9932 seconds and 461.2 Mi memory [2022-11-09T14:16:16+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:16.811034: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/664b8033018b57ed8eeaf89049092aea/64cb/4e8e/tmppae_rwmx/Bamei.fa Bamei" in 3.0989 seconds and 461.1 Mi memory [2022-11-09T14:16:17+0000] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_headers' kind-sanitize_fasta_headers/instance-r77qsddz v2 with job batch system ID: 7 and cores: 4, disk: 37.3 Gi, and memory: 37.3 Gi [2022-11-09T14:16:17+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/9942/worker_log.txt [2022-11-09T14:16:17+0000] [MainThread] [I] [toil.leader] Issued job 'minigraph_map_all' kind-minigraph_map_all/instance-f3cd9j8i v1 with job batch system ID: 8 and cores: 4, disk: 37.3 Gi, and memory: 37.3 Gi [2022-11-09T14:16:17+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/9fb6/worker_log.txt [2022-11-09T14:16:18+0000] [MainThread] [I] [toil.leader] Issued job 'Job' kind-Job/instance-ydxvy8p5 v1 with job batch system ID: 9 and cores: 4, disk: 37.3 Gi, and memory: 37.3 Gi [2022-11-09T14:16:18+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/2c36/worker_log.txt [2022-11-09T14:16:18+0000] [MainThread] [I] [toil.leader] Issued job 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v1 with job batch system ID: 10 and cores: 30, disk: 1.6 Gi, and memory: 37.3 Gi [2022-11-09T14:16:18+0000] [MainThread] [I] [toil.leader] Issued job 'minigraph_map_one' kind-minigraph_map_one/instance-tvmob5c3 v1 with job batch system ID: 11 and cores: 30, disk: 1.6 Gi, and memory: 37.3 Gi [2022-11-09T14:16:18+0000] [MainThread] [I] [toil.leader] Issued job 'minigraph_map_one' kind-minigraph_map_one/instance-7m1tr7eo v1 with job batch system ID: 12 and cores: 30, disk: 1.6 Gi, and memory: 37.3 Gi [2022-11-09T14:16:19+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/35c4/worker_log.txt [2022-11-09T14:16:19+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/a8af/worker_log.txt [2022-11-09T14:16:19+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:19.324849: Running the command: "bash -c set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.gaf -c -xasm -t 30" [2022-11-09T14:16:19+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:19.327514: Running the command: "bash -c set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/Bama.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/Bama.gaf -c -xasm -t 30" [2022-11-09T14:19:59+0000] [Thread-1 (daddy)] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker minigraph_map_one file:/data/jobstore kind-minigraph_map_one/instance-dmpuez2f. [2022-11-09T14:19:59+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v1 Exit reason: None [2022-11-09T14:19:59+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v2 [2022-11-09T14:19:59+0000] [MainThread] [W] [toil.leader] Log from job "kind-minigraph_map_one/instance-dmpuez2f" follows: =========> [2022-11-09T14:16:19+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-11-09T14:16:19+0000] [MainThread] [I] [toil] Running Toil version 5.7.1-b5cae9634820d76cb6c13b2a6312895122017d54 on host 0c80ecf4199d. [2022-11-09T14:16:19+0000] [MainThread] [I] [toil.worker] Working on job 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v1 [2022-11-09T14:16:19+0000] [MainThread] [I] [toil.worker] Loaded body Job('minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v1) from description 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v1 [2022-11-09T14:16:19+0000] [MainThread] [I] [cactus.shared.common] Running the command ['bash', '-c', 'set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.gaf -c -xasm -t 30'] [2022-11-09T14:16:19+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:16:19.324849: Running the command: "bash -c set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.gaf -c -xasm -t 30" [2022-11-09T14:19:58+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-11-09T14:19:58+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-unzip_gz/instance-kd5k3m62/file-2a1f70233aad4961bbeb9c3a0b788c13/primates.gfa' to path '/tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/mg.gfa' [2022-11-09T14:19:58+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-sanitize_fasta_header/instance-hocpio60/file-ce97952cf5a14a3da4a1c2b52dd2ad57/Duroc.sanitized.fa' to path '/tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.fa' Traceback (most recent call last): File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/worker.py", line 407, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2406, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2324, in _run return self.run(fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2547, in run rValue = userFunction(*((self,) + tuple(self._args)), *self._kwargs) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/refmap/cactus_graphmap.py", line 350, in minigraph_map_one cactus_call(parameters=cmd) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/shared/common.py", line 808, in cactus_call raise RuntimeError("{}Command {} exited {}: {}".format(sigill_msg, call, process.returncode, out)) RuntimeError: Command /usr/bin/time -v bash -c 'set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.gaf -c -xasm -t 30' exited 134: stdout=None, stderr=[M::main::0.4641.00] loaded the graph from "/tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/mg.gfa" [M::mg_index::7.9171.58] indexed the graph [M::mg_opt_update::8.4481.55] occ_max1=100; lc_max_occ=2 minigraph: galign.c:133: mg_gchain_cigar: Assertion `l == gc->qe - gc->qs && gc->p->aplen == gc->pe - gc->ps' failed. Command terminated by signal 6 Command being timed: "bash -c set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/35c4/4b7d/tmpdsu3dzqo/Duroc.gaf -c -xasm -t 30" User time (seconds): 216.14 System time (seconds): 7.93 Percent of CPU this job got: 102% Elapsed (wall clock) time (h:mm:ss or m:ss): 3:39.65 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 10095832 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 4 Minor (reclaiming a frame) page faults: 7446624 Voluntary context switches: 143186 Involuntary context switches: 1144 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

[2022-11-09T14:19:58+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 0c80ecf4199d

<========= [2022-11-09T14:19:59+0000] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v2 with ID kind-minigraph_map_one/instance-dmpuez2f to 1 [2022-11-09T14:19:59+0000] [MainThread] [W] [toil.job] We have increased the disk of the failed job 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v2 to the default of 40000000000 bytes [2022-11-09T14:19:59+0000] [MainThread] [I] [toil.leader] Issued job 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v3 with job batch system ID: 13 and cores: 30, disk: 37.3 Gi, and memory: 37.3 Gi [2022-11-09T14:19:59+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/8375/worker_log.txt [2022-11-09T14:19:59+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:19:59.661220: Running the command: "bash -c set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/Bamei.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/Bamei.gaf -c -xasm -t 30"

[2022-11-09T14:23:02+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:23:02.509831: Successfully ran: "bash -c 'set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/Bama.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/Bama.gaf -c -xasm -t 30'" in 403.1813 seconds and 9.5 Gi memory [2022-11-09T14:23:02+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:23:02.510266: Running the command: "bash -c set -eo pipefail && gaf2unstable /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/Bama.gaf -g /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/mg.gfa -o /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/mg.gfa.node_lengths.tsv | gaffilter - -r 5.0 -m 0.25 -q 5 -b 250000 -o 0 -i 0.5" [2022-11-09T14:23:06+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:23:06.470017: Successfully ran: "bash -c 'set -eo pipefail && gaf2unstable /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/Bama.gaf -g /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/mg.gfa -o /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/mg.gfa.node_lengths.tsv | gaffilter - -r 5.0 -m 0.25 -q 5 -b 250000 -o 0 -i 0.5'" in 3.9589 seconds and 277.4 Mi memory [2022-11-09T14:23:06+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:23:06.470452: Running the command: "bash -c set -eo pipefail && gaf2paf /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/Bama.gaf.unstable -l /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/mg.gfa.node_lengths.tsv | awk 'BEGIN{OFS=" "} {$6="id=MINIGRAPH|"$6; print}'" [2022-11-09T14:23:06+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:23:06.980717: Successfully ran: "bash -c 'set -eo pipefail && gaf2paf /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/Bama.gaf.unstable -l /tmp/664b8033018b57ed8eeaf89049092aea/a8af/90a9/tmp0z_s0if0/mg.gfa.node_lengths.tsv | awk '"'"'BEGIN{OFS=" "} {$6="id=MINIGRAPH|"$6; print}'"'"''" in 0.5096 seconds and 62.8 Mi memory [2022-11-09T14:23:07+0000] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/664b8033018b57ed8eeaf89049092aea/cb81/worker_log.txt [2022-11-09T14:23:07+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:23:07.689190: Running the command: "bash -c set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.gaf -c -xasm -t 30" [2022-11-09T14:24:10+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:24:10.562677: Successfully ran: "bash -c 'set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/Bamei.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/Bamei.gaf -c -xasm -t 30'" in 250.9004 seconds and 9.6 Gi memory [2022-11-09T14:24:10+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:24:10.563041: Running the command: "bash -c set -eo pipefail && gaf2unstable /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/Bamei.gaf -g /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/mg.gfa -o /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/mg.gfa.node_lengths.tsv | gaffilter - -r 5.0 -m 0.25 -q 5 -b 250000 -o 0 -i 0.5" [2022-11-09T14:24:14+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:24:14.493801: Successfully ran: "bash -c 'set -eo pipefail && gaf2unstable /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/Bamei.gaf -g /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/mg.gfa -o /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/mg.gfa.node_lengths.tsv | gaffilter - -r 5.0 -m 0.25 -q 5 -b 250000 -o 0 -i 0.5'" in 3.93 seconds and 277.4 Mi memory [2022-11-09T14:24:14+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:24:14.494159: Running the command: "bash -c set -eo pipefail && gaf2paf /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/Bamei.gaf.unstable -l /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/mg.gfa.node_lengths.tsv | awk 'BEGIN{OFS=" "} {$6="id=MINIGRAPH|"$6; print}'" [2022-11-09T14:24:15+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:24:15.011177: Successfully ran: "bash -c 'set -eo pipefail && gaf2paf /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/Bamei.gaf.unstable -l /tmp/664b8033018b57ed8eeaf89049092aea/8375/6e1a/tmpv10yzxkm/mg.gfa.node_lengths.tsv | awk '"'"'BEGIN{OFS=" "} {$6="id=MINIGRAPH|"$6; print}'"'"''" in 0.5165 seconds and 62.4 Mi memory [2022-11-09T14:26:47+0000] [Thread-1 (daddy)] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker minigraph_map_one file:/data/jobstore kind-minigraph_map_one/instance-dmpuez2f. [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v3 Exit reason: None [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v5 [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.leader] Log from job "kind-minigraph_map_one/instance-dmpuez2f" follows: =========> [2022-11-09T14:23:07+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-11-09T14:23:07+0000] [MainThread] [I] [toil] Running Toil version 5.7.1-b5cae9634820d76cb6c13b2a6312895122017d54 on host 0c80ecf4199d. [2022-11-09T14:23:07+0000] [MainThread] [I] [toil.worker] Working on job 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v4 [2022-11-09T14:23:07+0000] [MainThread] [I] [toil.worker] Loaded body Job('minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v4) from description 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v4 [2022-11-09T14:23:07+0000] [MainThread] [I] [cactus.shared.common] Running the command ['bash', '-c', 'set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.gaf -c -xasm -t 30'] [2022-11-09T14:23:07+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:23:07.689190: Running the command: "bash -c set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.gaf -c -xasm -t 30" [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-unzip_gz/instance-kd5k3m62/file-2a1f70233aad4961bbeb9c3a0b788c13/primates.gfa' to path '/tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa' [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-sanitize_fasta_header/instance-hocpio60/file-ce97952cf5a14a3da4a1c2b52dd2ad57/Duroc.sanitized.fa' to path '/tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa' Traceback (most recent call last): File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/worker.py", line 407, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2406, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2324, in _run return self.run(fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2547, in run rValue = userFunction(*((self,) + tuple(self._args)), *self._kwargs) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/refmap/cactus_graphmap.py", line 350, in minigraph_map_one cactus_call(parameters=cmd) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/shared/common.py", line 808, in cactus_call raise RuntimeError("{}Command {} exited {}: {}".format(sigill_msg, call, process.returncode, out)) RuntimeError: Command /usr/bin/time -v bash -c 'set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.gaf -c -xasm -t 30' exited 134: stdout=None, stderr=[M::main::0.4511.00] loaded the graph from "/tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa" [M::mg_index::7.6231.80] indexed the graph [M::mg_opt_update::8.2081.75] occ_max1=100; lc_max_occ=2 minigraph: galign.c:133: mg_gchain_cigar: Assertion `l == gc->qe - gc->qs && gc->p->aplen == gc->pe - gc->ps' failed. Command terminated by signal 6 Command being timed: "bash -c set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.gaf -c -xasm -t 30" User time (seconds): 215.46 System time (seconds): 10.27 Percent of CPU this job got: 102% Elapsed (wall clock) time (h:mm:ss or m:ss): 3:39.77 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 10095328 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 7948306 Voluntary context switches: 146203 Involuntary context switches: 1060 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

[2022-11-09T14:26:47+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 0c80ecf4199d

<========= [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v5 with ID kind-minigraph_map_one/instance-dmpuez2f to 0 [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.leader] Job 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v6 is completely failed [2022-11-09T14:26:57+0000] [MainThread] [I] [toil.leader] Finished toil run with 4 failed jobs. [2022-11-09T14:26:57+0000] [MainThread] [I] [toil.leader] Failed jobs at end of the run: 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v6 'Job' kind-minigraph_workflow/instance-ecq2c16j v2 'Job' kind-Job/instance-ydxvy8p5 v2 'minigraph_map_all' kind-minigraph_map_all/instance-f3cd9j8i v3

Workflow Progress 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 (2 failures) [10:44<00:00, 0.02 jobs/s] [2022-11-09T14:26:57+0000] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server. [2022-11-09T14:26:57+0000] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread. Traceback (most recent call last): File "/home/cactus/cactus_env/bin/cactus-graphmap", line 8, in sys.exit(main()) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/refmap/cactus_graphmap.py", line 93, in main graph_map(options) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/refmap/cactus_graphmap.py", line 184, in graph_map paf_id, gfa_fa_id, gaf_id, unfiltered_paf_id, paf_filter_log = toil.start(Job.wrapJobFn( File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 966, in start return self._runMainLoop(rootJobDescription) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 1391, in _runMainLoop jobCache=self._jobCache).run() File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/leader.py", line 309, in run raise FailedJobsException(self.jobStore, failed_jobs, exit_code=self.recommended_fail_exit_code) toil.leader.FailedJobsException: The job store '/data/jobstore' contains 4 failed jobs: 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v6, 'Job' kind-minigraph_workflow/instance-ecq2c16j v2, 'Job' kind-Job/instance-ydxvy8p5 v2, 'minigraph_map_all' kind-minigraph_map_all/instance-f3cd9j8i v3 Log from job "'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v6" follows: =========> [2022-11-09T14:23:07+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2022-11-09T14:23:07+0000] [MainThread] [I] [toil] Running Toil version 5.7.1-b5cae9634820d76cb6c13b2a6312895122017d54 on host 0c80ecf4199d. [2022-11-09T14:23:07+0000] [MainThread] [I] [toil.worker] Working on job 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v4 [2022-11-09T14:23:07+0000] [MainThread] [I] [toil.worker] Loaded body Job('minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v4) from description 'minigraph_map_one' kind-minigraph_map_one/instance-dmpuez2f v4 [2022-11-09T14:23:07+0000] [MainThread] [I] [cactus.shared.common] Running the command ['bash', '-c', 'set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.gaf -c -xasm -t 30'] [2022-11-09T14:23:07+0000] [MainThread] [I] [toil-rt] 2022-11-09 14:23:07.689190: Running the command: "bash -c set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.gaf -c -xasm -t 30" [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-unzip_gz/instance-kd5k3m62/file-2a1f70233aad4961bbeb9c3a0b788c13/primates.gfa' to path '/tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa' [2022-11-09T14:26:47+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-sanitize_fasta_header/instance-hocpio60/file-ce97952cf5a14a3da4a1c2b52dd2ad57/Duroc.sanitized.fa' to path '/tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa' Traceback (most recent call last): File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/worker.py", line 407, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2406, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2324, in _run return self.run(fileStore) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 2547, in run rValue = userFunction(*((self,) + tuple(self._args)), *self._kwargs) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/refmap/cactus_graphmap.py", line 350, in minigraph_map_one cactus_call(parameters=cmd) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/shared/common.py", line 808, in cactus_call raise RuntimeError("{}Command {} exited {}: {}".format(sigill_msg, call, process.returncode, out)) RuntimeError: Command /usr/bin/time -v bash -c 'set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.gaf -c -xasm -t 30' exited 134: stdout=None, stderr=[M::main::0.4511.00] loaded the graph from "/tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa" [M::mg_index::7.6231.80] indexed the graph [M::mg_opt_update::8.2081.75] occ_max1=100; lc_max_occ=2 minigraph: galign.c:133: mg_gchain_cigar: Assertion `l == gc->qe - gc->qs && gc->p->aplen == gc->pe - gc->ps' failed. Command terminated by signal 6 Command being timed: "bash -c set -eo pipefail && minigraph /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/mg.gfa /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.fa -o /tmp/664b8033018b57ed8eeaf89049092aea/cb81/77f8/tmp23olelmn/Duroc.gaf -c -xasm -t 30" User time (seconds): 215.46 System time (seconds): 10.27 Percent of CPU this job got: 102% Elapsed (wall clock) time (h:mm:ss or m:ss): 3:39.77 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 10095328 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 7948306 Voluntary context switches: 146203 Involuntary context switches: 1060 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0

[2022-11-09T14:26:47+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host 0c80ecf4199d

<=========