search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
499 stars 109 forks source link

error in cactus-align-batch step #1099

Open Wong718 opened 1 year ago

Wong718 commented 1 year ago

I have test a small set of 2 maize genomes (B73 and Mo17) according to the pangenome pipeline However, when I try to align each chromosome with Cactus, It comes to an error. The command I have made is: cactus-align-batch minigraph_tmp chromfile.txt minigraph_output/chrom-alignments --alignCores 16 --alignOptions "--pangenome --reference B73 --outVG " And the error information is： [2023-07-13T18:02:29+0800] [MainThread] [I] [toil.statsAndLogging] Enabling realtime logging in Toil [2023-07-13T18:02:29+0800] [MainThread] [W] [toil.statsAndLogging] ** WARNING **** [2023-07-13T18:02:29+0800] [MainThread] [W] [toil.statsAndLogging] cactus-align-batch is deprecated and will be eventually disabled. [2023-07-13T18:02:29+0800] [MainThread] [W] [toil.statsAndLogging] Please use cactus-align --batch instead [2023-07-13T18:02:29+0800] [MainThread] [W] [toil.statsAndLogging] *** [2023-07-13T18:02:29+0800] [MainThread] [I] [toil.statsAndLogging] Cactus Command: /data21/wongzj/Maize_embryo/Minigraph-Cactus/Minigraph-Cactus-env/bin/cactus-align-batch minigraph_tmp chromfile.txt minigraph_output/chrom-alignments --alignCores 16 --alignOptions --pangenome --reference B73 --outVG [2023-07-13T18:02:29+0800] [MainThread] [I] [toil.statsAndLogging] Cactus Commit: ce2bd972d997340845c78f69671106178ce6491e [2023-07-13T18:02:34+0800] [MainThread] [I] [toil.job] Saving graph of 1 jobs, 1 non-service, 1 new [2023-07-13T18:02:34+0800] [MainThread] [I] [toil.job] Processing job 'align_toil_batch' kind-align_toil_batch/instance-2xts7va8 v0 [2023-07-13T18:02:34+0800] [MainThread] [I] [toil] Running Toil version 5.10.0-21422a3440f8a6d5e9d2f1c9695c4fbc57fa5372 on host master. [2023-07-13T18:02:34+0800] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging. [2023-07-13T18:02:34+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil_batch' kind-align_toil_batch/instance-2xts7va8 v1 with job batch system ID: 0 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False [2023-07-13T18:02:34+0800] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/fa04/worker_log.txt [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] 0 jobs are running, 0 jobs are issued and waiting to run [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil' kind-align_toil/instance-h6ghbfvl v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 16, accelerators: [], preemptible: False [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil' kind-align_toil/instance-ryb3j4or v1 with job batch system ID: 2 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 16, accelerators: [], preemptible: False [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil' kind-align_toil/instance-djgzyzu3 v1 with job batch system ID: 3 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 16, accelerators: [], preemptible: False [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil' kind-align_toil/instance-1qes50rn v1 with job batch system ID: 4 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 16, accelerators: [], preemptible: False [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil' kind-align_toil/instance-2zlymkro v1 with job batch system ID: 5 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 16, accelerators: [], preemptible: False [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil' kind-align_toil/instance-ou5d2cg7 v1 with job batch system ID: 6 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 16, accelerators: [], preemptible: False [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil' kind-align_toil/instance-aiqxgj6v v1 with job batch system ID: 7 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 16, accelerators: [], preemptible: False [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil' kind-align_toil/instance-chka17j7 v1 with job batch system ID: 8 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 16, accelerators: [], preemptible: False [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil' kind-align_toil/instance-_ebjmdkc v1 with job batch system ID: 9 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 16, accelerators: [], preemptible: False [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.leader] Issued job 'align_toil' kind-align_toil/instance-zug61nfg v1 with job batch system ID: 10 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 16, accelerators: [], preemptible: False [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/976e/worker_log.txt [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/0d27/worker_log.txt [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/f195/worker_log.txt [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/e3d7/worker_log.txt [2023-07-13T18:02:35+0800] [MainThread] [I] [toil-rt] 2023-07-13 18:02:35.885627: Running the command: "docker run --interactive --net=host --log-driver=none -u 1009:1001 -v /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/976e/6107/tmpoqvns3s6:/data --entrypoint /opt/cactus/wrapper.sh --name ce017fa7-8a5c-4f0b-95c5-84e038449ab2 --rm quay.io/comparative-genomics-toolkit/cactus:ce2bd972d997340845c78f69671106178ce6491e cactus-align js B73.chr8_seq_file.txt B73.chr8.paf B73.chr8.hal --logFile B73.chr8.hal.log --configFile config.xml --pangenome --reference B73 --outVG --consCores 16 --maxCores 16" [2023-07-13T18:02:35+0800] [MainThread] [I] [toil-rt] 2023-07-13 18:02:35.897037: Running the command: "docker run --interactive --net=host --log-driver=none -u 1009:1001 -v /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/f195/768a/tmp5affw5nc:/data --entrypoint /opt/cactus/wrapper.sh --name ae511006-0695-4899-9d3b-910e2aea72c9 --rm quay.io/comparative-genomics-toolkit/cactus:ce2bd972d997340845c78f69671106178ce6491e cactus-align js B73.chr5_seq_file.txt B73.chr5.paf B73.chr5.hal --logFile B73.chr5.hal.log --configFile config.xml --pangenome --reference B73 --outVG --consCores 16 --maxCores 16" [2023-07-13T18:02:35+0800] [MainThread] [I] [toil-rt] 2023-07-13 18:02:35.902592: Running the command: "docker run --interactive --net=host --log-driver=none -u 1009:1001 -v /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/0d27/06d1/tmpfs__3php:/data --entrypoint /opt/cactus/wrapper.sh --name 2aa8c8d2-a9e4-4089-9ba8-f303c4d3cd9e --rm quay.io/comparative-genomics-toolkit/cactus:ce2bd972d997340845c78f69671106178ce6491e cactus-align js B73.chr2_seq_file.txt B73.chr2.paf B73.chr2.hal --logFile B73.chr2.hal.log --configFile config.xml --pangenome --reference B73 --outVG --consCores 16 --maxCores 16" [2023-07-13T18:02:35+0800] [MainThread] [I] [toil-rt] 2023-07-13 18:02:35.920124: Running the command: "docker run --interactive --net=host --log-driver=none -u 1009:1001 -v /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/e3d7/7acb/tmpc5ld1q_a:/data --entrypoint /opt/cactus/wrapper.sh --name b40869bc-5ed9-435c-9a4a-b6cd453dc307 --rm quay.io/comparative-genomics-toolkit/cactus:ce2bd972d997340845c78f69671106178ce6491e cactus-align js B73.chr6_seq_file.txt B73.chr6.paf B73.chr6.hal --logFile B73.chr6.hal.log --configFile config.xml --pangenome --reference B73 --outVG --consCores 16 --maxCores 16" [2023-07-13T18:02:43+0800] [Thread-1 ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker align_toil file:/data21/wongzj/Maize_embryo/Minigraph-Cactus/minigraph_tmp kind-align_toil/instance-h6ghbfvl. [2023-07-13T18:02:43+0800] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'align_toil' kind-align_toil/instance-h6ghbfvl v1 Exit reason: None [2023-07-13T18:02:43+0800] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'align_toil' kind-align_toil/instance-h6ghbfvl v2 [2023-07-13T18:02:43+0800] [MainThread] [W] [toil.leader] Log from job "kind-align_toil/instance-h6ghbfvl" follows: =========> [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2023-07-13T18:02:35+0800] [MainThread] [I] [toil] Running Toil version 5.10.0-21422a3440f8a6d5e9d2f1c9695c4fbc57fa5372 on host master. [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.worker] Working on job 'align_toil' kind-align_toil/instance-h6ghbfvl v1 [2023-07-13T18:02:35+0800] [MainThread] [I] [toil.worker] Loaded body Job('align_toil' kind-align_toil/instance-h6ghbfvl v1) from description 'align_toil' kind-align_toil/instance-h6ghbfvl v1 [2023-07-13T18:02:35+0800] [MainThread] [I] [cactus.shared.common] Work dirs: {'/tmp/71c8fa6d2b5f55cbb823ed8720363fc6/976e/6107/tmpoqvns3s6'} [2023-07-13T18:02:35+0800] [MainThread] [I] [cactus.shared.common] Docker work dir: /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/976e/6107/tmpoqvns3s6 [2023-07-13T18:02:35+0800] [MainThread] [I] [cactus.shared.common] Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '1009:1001', '-v', '/tmp/71c8fa6d2b5f55cbb823ed8720363fc6/976e/6107/tmpoqvns3s6:/data', '--entrypoint', '/opt/cactus/wrapper.sh', '--name', 'ce017fa7-8a5c-4f0b-95c5-84e038449ab2', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:ce2bd972d997340845c78f69671106178ce6491e', 'cactus-align', 'js', 'B73.chr8_seq_file.txt', 'B73.chr8.paf', 'B73.chr8.hal', '--logFile', 'B73.chr8.hal.log', '--configFile', 'config.xml', '--pangenome', '--reference', 'B73', '--outVG', '--consCores', '16', '--maxCores', '16'] [2023-07-13T18:02:35+0800] [MainThread] [I] [toil-rt] 2023-07-13 18:02:35.885627: Running the command: "docker run --interactive --net=host --log-driver=none -u 1009:1001 -v /tmp/71c8fa6d2b5f55cbb823ed8720363fc6/976e/6107/tmpoqvns3s6:/data --entrypoint /opt/cactus/wrapper.sh --name ce017fa7-8a5c-4f0b-95c5-84e038449ab2 --rm quay.io/comparative-genomics-toolkit/cactus:ce2bd972d997340845c78f69671106178ce6491e cactus-align js B73.chr8_seq_file.txt B73.chr8.paf B73.chr8.hal --logFile B73.chr8.hal.log --configFile config.xml --pangenome --reference B73 --outVG --consCores 16 --maxCores 16" [2023-07-13T18:02:43+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files: [2023-07-13T18:02:43+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-8a9350fc1ced47948c169cfacf9b24a4/cactus_progressive_config.xml' to path '/tmp/71c8fa6d2b5f55cbb823ed8720363fc6/976e/6107/tmpoqvns3s6/config.xml' [2023-07-13T18:02:43+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-7d0dca53f2d541a1a0f500e148adc832/B73.chr8.seqfile' to path '/tmp/71c8fa6d2b5f55cbb823ed8720363fc6/976e/6107/tmpoqvns3s6/B73.chr8_seq_file.txt' [2023-07-13T18:02:43+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-6e0ffe0e2bb740bc8af1515a1d4891c7/B73.chr8.paf' to path '/tmp/71c8fa6d2b5f55cbb823ed8720363fc6/976e/6107/tmpoqvns3s6/B73.chr8.paf' Traceback (most recent call last): File "/data21/wongzj/Maize_embryo/Minigraph-Cactus/Minigraph-Cactus-env/lib/python3.7/site-packages/toil/worker.py", line 403, in workerScript job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/data21/wongzj/Maize_embryo/Minigraph-Cactus/Minigraph-Cactus-env/lib/python3.7/site-packages/toil/job.py", line 2768, in _runner returnValues = self._run(jobGraph=None, fileStore=fileStore) File "/data21/wongzj/Maize_embryo/Minigraph-Cactus/Minigraph-Cactus-env/lib/python3.7/site-packages/toil/job.py", line 2685, in _run return self.run(fileStore) File "/data21/wongzj/Maize_embryo/Minigraph-Cactus/Minigraph-Cactus-env/lib/python3.7/site-packages/toil/job.py", line 2913, in run rValue = userFunction(*((self,) + tuple(self._args)), self._kwargs) File "/data21/wongzj/Maize_embryo/Minigraph-Cactus/Minigraph-Cactus-env/lib/python3.7/site-packages/cactus/setup/cactus_align.py", line 619, in align_toil cactus_call(parameters=cmd) File "/data21/wongzj/Maize_embryo/Minigraph-Cactus/Minigraph-Cactus-env/lib/python3.7/site-packages/cactus/shared/common.py", line 876, in cactus_call raise RuntimeError("{}Command {} exited {}: {}".format(sigill_msg, call, process.returncode, out)) RuntimeError: Command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '1009:1001', '-v', '/tmp/71c8fa6d2b5f55cbb823ed8720363fc6/976e/6107/tmpoqvns3s6:/data', '--entrypoint', '/opt/cactus/wrapper.sh', '--name', 'ce017fa7-8a5c-4f0b-95c5-84e038449ab2', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:ce2bd972d997340845c78f69671106178ce6491e', 'cactus-align', 'js', 'B73.chr8_seq_file.txt', 'B73.chr8.paf', 'B73.chr8.hal', '--logFile', 'B73.chr8.hal.log', '--configFile', 'config.xml', '--pangenome', '--reference', 'B73', '--outVG', '--consCores', '16', '--maxCores', '16'] exited 1: stderr=Running command catchsegv 'cactus-align' 'js' 'B73.chr8_seq_file.txt' 'B73.chr8.paf' 'B73.chr8.hal' '--logFile' 'B73.chr8.hal.log' '--configFile' 'config.xml' '--pangenome' '--reference' 'B73' '--outVG' '--consCores' '16' '--maxCores' '16' [2023-07-13T10:02:37+0000] [MainThread] [I] [toil.statsAndLogging] Enabling realtime logging in Toil [2023-07-13T10:02:37+0000] [MainThread] [I] [toil.statsAndLogging] Cactus Command: /home/cactus/cactus_env/bin/cactus-align js B73.chr8_seq_file.txt B73.chr8.paf B73.chr8.hal --logFile B73.chr8.hal.log --configFile config.xml --pangenome --reference B73 --outVG --consCores 16 --maxCores 16 [2023-07-13T10:02:37+0000] [MainThread] [I] [toil.statsAndLogging] Cactus Commit: ce2bd972d997340845c78f69671106178ce6491e [2023-07-13T10:02:42+0000] [MainThread] [I] [toil.statsAndLogging] Importing file:///data21/wongzj/Maize_embryo/Minigraph-Cactus/minigraph_output/chroms/B73.chr8/fasta/Mo17.0_B73.chr8.fa Traceback (most recent call last): File "/home/cactus/cactus_env/bin/cactus-align", line 8, in sys.exit(main()) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/setup/cactus_align.py", line 176, in main align_jobs = make_batch_align_jobs(options, toil) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/setup/cactus_align.py", line 242, in make_batch_align_jobs result_dict[None] = make_align_job(options, toil) File "/home/cactus/cactus_env/lib/python3.10/site-packages/cactus/setup/cactus_align.py", line 337, in make_align_job input_seq_id_map[genome] = toil.importFile(seq) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/lib/compatibility.py", line 12, in call return func(*args, **kwargs) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 1263, in importFile return self.import_file(srcUrl, sharedFileName, symlink) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 1288, in import_file src_uri = self.normalize_uri(src_uri, check_existence=True) File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/common.py", line 1321, in normalize_uri raise FileNotFoundError( FileNotFoundError: Could not find local file "/data21/wongzj/Maize_embryo/Minigraph-Cactus/minigraph_output/chroms/B73.chr8/fasta/Mo17.0_B73.chr8.fa" when importing "/data21/wongzj/Maize_embryo/Minigraph-Cactus/minigraph_output/chroms/B73.chr8/fasta/Mo17.0_B73.chr8.fa". Make sure paths are relative to "/data" or use absolute paths. If this is not a local file, please include the scheme (s3:/, gs:/, ftp://, etc.).

    [2023-07-13T18:02:43+0800] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host master

Please figure out what is going wrong, thank you

glennhickey commented 1 year ago

Sorry, I no longer support cactus-align-batch (see deprecation notice at the top of your log). Please try using cactus-align --batch (see examples in README) instead.