search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
529 stars 111 forks source link

'CheckUniqueHeaders' failed using singularity #379

Closed zhenzhenyang-psu closed 3 years ago

zhenzhenyang-psu commented 3 years ago

Hello Glenn, I was testing a pairwise alignment with cactus using singularity: singularity exec $containerImage cactus jobstore pairwise/2genomes_tree_gzlink.txt pairwise/2genomes.hal --binariesMode local

After running for 6 mins, I got the following error, do you think this may be singularity specific? File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/checkUniqueHeaders.py", line 9, in checkUniqueHeaders for header, seq in fastaRead(inputFile): File "/usr/local/lib/python3.6/dist-packages/sonLib/bioio.py", line 805, in fastaRead line = fileHandle.readline() File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128) [2020-12-15T14:22:13+0800] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host z055 <========= [2020-12-15T14:22:13+0800] [MainThread] [W] [toil.jobGraph] Due to failure we are reducing the remaining retry count of job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-9qt7jufl with ID kind-CheckUniqueHeaders/instance-9qt7jufl to 1 [2020-12-15T14:22:13+0800] [MainThread] [W] [toil.jobGraph] We have increased the default memory of the failed job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-9qt7jufl to 2147483648 bytes [2020-12-15T14:22:13+0800] [MainThread] [I] [toil.leader] Issued job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-9qt7jufl with job batch system ID: 12 and cores: 1, disk: 2.1 G, and memory: 2.0 G [2020-12-15T14:22:13+0800] [Thread-1 ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker CheckUniqueHeaders file:/scratch/pawsey0263/yangzhzh/projects/1_cactus/2_test_pairwise/jobstore kind-CheckUniqueHeaders/instance-1104cz3i. [2020-12-15T14:22:13+0800] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i [2020-12-15T14:22:13+0800] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i [2020-12-15T14:22:13+0800] [MainThread] [W] [toil.leader] Log from job kind-CheckUniqueHeaders/instance-1104cz3i follows: =========> [2020-12-15T14:22:12+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2020-12-15T14:22:12+0800] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host z055. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/toil/worker.py", line 368, in workerScript job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.6/dist-packages/cactus/shared/common.py", line 1369, in _runner super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1424, in _runner returnValues = self._run(jobGraph, fileStore) File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1361, in _run return self.run(fileStore) File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/cactus_preprocessor.py", line 72, in run checkUniqueHeaders(inFile, checkAssemblyHub=self.prepOptions.checkAssemblyHub) File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/checkUniqueHeaders.py", line 9, in checkUniqueHeaders for header, seq in fastaRead(inputFile): File "/usr/local/lib/python3.6/dist-packages/sonLib/bioio.py", line 805, in fastaRead line = fileHandle.readline() File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128) [2020-12-15T14:22:13+0800] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host z055 <========= [2020-12-15T14:22:13+0800] [MainThread] [W] [toil.jobGraph] Due to failure we are reducing the remaining retry count of job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i with ID kind-CheckUniqueHeaders/instance-1104cz3i to 1 [2020-12-15T14:22:13+0800] [MainThread] [W] [toil.jobGraph] We have increased the default memory of the failed job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i to 2147483648 bytes [2020-12-15T14:22:13+0800] [MainThread] [I] [toil.leader] Issued job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i with job batch system ID: 13 and cores: 1, disk: 2.2 G, and memory: 2.0 G [2020-12-15T14:22:14+0800] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/node-1ab20341-7e95-4a4c-807d-0eac3e998446-8df7ff2e-2519-421e-9437-1c7649b8f518/tmp4znl1hab/worker_log.txt [2020-12-15T14:22:14+0800] [MainThread] [I] [toil.worker] Redirecting logging to /tmp/node-1ab20341-7e95-4a4c-807d-0eac3e998446-8df7ff2e-2519-421e-9437-1c7649b8f518/tmpsygmf2ux/worker_log.txt [2020-12-15T14:22:16+0800] [Thread-1 ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker CheckUniqueHeaders file:/scratch/pawsey0263/yangzhzh/projects/1_cactus/2_test_pairwise/jobstore kind-CheckUniqueHeaders/instance-9qt7jufl. [2020-12-15T14:22:16+0800] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-9qt7jufl [2020-12-15T14:22:16+0800] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-9qt7jufl [2020-12-15T14:22:16+0800] [MainThread] [W] [toil.leader] Log from job kind-CheckUniqueHeaders/instance-9qt7jufl follows: =========> [2020-12-15T14:22:14+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2020-12-15T14:22:14+0800] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host z055. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/toil/worker.py", line 368, in workerScript job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.6/dist-packages/cactus/shared/common.py", line 1369, in _runner super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1424, in _runner returnValues = self._run(jobGraph, fileStore) File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1361, in _run return self.run(fileStore) File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/cactus_preprocessor.py", line 72, in run checkUniqueHeaders(inFile, checkAssemblyHub=self.prepOptions.checkAssemblyHub) File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/checkUniqueHeaders.py", line 9, in checkUniqueHeaders for header, seq in fastaRead(inputFile): File "/usr/local/lib/python3.6/dist-packages/sonLib/bioio.py", line 805, in fastaRead line = fileHandle.readline() File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128) [2020-12-15T14:22:15+0800] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host z055 <========= [2020-12-15T14:22:16+0800] [MainThread] [W] [toil.jobGraph] Due to failure we are reducing the remaining retry count of job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-9qt7jufl with ID kind-CheckUniqueHeaders/instance-9qt7jufl to 0 [2020-12-15T14:22:16+0800] [MainThread] [W] [toil.leader] Job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-9qt7jufl with ID kind-CheckUniqueHeaders/instance-9qt7jufl is completely failed [2020-12-15T14:22:16+0800] [Thread-1 ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker CheckUniqueHeaders file:/scratch/pawsey0263/yangzhzh/projects/1_cactus/2_test_pairwise/jobstore kind-CheckUniqueHeaders/instance-1104cz3i. [2020-12-15T14:22:16+0800] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i [2020-12-15T14:22:16+0800] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i [2020-12-15T14:22:16+0800] [MainThread] [W] [toil.leader] Log from job kind-CheckUniqueHeaders/instance-1104cz3i follows: =========> [2020-12-15T14:22:14+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2020-12-15T14:22:14+0800] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host z055. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/toil/worker.py", line 368, in workerScript job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.6/dist-packages/cactus/shared/common.py", line 1369, in _runner super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1424, in _runner returnValues = self._run(jobGraph, fileStore) File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1361, in _run return self.run(fileStore) File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/cactus_preprocessor.py", line 72, in run checkUniqueHeaders(inFile, checkAssemblyHub=self.prepOptions.checkAssemblyHub) File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/checkUniqueHeaders.py", line 9, in checkUniqueHeaders for header, seq in fastaRead(inputFile): File "/usr/local/lib/python3.6/dist-packages/sonLib/bioio.py", line 805, in fastaRead line = fileHandle.readline() File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128) [2020-12-15T14:22:15+0800] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host z055 <========= [2020-12-15T14:22:16+0800] [MainThread] [W] [toil.jobGraph] Due to failure we are reducing the remaining retry count of job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i with ID kind-CheckUniqueHeaders/instance-1104cz3i to 0 [2020-12-15T14:22:16+0800] [MainThread] [W] [toil.leader] Job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i with ID kind-CheckUniqueHeaders/instance-1104cz3i is completely failed [2020-12-15T14:22:30+0800] [MainThread] [I] [toil.leader] Finished toil run with 10 failed jobs. [2020-12-15T14:22:30+0800] [MainThread] [I] [toil.leader] Failed jobs at end of the run: 'PreprocessSequence' kind-PreprocessSequence/instance-_ubs4q_w 'CactusPreprocessor' kind-CactusPreprocessor/instance-peyjj4j8 'CactusPreprocessor2' kind-CactusPreprocessor2/instance-1qwgzxe2 'BatchPreprocessor' kind-BatchPreprocessor/instance-eyw_iry5 'CactusPreprocessor2' kind-CactusPreprocessor2/instance-s2ye9mhi 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-9qt7jufl 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instance-lytwd43r 'PreprocessSequence' kind-PreprocessSequence/instance-pfojq5q4 'BatchPreprocessor' kind-BatchPreprocessor/instance-gjcbsctg 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i Traceback (most recent call last): File "/usr/local/bin/cactus", line 8, in sys.exit(main()) File "/usr/local/lib/python3.6/dist-packages/cactus/progressive/cactus_progressive.py", line 386, in main runCactusProgressive(options) File "/usr/local/lib/python3.6/dist-packages/cactus/progressive/cactus_progressive.py", line 434, in runCactusProgressive halID = toil.start(RunCactusPreprocessorThenProgressiveDown(options, project, memory=configWrapper.getDefaultMemory())) File "/usr/local/lib/python3.6/dist-packages/toil/common.py", line 846, in start return self._runMainLoop(rootJobGraph) File "/usr/local/lib/python3.6/dist-packages/toil/common.py", line 1138, in _runMainLoop jobCache=self._jobCache).run() File "/usr/local/lib/python3.6/dist-packages/toil/leader.py", line 269, in run raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) toil.leader.FailedJobsException: The job store 'file:/scratch/pawsey0263/yangzhzh/projects/1_cactus/2_test_pairwise/jobstore' contains 10 failed jobs: 'PreprocessSequence' kind-PreprocessSequence/instance-_ubs4q_w, 'CactusPreprocessor' kind-CactusPreprocessor/instance-peyjj4j8, 'CactusPreprocessor2' kind-CactusPreprocessor2/instance-1qwgzxe2, 'BatchPreprocessor' kind-BatchPreprocessor/instance-eyw_iry5, 'CactusPreprocessor2' kind-CactusPreprocessor2/instance-s2ye9mhi, 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-9qt7jufl, 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instance-lytwd43r, 'PreprocessSequence' kind-PreprocessSequence/instance-pfojq5q4, 'BatchPreprocessor' kind-BatchPreprocessor/instance-gjcbsctg, 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i Log from job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-9qt7jufl follows: =========> [2020-12-15T14:22:14+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2020-12-15T14:22:14+0800] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host z055. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/toil/worker.py", line 368, in workerScript job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.6/dist-packages/cactus/shared/common.py", line 1369, in _runner super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1424, in _runner returnValues = self._run(jobGraph, fileStore) File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1361, in _run return self.run(fileStore) File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/cactus_preprocessor.py", line 72, in run checkUniqueHeaders(inFile, checkAssemblyHub=self.prepOptions.checkAssemblyHub) File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/checkUniqueHeaders.py", line 9, in checkUniqueHeaders for header, seq in fastaRead(inputFile): File "/usr/local/lib/python3.6/dist-packages/sonLib/bioio.py", line 805, in fastaRead line = fileHandle.readline() File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128) [2020-12-15T14:22:15+0800] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host z055 <========= Log from job 'CheckUniqueHeaders' kind-CheckUniqueHeaders/instance-1104cz3i follows: =========> [2020-12-15T14:22:14+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG--- [2020-12-15T14:22:14+0800] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host z055. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/toil/worker.py", line 368, in workerScript job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.6/dist-packages/cactus/shared/common.py", line 1369, in _runner super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer) File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1424, in _runner returnValues = self._run(jobGraph, fileStore) File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1361, in _run return self.run(fileStore) File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/cactus_preprocessor.py", line 72, in run checkUniqueHeaders(inFile, checkAssemblyHub=self.prepOptions.checkAssemblyHub) File "/usr/local/lib/python3.6/dist-packages/cactus/preprocessor/checkUniqueHeaders.py", line 9, in checkUniqueHeaders for header, seq in fastaRead(inputFile): File "/usr/local/lib/python3.6/dist-packages/sonLib/bioio.py", line 805, in fastaRead line = fileHandle.readline() File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128) [2020-12-15T14:22:15+0800] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host z055 <========= srun: error: z055: task 0: Exited with exit code 1

zhenzhenyang-psu commented 3 years ago

my input file is a link of .fasta.gz file. Could it be that this .gz format is not supported? I will download the first first and compress it before running cactus.

glennhickey commented 3 years ago

Cactus does not support gzipped input, which is very annoying. There is already an issue here #303