search

BD2KGenomics / toil-rnaseq

UC Santa Cruz Computational Genomics Lab's Toil-based RNA-seq pipeline
Apache License 2.0
38 stars 10 forks source link

Toil run dying on SRA fastq file #183

Closed genec1 closed 3 years ago

genec1 commented 3 years ago

My attempt at a TOIL run on a SRA-derived fastq file is dying. Some samples do run to completion, but many -- like this one -- fail.

Here is the full run. Any debugging assistance is appreciated.

> sudo docker run -v /data/work:/data/work -v /home/ec2-user/indices:/indices -v /data/samples:/samples -v /var/run/docker.sock:/var/run/docker.sock quay.io/ucsc_cgl/rnaseq-cgl-pipeline --work_mount /data/work --star /indices/starIndex_hg38_no_alt.tar.gz --kallisto /indices/kallisto_hg38.idx --rsem /indices/rsem_ref_hg38_no_alt.tar.gz --logInfo --output-basename SRR1159995 --cores 8 --sample-single /samples/SRR1159995.fastq

INFO:root:Single FASTQS to run: /samples/SRR1159995.fastq
INFO:root:The work mount is: /data/work
INFO:root:Pipeline input locations:
/indices/starIndex_hg38_no_alt.tar.gz
/indices/rsem_ref_hg38_no_alt.tar.gz
/indices/kallisto_hg38.idx
INFO:root:Temporary directory created: /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2
INFO:root:Path to manifest: /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2
INFO:root:Docker Comand: ['toil-rnaseq', 'run', '/data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore', '--config', '/data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/toil-rnaseq.config', '--workDir', '/data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2', '--logInfo', '--retryCount', '1', '--maxCores=8', '--manifest=/data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/manifest-toil-rnaseq.tsv']
05b2c11e443c: 2020-11-16 21:54:14,837 INFO: toil.lib.bioio: Logging set at level: INFO
05b2c11e443c: 2020-11-16 21:54:14,837 INFO: toil.lib.bioio: Logging set at level: INFO
05b2c11e443c: 2020-11-16 21:54:14,838 INFO: toil.jobStores.fileJobStore: Path to job store directory is '/data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore'.
05b2c11e443c: 2020-11-16 21:54:14,839 INFO: toil.jobStores.abstractJobStore: The workflow ID is: 'acaaf1bc-55df-4b20-b0a9-c3c6b0b3c18c'
05b2c11e443c: 2020-11-16 21:54:14,841 INFO: toil.common: Using the single machine batch system
05b2c11e443c: 2020-11-16 21:54:14,841 WARNING: toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (66716438528).
05b2c11e443c: 2020-11-16 21:54:14,841 INFO: toil.common: Created the workflow directory at /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/toil-acaaf1bc-55df-4b20-b0a9-c3c6b0b3c18c
05b2c11e443c: 2020-11-16 21:54:14,841 WARNING: toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (695964704768).
05b2c11e443c: 2020-11-16 21:54:14,841 INFO: toil.batchSystems.singleMachine: Setting up the thread pool with 80 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 8.
05b2c11e443c: 2020-11-16 21:54:14,850 INFO: toil.common: Written the environment for the jobs to the environment file
05b2c11e443c: 2020-11-16 21:54:14,850 INFO: toil.common: Caching all jobs in job store
05b2c11e443c: 2020-11-16 21:54:14,850 INFO: toil.common: 0 jobs downloaded.
05b2c11e443c: 2020-11-16 21:54:14,860 INFO: toil.realtimeLogger: Real-time logging disabled
05b2c11e443c: 2020-11-16 21:54:14,862 INFO: toil.leader: (Re)building internal scheduler state
05b2c11e443c: 2020-11-16 21:54:14,862 INFO: toil.leader: Checked batch system has no running jobs and no updated jobs
05b2c11e443c: 2020-11-16 21:54:14,862 INFO: toil.leader: Found 1 jobs to start and 0 jobs with successors to run
05b2c11e443c: 2020-11-16 21:54:14,864 INFO: toil.leader: Starting the main loop
05b2c11e443c: 2020-11-16 21:54:14,865 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore 3/e/jobNBkssz'.
05b2c11e443c: 2020-11-16 21:54:15,130 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore w/R/jobVuRC5W'.
05b2c11e443c: 2020-11-16 21:54:15,367 INFO: toil.leader: Got message from job at time 11-16-2020 21:54:15: UUID: SRR1159995
URL: file:///data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/SRR1159995merged.fastq
Paired: False
File Type: fq
Cores: 8
CIMode: None
05b2c11e443c: 2020-11-16 21:54:17,371 INFO: toil.leader: Got message from job at time 11-16-2020 21:54:17: Successfully deleted local copies of file with ID 'a/w/tmpE5HY95.tmp'.
05b2c11e443c: 2020-11-16 21:54:17,394 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore R/6/jobyUFRRB'.
05b2c11e443c: 2020-11-16 21:54:17,659 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore q/G/job9Z2pz4'.
05b2c11e443c: 2020-11-16 21:54:17,872 INFO: toil.leader: Got message from job at time 11-16-2020 21:54:17: Added file with ID 't/I/tmpI4bLe6.tmp' to the list of files to be globally deleted.
05b2c11e443c: 2020-11-16 21:55:34,930 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore u/I/job1fkCpB'.
05b2c11e443c: 2020-11-16 21:55:34,994 INFO: toil.leader: Got message from job at time 11-16-2020 21:55:34: Processing sample: SRR1159995
05b2c11e443c: 2020-11-16 21:55:34,994 INFO: toil.leader: Got message from job at time 11-16-2020 21:55:34: Successfully deleted local copies of file with ID 'a/w/tmpE5HY95.tmp'.
05b2c11e443c: 2020-11-16 21:55:34,994 INFO: toil.leader: Got message from job at time 11-16-2020 21:55:34: Successfully deleted local copies of file with ID 'a/w/tmpE5HY95.tmp'.
05b2c11e443c: 2020-11-16 21:55:34,994 INFO: toil.leader: Got message from job at time 11-16-2020 21:55:34: Successfully deleted local copies of file with ID 'G/E/tmpmoirTy.tmp'.
05b2c11e443c: 2020-11-16 21:55:35,045 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore Q/Y/jobDoq3UF'.
05b2c11e443c: 2020-11-16 21:55:35,045 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore 3/R/jobSC18zG'.
05b2c11e443c: 2020-11-16 21:55:35,496 INFO: toil.leader: Got message from job at time 11-16-2020 21:55:35: Queueing FastQC job for:
05b2c11e443c: 2020-11-16 21:55:35,496 INFO: toil.leader: Got message from job at time 11-16-2020 21:55:35: Queueing Kallisto job for: SRR1159995
05b2c11e443c: 2020-11-16 21:55:35,496 INFO: toil.leader: Got message from job at time 11-16-2020 21:55:35: Queueing STAR alignment for: SRR1159995
05b2c11e443c: 2020-11-16 21:55:35,496 INFO: toil.leader: Got message from job at time 11-16-2020 21:55:35: Added file with ID 'c/q/tmpw0S4nN.tmp' to the list of files to be globally deleted.
05b2c11e443c: 2020-11-16 21:55:35,496 INFO: toil.leader: Got message from job at time 11-16-2020 21:55:35: Added file with ID 'K/P/tmpwZKf3c.tmp' to the list of files to be globally deleted.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f2eca7dba10>> ignored
05b2c11e443c: 2020-11-16 21:55:37,159 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: 3/R/jobSC18zG
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: Reporting file: 3/R/jobSC18zG
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  ---TOIL WORKER OUTPUT LOG---
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:Can't find resource for leader path '/usr/local/lib/python2.7/dist-packages/toil_rnaseq'
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/dist-packages', name='toil_rnaseq.rnaseq_cgl_pipeline', fromVirtualEnv=False)
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:Can't find resource for leader path '/usr/local/lib/python2.7/dist-packages/toil_rnaseq'
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/dist-packages', name='toil_rnaseq.rnaseq_cgl_pipeline', fromVirtualEnv=False)
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  INFO:toil.fileStore:LOG-TO-MASTER: Queueing RSEM job for: SRR1159995
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  Traceback (most recent call last):
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:      returnValues = self._run(jobWrapper, fileStore)
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:      return self.run(fileStore)
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:    File "/usr/local/lib/python2.7/dist-packages/toil_rnaseq/rnaseq_cgl_pipeline.py", line 137, in star_alignment
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:      disk = '2G' if config.ci_test else r1_id.size + r2_id.size + 80530636800  # 75 G for STAR index and tmp files
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  AttributeError: 'NoneType' object has no attribute 'size'
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  Exiting the worker because of a failed jobWrapper on host 05b2c11e443c
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host 05b2c11e443c
05b2c11e443c: 2020-11-16 21:55:37,160 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 3/R/jobSC18zG to 1
05b2c11e443c: 2020-11-16 21:55:37,161 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore 3/R/jobSC18zG'.
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f8bf6f65f90>> ignored
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: Q/Y/jobDoq3UF
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Reporting file: Q/Y/jobDoq3UF
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  ---TOIL WORKER OUTPUT LOG---
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:Can't find resource for leader path '/usr/local/lib/python2.7/dist-packages/toil_lib'
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/dist-packages', name='toil_lib.tools.QC', fromVirtualEnv=False)
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:Can't find resource for leader path '/usr/local/lib/python2.7/dist-packages/toil_lib'
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/dist-packages', name='toil_lib.tools.QC', fromVirtualEnv=False)
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  INFO:toil.fileStore:CACHE: Cache hit on file with ID 'G/E/tmpmoirTy.tmp'.
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  Started analysis of R1.fastq
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  Analysis complete for R1.fastq
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  Failed to process file R1.fastq
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:  java.lang.ArrayIndexOutOfBoundsException: -1
05b2c11e443c: 2020-11-16 21:55:39,128 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.calculateDistribution(SequenceLengthDistribution.java:100)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.raisesError(SequenceLengthDistribution.java:184)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Report.HTMLReportArchive.startDocument(HTMLReportArchive.java:336)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Report.HTMLReportArchive.<init>(HTMLReportArchive.java:84)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:155)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:110)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      at java.lang.Thread.run(Thread.java:745)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:  INFO:toil.fileStore:LOG-TO-MASTER: Successfully deleted local copies of file with ID 'G/E/tmpmoirTy.tmp'.
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:  Traceback (most recent call last):
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      returnValues = self._run(jobWrapper, fileStore)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      return self.run(fileStore)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil_lib/tools/QC.py", line 28, in run_fastqc
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      tarball_files(tar_name='fastqc.tar.gz', file_paths=output_files, output_dir=work_dir)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil_lib/files.py", line 23, in tarball_files
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      f_out.add(file_path, arcname=arcname)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/lib/python2.7/tarfile.py", line 1979, in add
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      tarinfo = self.gettarinfo(name, arcname)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/lib/python2.7/tarfile.py", line 1851, in gettarinfo
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:      statres = os.lstat(name)
05b2c11e443c: 2020-11-16 21:55:39,129 WARNING: toil.leader: Q/Y/jobDoq3UF:  OSError: [Errno 2] No such file or directory: '/data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/toil-acaaf1bc-55df-4b20-b0a9-c3c6b0b3c18c/tmpmvvS3L/d79074ca-1908-40cd-8cd8-5c11e368ad15/tH9aksq/R1_fastqc.html'
05b2c11e443c: 2020-11-16 21:55:39,130 WARNING: toil.leader: Q/Y/jobDoq3UF:  Exiting the worker because of a failed jobWrapper on host 05b2c11e443c
05b2c11e443c: 2020-11-16 21:55:39,130 WARNING: toil.leader: Q/Y/jobDoq3UF:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host 05b2c11e443c
05b2c11e443c: 2020-11-16 21:55:39,130 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job Q/Y/jobDoq3UF to 1
05b2c11e443c: 2020-11-16 21:55:39,130 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore Q/Y/jobDoq3UF'.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f88fd425a10>> ignored
05b2c11e443c: 2020-11-16 21:55:39,294 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: 3/R/jobSC18zG
05b2c11e443c: 2020-11-16 21:55:39,294 WARNING: toil.leader: Reporting file: 3/R/jobSC18zG
05b2c11e443c: 2020-11-16 21:55:39,294 WARNING: toil.leader: 3/R/jobSC18zG:  ---TOIL WORKER OUTPUT LOG---
05b2c11e443c: 2020-11-16 21:55:39,294 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
05b2c11e443c: 2020-11-16 21:55:39,294 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:Can't find resource for leader path '/usr/local/lib/python2.7/dist-packages/toil_rnaseq'
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/dist-packages', name='toil_rnaseq.rnaseq_cgl_pipeline', fromVirtualEnv=False)
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:Can't find resource for leader path '/usr/local/lib/python2.7/dist-packages/toil_rnaseq'
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/dist-packages', name='toil_rnaseq.rnaseq_cgl_pipeline', fromVirtualEnv=False)
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  INFO:toil.fileStore:LOG-TO-MASTER: Queueing RSEM job for: SRR1159995
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  Traceback (most recent call last):
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:      returnValues = self._run(jobWrapper, fileStore)
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:      return self.run(fileStore)
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:    File "/usr/local/lib/python2.7/dist-packages/toil_rnaseq/rnaseq_cgl_pipeline.py", line 137, in star_alignment
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:      disk = '2G' if config.ci_test else r1_id.size + r2_id.size + 80530636800  # 75 G for STAR index and tmp files
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  AttributeError: 'NoneType' object has no attribute 'size'
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  Exiting the worker because of a failed jobWrapper on host 05b2c11e443c
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host 05b2c11e443c
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 3/R/jobSC18zG to 0
05b2c11e443c: 2020-11-16 21:55:39,296 WARNING: toil.leader: Job: 3/R/jobSC18zG is completely failed
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7fc7234f5f90>> ignored
05b2c11e443c: 2020-11-16 21:55:43,212 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: Q/Y/jobDoq3UF
05b2c11e443c: 2020-11-16 21:55:43,212 WARNING: toil.leader: Reporting file: Q/Y/jobDoq3UF
05b2c11e443c: 2020-11-16 21:55:43,212 WARNING: toil.leader: Q/Y/jobDoq3UF:  ---TOIL WORKER OUTPUT LOG---
05b2c11e443c: 2020-11-16 21:55:43,212 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
05b2c11e443c: 2020-11-16 21:55:43,212 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:Can't find resource for leader path '/usr/local/lib/python2.7/dist-packages/toil_lib'
05b2c11e443c: 2020-11-16 21:55:43,212 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/dist-packages', name='toil_lib.tools.QC', fromVirtualEnv=False)
05b2c11e443c: 2020-11-16 21:55:43,212 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
05b2c11e443c: 2020-11-16 21:55:43,212 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:Can't find resource for leader path '/usr/local/lib/python2.7/dist-packages/toil_lib'
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/usr/local/lib/python2.7/dist-packages', name='toil_lib.tools.QC', fromVirtualEnv=False)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:  INFO:toil.fileStore:CACHE: Cache hit on file with ID 'G/E/tmpmoirTy.tmp'.
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:  Started analysis of R1.fastq
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:  Analysis complete for R1.fastq
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:  Failed to process file R1.fastq
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:  java.lang.ArrayIndexOutOfBoundsException: -1
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.calculateDistribution(SequenceLengthDistribution.java:100)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.raisesError(SequenceLengthDistribution.java:184)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Report.HTMLReportArchive.startDocument(HTMLReportArchive.java:336)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Report.HTMLReportArchive.<init>(HTMLReportArchive.java:84)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:155)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:110)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      at java.lang.Thread.run(Thread.java:745)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:  INFO:toil.fileStore:LOG-TO-MASTER: Successfully deleted local copies of file with ID 'G/E/tmpmoirTy.tmp'.
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:  Traceback (most recent call last):
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      returnValues = self._run(jobWrapper, fileStore)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      return self.run(fileStore)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil_lib/tools/QC.py", line 28, in run_fastqc
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      tarball_files(tar_name='fastqc.tar.gz', file_paths=output_files, output_dir=work_dir)
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/local/lib/python2.7/dist-packages/toil_lib/files.py", line 23, in tarball_files
05b2c11e443c: 2020-11-16 21:55:43,213 WARNING: toil.leader: Q/Y/jobDoq3UF:      f_out.add(file_path, arcname=arcname)
05b2c11e443c: 2020-11-16 21:55:43,214 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/lib/python2.7/tarfile.py", line 1979, in add
05b2c11e443c: 2020-11-16 21:55:43,214 WARNING: toil.leader: Q/Y/jobDoq3UF:      tarinfo = self.gettarinfo(name, arcname)
05b2c11e443c: 2020-11-16 21:55:43,214 WARNING: toil.leader: Q/Y/jobDoq3UF:    File "/usr/lib/python2.7/tarfile.py", line 1851, in gettarinfo
05b2c11e443c: 2020-11-16 21:55:43,214 WARNING: toil.leader: Q/Y/jobDoq3UF:      statres = os.lstat(name)
05b2c11e443c: 2020-11-16 21:55:43,214 WARNING: toil.leader: Q/Y/jobDoq3UF:  OSError: [Errno 2] No such file or directory: '/data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/toil-acaaf1bc-55df-4b20-b0a9-c3c6b0b3c18c/tmp8Kk9aD/1658618b-06b7-4b81-8c41-78baccbedd6d/tAqX3le/R1_fastqc.html'
05b2c11e443c: 2020-11-16 21:55:43,214 WARNING: toil.leader: Q/Y/jobDoq3UF:  Exiting the worker because of a failed jobWrapper on host 05b2c11e443c
05b2c11e443c: 2020-11-16 21:55:43,214 WARNING: toil.leader: Q/Y/jobDoq3UF:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host 05b2c11e443c
05b2c11e443c: 2020-11-16 21:55:43,214 WARNING: toil.leader: Q/Y/jobDoq3UF:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job Q/Y/jobDoq3UF to 0
05b2c11e443c: 2020-11-16 21:55:43,214 WARNING: toil.leader: Job: Q/Y/jobDoq3UF is completely failed
05b2c11e443c: 2020-11-16 21:55:43,217 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore F/w/jobJ4ejSN'.
05b2c11e443c: 2020-11-16 21:56:01,545 INFO: toil.leader: Got message from job at time 11-16-2020 21:56:01: Successfully deleted local copies of file with ID 'C/0/tmp05DJG2.tmp'.
05b2c11e443c: 2020-11-16 21:56:01,545 INFO: toil.leader: Got message from job at time 11-16-2020 21:56:01: Successfully deleted local copies of file with ID 'G/E/tmpmoirTy.tmp'.
05b2c11e443c: 2020-11-16 21:56:05,680 INFO: toil.leader: No jobs left to run so exiting.
05b2c11e443c: 2020-11-16 21:56:05,680 INFO: toil.leader: Finished the main loop
05b2c11e443c: 2020-11-16 21:56:05,680 INFO: toil.leader: Waiting for stats and logging collator process to finish ...
05b2c11e443c: 2020-11-16 21:56:06,055 INFO: toil.leader: ... finished collating stats and logs. Took 0.374834060669 seconds
05b2c11e443c: 2020-11-16 21:56:06,055 INFO: toil.leader: Waiting for service manager thread to finish ...
05b2c11e443c: 2020-11-16 21:56:06,874 INFO: toil.leader: ... finished shutting down the service manager. Took 0.818706035614 seconds
05b2c11e443c: 2020-11-16 21:56:06,874 INFO: toil.leader: Finished toil run with 5 failed jobs
05b2c11e443c: 2020-11-16 21:56:06,874 INFO: toil.leader: Failed jobs at end of the run: set(['3/R/jobSC18zG', 'R/6/jobyUFRRB', 'u/I/job1fkCpB', 'Q/Y/jobDoq3UF', '3/e/jobNBkssz'])
Traceback (most recent call last):
  File "/usr/local/bin/toil-rnaseq", line 9, in <module>
    load_entry_point('toil-rnaseq==3.1.1', 'console_scripts', 'toil-rnaseq')()
  File "/usr/local/lib/python2.7/dist-packages/toil_rnaseq/rnaseq_cgl_pipeline.py", line 595, in main
    Job.Runner.startToil(Job.wrapJobFn(map_job, download_sample, samples, config), args)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 500, in startToil
    return toil.start(job)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 562, in start
    return self._runMainLoop(job)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 791, in _runMainLoop
    jobCache=self._jobCache)
  File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 712, in mainLoop
    raise FailedJobsException( config.jobStore, len(toilState.totalFailedJobs) )
toil.leader.FailedJobsException: The job store '/data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2/jobStore' contains 5 failed jobs

INFO:root:Pipeline terminated, changing ownership of output files from root to user.
INFO:root:Cleaning up temporary directory: /data/work/Toil-RNAseq-0b5bdadd-6e93-4782-93d4-3ebcbc2c0fe2
INFO:root:output dir is /data/work and files are:
:
INFO:root:fail files are:
:
jvivian commented 3 years ago

Hi @genec1 — Sorry you ran into this issue. The root cause looks to be:

05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:      disk = '2G' if config.ci_test else r1_id.size + r2_id.size + 80530636800  # 75 G for STAR index and tmp files
05b2c11e443c: 2020-11-16 21:55:39,295 WARNING: toil.leader: 3/R/jobSC18zG:  AttributeError: 'NoneType' object has no attribute 'size'

Given that you're running single samples,r2_id is set to None which is a pretty embarrassing bug. I tried to find this in the code but it does not exist in the latest version, which is when I noticed that you're running the quay.io Docker container. The container is unfortunately out of sync with the main codebase (the group changed CI after I moved on, which is what built/pushed the containers). So you'll either need to build your own Docker container or use the python (pip) version of the code.

While I generally support using containers wherever possible, the python version is the preferred way to run this given the above issue. I also didn't write the Docker support for this pipeline and have noticed several other issues besides the one you ran into.

You will likely need to change the Toil version after pip installing to: pip install toil[all]==3.12.0 as the team made several backwards-breaking changes after I left. Apologies that support for this is limited, but I graduated a while ago. Feel free to continue to open issues or let me know if you run into any problems and I'll try and get to them when I can.

Cheers, John

genec1 commented 3 years ago

Thanks for letting me know that I'm on a deprecated version! I just tried running toil installed via pip as you recommend. Unfortunately I'm having issues with that too, but I'll post that as a separate issue.