toil-rnaseq failure: 'Expando' object has no attribute 'ci_test'

genec1 commented 3 years ago

My toil-rnaseq runs are failing. The error messages are a mystery to me. Any advice is appreciated!

Run commands:

sudo pip install --upgrade pip
pip install toil[all]==3.12.0
virtualenv ~/toil-rnaseq
source ~/toil-rnaseq/bin/activate
export SRA="SRR3466057"
toil-rnaseq run \
    --logError --logFile /data/work/$SRA.log \
    --workDir /data/work \
    --clean onSuccess --cleanWorkDir onSuccess \
    --defaultMemory 60G --defaultCores 8 \
    --config  /home/ec2-user/toil_config.yaml \
    --manifest /home/ec2-user/manifest.tsv \
    /data/work/$SRA.jobStore

Contents of toil_config.yaml:

output-dir: /data/work
star-index: file://home/ec2-user/indices/starindex_hg38_no_alt.tar.gz
rsem-ref: file://home/ec2-user/indices/rsem_ref_hg38_no_alt.tar.gz
kallisto-index: file://home/ec2-user/indices/kallisto_hg38.idx
hera-index: file://home/ec2-user/indices/hera-index.tar.gz
max-sample-size: 20G
cutadapt: true
fwd-3pr-adapter: AGATCGGAAGAG
rev-3pr-adapter: AGATCGGAAGAG
fastqc: true

Contents of manifest.tsv:

fq  paired  SRR3466057  file:///data/samples/SRR3466057_1.fastq,file:///data/samples/SRR3466057_2.fastq

Output of run:

/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil_rnaseq/toil_rnaseq.py:203: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = rexpando(yaml.load(open(args.config).read()))
Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Traceback (most recent call last):
  File "/home/ec2-user/toil-rnaseq/bin/toil-rnaseq", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil_rnaseq/toil_rnaseq.py", line 214, in main
    toil.start(Job.wrapJobFn(map_job, workflow, samples, config))
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/common.py", line 735, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/common.py", line 1011, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/leader.py", line 209, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/data/work/SRR3466057.jobStore' contains 1 failed jobs: 'workflow' N/4/jobdCCZw0
=========> Failed job 'workflow' N/4/jobdCCZw0
---TOIL WORKER OUTPUT LOG---
Traceback (most recent call last):
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/worker.py", line 316, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1318, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1263, in _run
    return self.run(fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1447, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil_rnaseq/toil_rnaseq.py", line 102, in workflow
    if config.ci_test:
AttributeError: 'Expando' object has no attribute 'ci_test'
ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-56-180
<=========

jvivian-atreca commented 3 years ago

Hey @genec1 — you can use toil-rnaseq generate to create a template toil_config.yaml. The issue you're running into is that all of the fields in the config file need to be present, even if left blank (such as ci-test, etc).

genec1 commented 3 years ago

Thanks! That helped me get further. Will post the new error message as a new issue.

BD2KGenomics / toil-rnaseq

toil-rnaseq failure: 'Expando' object has no attribute 'ci_test' #184