search

BD2KGenomics / toil-rnaseq

UC Santa Cruz Computational Genomics Lab's Toil-based RNA-seq pipeline
Apache License 2.0
38 stars 10 forks source link

fastqc failed with file not found #185

Closed genec1 closed 3 years ago

genec1 commented 3 years ago

I'm getting further with getting toil-rnaseq to run, but it's still not going all the way. Now I seem to be dying with OSError: [Errno 2] No such file or directory: '/data/work/toil-59fec6ca-1a65-4ae0-b9db-da5c9f650c22-f81cf66e29f975035d1375c55f43d968/tmpEhraLA/6369919f-68ec-4bfb-9d9e-5704975041b5/tMqc9eF/R1_fastqc.html'

Here is the full run:

toil-rnaseq run \
--logError --logFile /data/work/$SRA.log  \
--workDir /data/work     \
--clean onSuccess --cleanWorkDir onSuccess \
--defaultMemory 60G --defaultCores 8  \
--config  /home/ec2-user/toil_config.yaml  --manifest /home/ec2-user/manifest.tsv \
/data/work/$SRA.jobStore

/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil_rnaseq/toil_rnaseq.py:203: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = rexpando(yaml.load(open(args.config).read()))
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f5ed1fc9fd0>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7efbfb6f6fd0>> ignored
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f5d44da6fd0>> ignored
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f8b28105fd0>> ignored
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f14f81e5fd0>> ignored
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 757, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/fileStore.py", line 1475, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f80bd766fd0>> ignored
Traceback (most recent call last):
  File "/home/ec2-user/toil-rnaseq/bin/toil-rnaseq", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil_rnaseq/toil_rnaseq.py", line 214, in main
    toil.start(Job.wrapJobFn(map_job, workflow, samples, config))
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/common.py", line 735, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/common.py", line 1011, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/leader.py", line 209, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/data/work/SRR1303078.jobStore' contains 9 failed jobs: 'Job' W/9/job0dp_wi, 'run_rsem' g/x/jobMpP8P7, 'run_rsem' f/J/jobSfgmQd, 'multiple_fastq_dowloading' 2/i/jobFrr1Kv, 'run_fastqc' o/q/jobStBao1, 'run_kallisto' 6/M/jobjA9248, 'run_fastqc' C/Z/jobZR7k6Q, 'run_kallisto' V/P/jobUYDP5T, 'run_star' V/b/jobYvysQ1
=========> Failed job 'run_rsem' g/x/jobMpP8P7
---TOIL WORKER OUTPUT LOG---
rsem_ref_hg38/
rsem_ref_hg38/hg38.seq
rsem_ref_hg38/hg38.fa
rsem_ref_hg38/hg38.n2g.idx.fa
rsem_ref_hg38/hg38.chrlist
rsem_ref_hg38/gencode.v23.annotation.gtf
rsem_ref_hg38/hg38.grp
rsem_ref_hg38/hg38.idx.fa
rsem_ref_hg38/hg38.transcripts.fa
rsem_ref_hg38/hg38.ti
Cannot open rsem.temp/rsem_alignable_1.fa! It may not exist.
Traceback (most recent call last):
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/worker.py", line 316, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1318, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1263, in _run
    return self.run(fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1447, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil_rnaseq/tools/quantifiers.py", line 90, in run_rsem
    dockerCall(job, parameters=parameters, workDir=job.tempDir, tool=rsem_version)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/lib/docker.py", line 55, in dockerCall
    return subprocessDockerCall(job=job, *args, **kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/lib/docker.py", line 177, in subprocessDockerCall
    out = subprocess.check_output(call, **params)
  File "/usr/lib64/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['docker', 'run', '--rm', '--log-driver', 'none', '-v', u'/data/work/toil-59fec6ca-1a65-4ae0-b9db-da5c9f650c22-f81cf66e29f975035d1375c55f43d968/tmptA1hRA/baf38069-2073-4fd3-8093-2f67a0e7fdcd/tJMYkKn:/data', '--name', 'runrsem--8j5VPk2tnHTt', 'quay.io/ucsc_cgl/rsem:1.2.25--d4275175cc8df36967db460b06337a14f40d2f21', '--paired-end', '--quiet', '--no-qualities', '-p', '8', '--forward-prob', '0.5', '--seed-length', '25', '--fragment-length-mean', '-1.0', '--bam', '/data/transcriptome.bam', u'/data/rsem_ref_hg38/hg38', 'rsem']' returned non-zero exit status 255
ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-57-5
<=========

=========> Failed job 'run_kallisto' 6/M/jobjA9248
---TOIL WORKER OUTPUT LOG---

[quant] fragment length distribution will be estimated from the data
[index] k-mer length: 31
[index] number of targets: 197,044
[index] number of k-mers: 115,344,928
[index] number of equivalence classes: 750,215
[quant] running in paired-end mode
[quant] will process pair 1: /data/R1.fastq
                             /data/R2.fastq
[quant] finding pseudoalignments for the reads ... done
[quant] processed 0 reads, 0 reads pseudoaligned
[~warn] no reads pseudoaligned.
[quant] estimated average fragment length: 0
[   em] quantifying the abundances ... done
[   em] the Expectation-Maximization algorithm ran for 52 rounds
terminate called after throwing an instance of 'std::domain_error'
  what():  nsamp must be -1 or >=1
Aborted
Traceback (most recent call last):
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/worker.py", line 316, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1318, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1263, in _run
    return self.run(fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1447, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil_rnaseq/tools/quantifiers.py", line 43, in run_kallisto
    dockerCall(job, workDir=job.tempDir, parameters=parameters, tool=kallisto_version)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/lib/docker.py", line 55, in dockerCall
    return subprocessDockerCall(job=job, *args, **kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/lib/docker.py", line 177, in subprocessDockerCall
    out = subprocess.check_output(call, **params)
  File "/usr/lib64/python2.7/subprocess.py", line 223, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['docker', 'run', '--rm', '--log-driver', 'none', '-v', u'/data/work/toil-59fec6ca-1a65-4ae0-b9db-da5c9f650c22-f81cf66e29f975035d1375c55f43d968/tmplwrUEX/4c55b657-5405-406b-9b29-54b77a946932/t240hQA:/data', '--name', 'runkallisto--qEz45dxyC7Fy', 'quay.io/ucsc_cgl/kallisto:0.43.1--355c19b1fb6fbb85f7f8293e95fb8a1e9d0da163', 'quant', '-i', '/data/kallisto_hg38.idx', '-t', '8', '-o', '/data/', '-b', '100', '--fusion', '/data/R1.fastq', '/data/R2.fastq']' returned non-zero exit status 134
ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-57-5
<=========

=========> Failed job 'run_fastqc' C/Z/jobZR7k6Q
---TOIL WORKER OUTPUT LOG---
Started analysis of R1.fastq
Failed to process file R1.fastq
java.lang.ArrayIndexOutOfBoundsException: -1
    at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.calculateDistribution(SequenceLengthDistribution.java:100)
    at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.raisesError(SequenceLengthDistribution.java:184)
    at uk.ac.babraham.FastQC.Report.HTMLReportArchive.startDocument(HTMLReportArchive.java:336)
    at uk.ac.babraham.FastQC.Report.HTMLReportArchive.<init>(HTMLReportArchive.java:84)
    at uk.ac.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:155)
    at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:110)
    at java.lang.Thread.run(Thread.java:745)
Started analysis of R2.fastq
Failed to process file R2.fastq
java.lang.ArrayIndexOutOfBoundsException: -1
    at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.calculateDistribution(SequenceLengthDistribution.java:100)
    at uk.ac.babraham.FastQC.Modules.SequenceLengthDistribution.raisesError(SequenceLengthDistribution.java:184)
    at uk.ac.babraham.FastQC.Report.HTMLReportArchive.startDocument(HTMLReportArchive.java:336)
    at uk.ac.babraham.FastQC.Report.HTMLReportArchive.<init>(HTMLReportArchive.java:84)
    at uk.ac.babraham.FastQC.Analysis.OfflineRunner.analysisComplete(OfflineRunner.java:155)
    at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:110)
    at java.lang.Thread.run(Thread.java:745)
Traceback (most recent call last):
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/worker.py", line 316, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1318, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1263, in _run
    return self.run(fileStore)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil/job.py", line 1447, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil_rnaseq/tools/qc.py", line 35, in run_fastqc
    tarball_files(tar_name='fastqc.tar.gz', file_paths=output_files, output_dir=job.tempDir)
  File "/home/ec2-user/toil-rnaseq/local/lib/python2.7/site-packages/toil_rnaseq/utils/files.py", line 20, in tarball_files
    f_out.add(file_path, arcname=arcname)
  File "/usr/lib64/python2.7/tarfile.py", line 2009, in add
    tarinfo = self.gettarinfo(name, arcname)
  File "/usr/lib64/python2.7/tarfile.py", line 1881, in gettarinfo
    statres = os.lstat(name)
OSError: [Errno 2] No such file or directory: '/data/work/toil-59fec6ca-1a65-4ae0-b9db-da5c9f650c22-f81cf66e29f975035d1375c55f43d968/tmpEhraLA/6369919f-68ec-4bfb-9d9e-5704975041b5/tMqc9eF/R1_fastqc.html'
ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-57-5
<=========

If I look at the directory where the missing file should be, I see this:

> ls -l /data/work/toil-59fec6ca-1a65-4ae0-b9db-da5c9f650c22-f81cf66e29f975035d1375c55f43d968/tmpEhraLA/6369919f-68ec-4bfb-9d9e-5704975041b5/tMqc9eF/
total 20
-rw-rw-r-- 1 ec2-user ec2-user   31 Nov 18 23:20 fastqc.tar.gz
-rw-r--r-- 1 ec2-user ec2-user 6421 Nov 18 23:20 R1_fastqc.zip
-rw-r--r-- 1 ec2-user ec2-user 6421 Nov 18 23:20 R2_fastqc.zip
jvivian commented 3 years ago

Hi @genec1 — It looks like multiple jobs are failing which makes me think it's an issue with the inputs. May I see the contents of /home/ec2-user/manifest.tsv and /home/ec2-user/toil_config.yaml please? I see the tools are being run in paired mode — are you submitting single-end samples?

genec1 commented 3 years ago

Below are the config files. In this case I was running a paired-end sample.

> cat ~/toil_config.yaml
output-dir: /data/work
star-index: file:///home/ec2-user/indices/starIndex_hg38_no_alt.tar.gz
rsem-ref: file:///home/ec2-user/indices/rsem_ref_hg38_no_alt.tar.gz
kallisto-index: file:///home/ec2-user/indices/kallisto_hg38.idx
hera-index: 
max-sample-size: 100G
cutadapt: true
fwd-3pr-adapter: AGATCGGAAGAG
rev-3pr-adapter: AGATCGGAAGAG
fastqc: true
bamqc:
ssec:
gdc-token:
wiggle:
save-bam:
ci-test:

> cat ~/manifest.tsv
fq  paired  SRR1303078  file:///data/samples/SRR1303078_1.fastq,file:///data/samples/SRR1303078_2.fastq
jvivian-atreca commented 3 years ago

Hi @genec1  — I can't see anything wrong with the config or manifest. I looked for the samples to see if i can replicate it, but it looks like I need privileges to download the files from SRA.

After looking through the logs, I don't believe this is actually a pipeline issue. Every step is failing on the samples, which indicates that there is some issue with them. For example, FASTQC:

Failed to process file R1.fastq

Kallisto:

[quant] running in paired-end mode [quant] will process pair 1: /data/R1.fastq /data/R2.fastq [quant] finding pseudoalignments for the reads ... done [quant] processed 0 reads, 0 reads pseudoaligned [~warn] no reads pseudoaligned.

You can test this out by downloading one of the tools, such as FASTQC, and testing it

docker run -v $(pwd):/data quay.io/ucsc_cgl/fastqc:0.11.5--be13567d00cd4c586edf8ae47d991815c8c72a49 /data/SRR1303078_1.fastq

Is there a chance the fastq files are actually gzipped but the extension is wrong? Can't really think of anything else.

Best, John

genec1 commented 3 years ago

I've discovered that there are some non-ASCII characters in the fastq files being generated by fasterq-dump. I believe that is the source of the problem. I'm now adding an additional step in the pipeline between fasterq-dump and toil-rnaseq to filter out these corrupted reads.