search

BD2KGenomics / toil-rnaseq

UC Santa Cruz Computational Genomics Lab's Toil-based RNA-seq pipeline
Apache License 2.0
38 stars 10 forks source link

OSError: [Errno 2] No such file or directory from STAR #78

Closed wshands closed 7 years ago

wshands commented 7 years ago

While running the pipeline failed with the STAR error below. The pipeline was run on a AWS c4.8xlarge instance with 1 Terabyte of disk space on root@4f374c77a227:/# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.1 LTS Release: 16.04 Codename: xenial

Linux 4f374c77a227 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Status: Downloaded newer image for quay.io/ucsc_cgl/rnaseq-cgl-pipeline:3.2.1-1 [job temp8427763556155856749.cwl] /datastore$ docker \ run \ -i \ --volume=/var/run/docker.sock:/var/run/docker.sock:rw \ --volume=/datastore:/datastore:rw \ --volume=/datastore/./datastore/launcher-59e474c9-633e-4049-8047-f92ab0f71743/inputs/ebefc999-b2ef-45ad-bdc2-f9bfadced1cb/THR21_0552_S01_3_R1.fq.gz:/var/lib/cwl/stg227ac5c2-80e7-4371-aaaf-a13a83852e93/THR21_0552_S01_3_R1.fq.gz:ro \ --volume=/datastore/./datastore/launcher-59e474c9-633e-4049-8047-f92ab0f71743/inputs/69ee863a-2780-48e8-ba4d-3864aac21321/THR21_0552_S01_3_R2.fq.gz:/var/lib/cwl/stg216dd37f-18d0-4a77-ac16-fc1e66871231/THR21_0552_S01_3_R2.fq.gz:ro \ --volume=/datastore/./datastore/launcher-59e474c9-633e-4049-8047-f92ab0f71743/inputs/fea318e5-587b-4b8c-8127-94eb3fdf2da1/THR21_0552_S01_8_R1.fq.gz:/var/lib/cwl/stg107430d8-2939-4ddc-822e-f0d0bfab3c7c/THR21_0552_S01_8_R1.fq.gz:ro \ --volume=/datastore/./datastore/launcher-59e474c9-633e-4049-8047-f92ab0f71743/inputs/ab99db05-fe87-44b6-bcf4-54b36c6f1e6b/kallisto_hg38.idx:/var/lib/cwl/stg7bce377c-c3ad-4cdf-b5e6-5bd53d40e9fa/kallisto_hg38.idx:ro \ --volume=/datastore/./datastore/launcher-59e474c9-633e-4049-8047-f92ab0f71743/inputs/89883824-e63e-4078-b343-c3a95afe7141/THR21_0552_S01_8_R2.fq.gz:/var/lib/cwl/stg03f67084-2671-44c0-af73-b6088081d7ce/THR21_0552_S01_8_R2.fq.gz:ro \ --volume=/datastore/./datastore/launcher-59e474c9-633e-4049-8047-f92ab0f71743/inputs/88e3ee3f-c8ed-4b1c-8472-2565072c7eda/rsem_ref_hg38_no_alt.tar.gz:/var/lib/cwl/stg9225103e-0947-4ae8-8284-3fb84f53feb1/rsem_ref_hg38_no_alt.tar.gz:ro \ --volume=/datastore/./datastore/launcher-59e474c9-633e-4049-8047-f92ab0f71743/inputs/b7f691bd-bbaa-41fe-849f-579ac74065ad/starIndex_hg38_no_alt.tar.gz:/var/lib/cwl/stg19260c52-1541-4a9b-9839-6042787f8117/starIndex_hg38_no_alt.tar.gz:ro \ --volume=/datastore/datastore/launcher-59e474c9-633e-4049-8047-f92ab0f71743:/tmp:rw \ --workdir=/datastore \ --read-only=true \ --user=0 \ --rm \ --env=TMPDIR=/datastore/datastore/launcher-59e474c9-633e-4049-8047-f92ab0f71743 \ --env=HOME=/datastore \ quay.io/ucsc_cgl/rnaseq-cgl-pipeline:3.2.1-1 \ --logDebug \ --bamqc \ --cores \ 36 \ --kallisto \ /var/lib/cwl/stg7bce377c-c3ad-4cdf-b5e6-5bd53d40e9fa/kallisto_hg38.idx \ --no-clean \ --output-basename \ THR21_0552_S01 \ --resume \ '' \ --rsem \ /var/lib/cwl/stg9225103e-0947-4ae8-8284-3fb84f53feb1/rsem_ref_hg38_no_alt.tar.gz \ --sample-paired \ /var/lib/cwl/stg227ac5c2-80e7-4371-aaaf-a13a83852e93/THR21_0552_S01_3_R1.fq.gz,/var/lib/cwl/stg216dd37f-18d0-4a77-ac16-fc1e66871231/THR21_0552_S01_3_R2.fq.gz,/var/lib/cwl/stg107430d8-2939-4ddc-822e-f0d0bfab3c7c/THR21_0552_S01_8_R1.fq.gz,/var/lib/cwl/stg03f67084-2671-44c0-af73-b6088081d7ce/THR21_0552_S01_8_R2.fq.gz \ --save-bam \ --save-wiggle \ --star \ /var/lib/cwl/stg19260c52-1541-4a9b-9839-6042787f8117/starIndex_hg38_no_alt.tar.gz \ --work_mount \ /datastore ...

8d054e16506b: 2017-04-13 10:06:01,851 WARNING: toil.leader: A/4/jobrA5ZIk: Status: Downloaded newer image for quay.io/ucsc_cgl/star:2.4.2a--bcbd5122b69ff6ac4ef61958e47bde94001cfe80 8d054e16506b: 2017-04-13 10:06:01,851 WARNING: toil.leader: A/4/jobrA5ZIk: Apr 13 09:04:59 ..... Started STAR run 8d054e16506b: 2017-04-13 10:06:01,851 WARNING: toil.leader: A/4/jobrA5ZIk: Apr 13 09:04:59 ..... Loading genome 8d054e16506b: 2017-04-13 10:06:01,851 WARNING: toil.leader: A/4/jobrA5ZIk: Apr 13 09:06:08 ..... Started mapping 8d054e16506b: 2017-04-13 10:06:01,851 WARNING: toil.leader: A/4/jobrA5ZIk: Apr 13 10:00:54 ..... Started sorting BAM 8d054e16506b: 2017-04-13 10:06:01,851 WARNING: toil.leader: A/4/jobrA5ZIk: Killed 8d054e16506b: 2017-04-13 10:06:01,851 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Obtained lock on file /datastore/Toil-RNAseq-5d684c33-bfec-4239-9330-ace18d833068/toil-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/cache-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/.cacheLock 8d054e16506b: 2017-04-13 10:06:01,851 WARNING: toil.leader: A/4/jobrA5ZIk: INFO:toil.fileStore:CACHE: Added file with ID '7/N/tmpyF9riJ.tmp' to the cache. 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Released lock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Obtained lock on file /datastore/Toil-RNAseq-5d684c33-bfec-4239-9330-ace18d833068/toil-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/cache-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/.cacheLock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: INFO:toil.fileStore:CACHE: Added file with ID 'c/b/tmpjBHFBm.tmp' to the cache. 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Released lock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Obtained lock on file /datastore/Toil-RNAseq-5d684c33-bfec-4239-9330-ace18d833068/toil-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/cache-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/.cacheLock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: INFO:toil.fileStore:LOG-TO-MASTER: Successfully deleted local copies of file with ID 'c/b/tmpjBHFBm.tmp'. 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Released lock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Obtained lock on file /datastore/Toil-RNAseq-5d684c33-bfec-4239-9330-ace18d833068/toil-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/cache-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/.cacheLock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: INFO:toil.fileStore:LOG-TO-MASTER: Successfully deleted local copies of file with ID 'G/u/tmpM3z6ih.tmp'. 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Released lock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Obtained lock on file /datastore/Toil-RNAseq-5d684c33-bfec-4239-9330-ace18d833068/toil-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/cache-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/.cacheLock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: INFO:toil.fileStore:LOG-TO-MASTER: Successfully deleted local copies of file with ID 'E/L/tmpjkW35D.tmp'. 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Released lock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Obtained lock on file /datastore/Toil-RNAseq-5d684c33-bfec-4239-9330-ace18d833068/toil-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/cache-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/.cacheLock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: INFO:toil.fileStore:LOG-TO-MASTER: Successfully deleted local copies of file with ID '7/N/tmpyF9riJ.tmp'. 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Released lock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Obtained lock on file /datastore/Toil-RNAseq-5d684c33-bfec-4239-9330-ace18d833068/toil-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/cache-cbe92979-c6f8-4cc2-9f4e-702004b4c7c7/.cacheLock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: DEBUG:toil.fileStore:CACHE: Released lock 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: Traceback (most recent call last): 8d054e16506b: 2017-04-13 10:06:01,852 WARNING: toil.leader: A/4/jobrA5ZIk: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: returnValues = self._run(jobWrapper, fileStore) 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: return self.run(fileStore) 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: File "/usr/local/lib/python2.7/dist-packages/toil_lib/tools/aligners.py", line 69, in run_star 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: log_id = job.fileStore.writeGlobalFile(os.path.join(work_dir, 'rnaLog.final.out')) 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 207, in writeGlobalFile 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: os.link(absLocalFileName, self.jobStore._getAbsPath(jobStoreFileID)) 8d054e16506b: 2017-04-13 10:06:01,853 WARNING: toil.leader: A/4/jobrA5ZIk: OSError: [Errno 2] No such file or directory

jvivian commented 7 years ago

Closing, could not reproduce.

Single-machine mode Cores = 16 Memory = 230GB Disk = 3.0TB Runtime = 13 hours