search

kundajelab / atac_dnase_pipelines

ATAC-seq and DNase-seq processing pipeline
BSD 3-Clause "New" or "Revised" License
161 stars 81 forks source link

SCG SLURM hg38 permissions error? #130

Closed mkdegorter closed 6 years ago

mkdegorter commented 6 years ago

I am getting an error that I believe is related to permission to access hg38 on SCG. At the call-bowtie2 step, the pipeline fails.

Stderr: Traceback (most recent call last): File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_bowtie2.py", line 207, in main() File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_bowtie2.py", line 151, in main untar(tar, args.out_dir) File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 113, in untar run_shell_cmd(cmd) File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 230, in run_shell_cmd os.killpg(pgid, signal.SIGKILL) OSError: [Errno 3] No such process ln: failed to access ‘.bam’: No such file or directory ln: failed to access ‘.bai’: No such file or directory ln: failed to access ‘.align.log’: No such file or directory ln: failed to access ‘.flagstat.qc’: No such file or directory

Stdout: [2018-07-05 12:39:20,742 INFO] ['/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_bowtie2.py', '/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/03a44c57-75aa-44b0-b167-2ca44e06407f/call-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar', '/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/03a44c57-75aa-44b0-b167-2ca44e06407f/call-bowtie2/shard-0/inputs/-1978335035/merge_fastqs_R1_SCS47_merge_R1.trim.merged.fastq.gz', '/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/03a44c57-75aa-44b0-b167-2ca44e06407f/call-bowtie2/shard-0/inputs/-1978335035/merge_fastqs_R2_SCS47_merge_R2.trim.merged.fastq.gz', '--paired-end', '--multimapping', '4', '--nth', '4'] [2018-07-05 12:39:20,742 INFO] Initializing and making output directory... [2018-07-05 12:39:20,742 INFO] Generating read length file... [2018-07-05 12:39:41,015 INFO] Unpacking bowtie2 index tar... [2018-07-05 12:39:41,020 INFO] run_shell_cmd: PID=198613, CMD=tar xvf /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/03a44c57-75aa-44b0-b167-2ca44e06407f/call-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar -C . PID=198613: tar: /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/03a44c57-75aa-44b0-b167-2ca44e06407f/call-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar: Cannot open: Permission denied PID=198613: tar: Error is not recoverable: exiting now [2018-07-05 12:39:41,092 ERROR] Unknown exception caught. Killing process group 198613... Traceback (most recent call last): File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 224, in run_shell_cmd p.returncode, cmd) CalledProcessError: Command 'tar xvf /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/03a44c57-75aa-44b0-b167-2ca44e06407f/call-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar -C .' returned non-zero exit status 2

In the .json file I am using this line to specify the genome: "atac.genome_tsv" : "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/genome/scg/hg38_scg.tsv",

I've started to install my own version of hg38 as a work around, but would prefer to use the common files as advised in the documentation - any help would be appreciated. Thanks!

Note: I am using my own installation of Cromwell (cromwell-33.1) as it did not appear to be present in the dependencies installation.

leepc12 commented 6 years ago

You don't have to install genome data on SCG. There is a shared genome data for hg38. Please try with a genome TSV file genome/scg/hg38_scg.tsv.

Also, please post issues on https://github.com/ENCODE-DCC/atac-seq-pipeline/ next time.

mkdegorter commented 6 years ago

Thanks Jin. I tried again with the following line in the .json file:

"atac.genome_tsv" : "genome/scg/hg38_scg.tsv",

The pipeline fails with this output to the screen:

[2018-07-06 12:13:48,53] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.trim_adapter:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_579835bc_trim_adapter \ -D /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/579835bc-3ce3-4489-bd44-250bf952540b/call-trim_adapter/shard-0 \ -o /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/579835bc-3ce3-4489-bd44-250bf952540b/call-trim_adapter/shard-0/execution/stdout \ -e /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/579835bc-3ce3-4489-bd44-250bf952540b/call-trim_adapter/shard-0/execution/stderr \ -t 1440 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=2 \ --mem=12000 \ \ --account smontgom \ --wrap "/bin/bash /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/579835bc-3ce3-4489-bd44-250bf952540b/call-trim_adapter/shard-0/execution/script" [2018-07-06 12:13:48,75] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.trim_adapter:0:1]: job id: 2129106 [2018-07-06 12:13:48,75] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.read_genome_tsv:NA:1]: job id: 2129105 [2018-07-06 12:13:48,77] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.trim_adapter:0:1]: Status change from - to WaitingForReturnCodeFile [2018-07-06 12:13:48,77] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.read_genome_tsv:NA:1]: Status change from - to WaitingForReturnCodeFile [2018-07-06 12:14:26,69] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.read_genome_tsv:NA:1]: Status change from WaitingForReturnCodeFile to Done [2018-07-06 12:52:36,69] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.trim_adapter:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-07-06 12:52:37,29] [error] WorkflowManagerActor Workflow 579835bc-3ce3-4489-bd44-250bf952540b failed (during ExecutingWorkflowState): cromwell.backend.standard.StandardAsyncExecutionActor$$anon$2: Failed to evaluate job outputs: Bad output 'read_genome_tsv.genome': java.nio.file.NoSuchFileException: /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/579835bc-3ce3-4489-bd44-250bf952540b/call-read_genome_tsv/execution/genome/scg/hg38_scg.tsv at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$handleExecutionSuccess$1(StandardAsyncExecutionActor.scala:836) at scala.util.Success.$anonfun$map$1(Try.scala:251) at scala.util.Success.map(Try.scala:209) at scala.concurrent.Future.$anonfun$map$1(Future.scala:288) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

leepc12 commented 6 years ago

Please use an absolute path for genome TSV file.

Jin

On Fri, Jul 6, 2018 at 3:00 PM, mkdegorter notifications@github.com wrote:

Thanks Jin. I tried again with the following line in the .json file:

"atac.genome_tsv" : "genome/scg/hg38_scg.tsv",

The pipeline fails with this output to the screen:

[2018-07-06 12:13:48,53] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.trim_adapter:0:1]: executing: sbatch --export=ALL -J cromwell_579835bc_trim_adapter -D /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell- executions/atac/579835bc-3ce3-4489-bd44-250bf952540b/call-trim_adapter/shard-0

-o /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell- executions/atac/579835bc-3ce3-4489-bd44-250bf952540b/call- trim_adapter/shard-0/execution/stdout -e /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell- executions/atac/579835bc-3ce3-4489-bd44-250bf952540b/call- trim_adapter/shard-0/execution/stderr -t 1440 -n 1 --ntasks-per-node=1 --cpus-per-task=2 --mem=12000

--account smontgom --wrap "/bin/bash /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell- executions/atac/579835bc-3ce3-4489-bd44-250bf952540b/call- trim_adapter/shard-0/execution/script" [2018-07-06 12:13:48,75] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.trim_adapter:0:1]: job id: 2129106 [2018-07-06 12:13:48,75] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.read_genome_tsv:NA:1]: job id: 2129105 [2018-07-06 12:13:48,77] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.trim_adapter:0:1]: Status change from - to WaitingForReturnCodeFile [2018-07-06 12:13:48,77] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.read_genome_tsv:NA:1]: Status change from - to WaitingForReturnCodeFile [2018-07-06 12:14:26,69] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.read_genome_tsv:NA:1]: Status change from WaitingForReturnCodeFile to Done [2018-07-06 12:52:36,69] [info] DispatchedConfigAsyncJobExecutionActor [579835bcatac.trim_adapter:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-07-06 12:52:37,29] [error] WorkflowManagerActor Workflow 579835bc-3ce3-4489-bd44-250bf952540b failed (during ExecutingWorkflowState): cromwell.backend.standard. StandardAsyncExecutionActor$$anon$2: Failed to evaluate job outputs: Bad output 'read_genome_tsv.genome': java.nio.file.NoSuchFileException: /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell- executions/atac/579835bc-3ce3-4489-bd44-250bf952540b/call- read_genome_tsv/execution/genome/scg/hg38_scg.tsv at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$ handleExecutionSuccess$1(StandardAsyncExecutionActor.scala:836) at scala.util.Success.$anonfun$map$1(Try.scala:251) at scala.util.Success.map(Try.scala:209) at scala.concurrent.Future.$anonfun$map$1(Future.scala:288) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch( BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1( BatchingExecutor.scala:91) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) at akka.dispatch.BatchingExecutor$BlockableBatch.run( BatchingExecutor.scala:91) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec( ForkJoinExecutorConfigurator.scala:43) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue. runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run( ForkJoinWorkerThread.java:107)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kundajelab/atac_dnase_pipelines/issues/130#issuecomment-403157267, or mute the thread https://github.com/notifications/unsubscribe-auth/AIOd_CawxuR3XPTuw3Nf3JbDOTZu-2qxks5uD94PgaJpZM4VEc4R .

mkdegorter commented 6 years ago

Hi Jin,

That's what I had tried in my first post,

In the .json file I am using this line to specify the genome: "atac.genome_tsv" : "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/genome/scg/hg38_scg.tsv"

With the absolute path it fails a little further along the pipeline.

leepc12 commented 6 years ago

How further did it go? Please post a new log.

mkdegorter commented 6 years ago

With the full path it gets to the same step in the pipeline as in the first post. This is the entire output to the screen:

--wrap "/bin/bash /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-trim_adapter/shard-0/execution/script" [2018-07-09 15:14:28,55] [info] DispatchedConfigAsyncJobExecutionActor [449accb2atac.trim_adapter:0:1]: job id: 2131269 [2018-07-09 15:14:28,57] [info] DispatchedConfigAsyncJobExecutionActor [449accb2atac.trim_adapter:0:1]: Status change from - to WaitingForReturnCodeFile [2018-07-09 15:14:57,68] [info] DispatchedConfigAsyncJobExecutionActor [449accb2atac.read_genome_tsv:NA:1]: Status change from WaitingForReturnCodeFile to Done [2018-07-09 15:54:17,48] [info] DispatchedConfigAsyncJobExecutionActor [449accb2atac.trim_adapter:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-07-09 15:54:19,17] [info] WorkflowExecutionActor-449accb2-e994-40e1-912f-753255616799 [449accb2]: Starting atac.bowtie2 [2018-07-09 15:54:19,21] [warn] DispatchedConfigAsyncJobExecutionActor [449accb2atac.bowtie2:0:1]: Unrecognized runtime attribute keys: preemptible, disks [2018-07-09 15:54:19,40] [warn] Localization via hard link has failed: /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar -> /ifs/scratch/leepc12/pipeline_genome_data/hg38/bowtie2_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar: Invalid cross-device link [2018-07-09 15:54:19,42] [info] DispatchedConfigAsyncJobExecutionActor [449accb2atac.bowtie2:0:1]: python $(which encode_bowtie2.py) \ /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar \ /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inputs/1145178294/merge_fastqs_R1_SCS47_merge_R1.trim.merged.fastq.gz /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inputs/1145178294/merge_fastqs_R2_SCS47_merge_R2.trim.merged.fastq.gz \ --paired-end \ --multimapping 4 \ \ --nth 4 [2018-07-09 15:54:19,44] [info] DispatchedConfigAsyncJobExecutionActor [449accb2atac.bowtie2:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_449accb2_bowtie2 \ -D /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0 \ -o /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/execution/stdout \ -e /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/execution/stderr \ -t 2880 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=4 \ --mem=20000 \ \ --account smontgom \ --wrap "/bin/bash /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/execution/script" [2018-07-09 15:54:23,53] [info] DispatchedConfigAsyncJobExecutionActor [449accb2atac.bowtie2:0:1]: job id: 2131354 [2018-07-09 15:54:23,54] [info] DispatchedConfigAsyncJobExecutionActor [449accb2atac.bowtie2:0:1]: Status change from - to WaitingForReturnCodeFile [2018-07-09 15:55:41,41] [info] DispatchedConfigAsyncJobExecutionActor [449accb2atac.bowtie2:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-07-09 15:55:41,82] [error] WorkflowManagerActor Workflow 449accb2-e994-40e1-912f-753255616799 failed (during ExecutingWorkflowState): Job atac.bowtie2:0:1 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details. Check the content of stderr for potential additional information: /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/execution/stderr. Traceback (most recent call last): File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 224, in run_shell_cmd p.returncode, cmd) subprocess.CalledProcessError: Command 'tar xvf /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar -C .' returned non-zero exit status 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_bowtie2.py", line 207, in main() File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_bowtie2.py", line 151, in main untar(tar, args.out_dir) File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 113, in untar run_shell_cmd(cmd) File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 230, in run_shell_cmd os.killpg(pgid, signal.SIGKILL) ProcessLookupError: [Errno 3] No such process ln: failed to access ‘.bam’: No such file or directory ln: failed to access ‘.bai’: No such file or directory ln: failed to access ‘.align.log’: No such file or directory ln: failed to access ‘.flagstat.qc’: No such file or directory

[2018-07-09 15:55:41,83] [info] WorkflowManagerActor WorkflowActor-449accb2-e994-40e1-912f-753255616799 is in a terminal state: WorkflowFailedState [2018-07-09 15:55:49,39] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'. [2018-07-09 15:55:53,53] [info] Workflow polling stopped [2018-07-09 15:55:53,54] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds [2018-07-09 15:55:53,55] [info] Aborting all running workflows. [2018-07-09 15:55:53,57] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds [2018-07-09 15:55:53,57] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds [2018-07-09 15:55:53,57] [info] WorkflowStoreActor stopped [2018-07-09 15:55:53,57] [info] WorkflowLogCopyRouter stopped [2018-07-09 15:55:53,57] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds [2018-07-09 15:55:53,57] [info] JobExecutionTokenDispenser stopped [2018-07-09 15:55:53,57] [info] WorkflowManagerActor All workflows finished [2018-07-09 15:55:53,58] [info] Connection pools shut down [2018-07-09 15:55:53,58] [info] WorkflowManagerActor stopped [2018-07-09 15:55:53,58] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds [2018-07-09 15:55:53,58] [info] Shutting down JobStoreActor - Timeout = 1800 seconds [2018-07-09 15:55:53,58] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds [2018-07-09 15:55:53,58] [info] JobStoreActor stopped [2018-07-09 15:55:53,58] [info] SubWorkflowStoreActor stopped [2018-07-09 15:55:53,58] [info] CallCacheWriteActor Shutting down: 0 queued messages to process [2018-07-09 15:55:53,58] [info] CallCacheWriteActor stopped [2018-07-09 15:55:53,58] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds [2018-07-09 15:55:53,58] [info] Shutting down DockerHashActor - Timeout = 1800 seconds [2018-07-09 15:55:53,58] [info] Shutting down IoProxy - Timeout = 1800 seconds [2018-07-09 15:55:53,58] [info] DockerHashActor stopped [2018-07-09 15:55:53,58] [info] WriteMetadataActor Shutting down: 0 queued messages to process [2018-07-09 15:55:53,58] [info] KvWriteActor Shutting down: 0 queued messages to process [2018-07-09 15:55:53,58] [info] IoProxy stopped [2018-07-09 15:55:53,59] [info] ServiceRegistryActor stopped [2018-07-09 15:55:53,61] [info] Database closed [2018-07-09 15:55:53,61] [info] Stream materializer shut down Workflow 449accb2-e994-40e1-912f-753255616799 transitioned to state Failed [2018-07-09 15:55:53,68] [info] Automatic shutdown of the async connection [2018-07-09 15:55:53,68] [info] Gracefully shutdown sentry threads. [2018-07-09 15:55:53,69] [info] Shutdown finished

The call-bowtie2 folder is created, and in it, the stderr file reads:

Traceback (most recent call last): File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 224, in run_shell_cmd p.returncode, cmd) subprocess.CalledProcessError: Command 'tar xvf /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inp uts/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar -C .' returned non-zero exit status 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_bowtie2.py", line 207, in main() File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_bowtie2.py", line 151, in main untar(tar, args.out_dir) File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 113, in untar run_shell_cmd(cmd) File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 230, in run_shell_cmd os.killpg(pgid, signal.SIGKILL) ProcessLookupError: [Errno 3] No such process ln: failed to access ‘.bam’: No such file or directory ln: failed to access ‘.bai’: No such file or directory ln: failed to access ‘.align.log’: No such file or directory ln: failed to access ‘.flagstat.qc’: No such file or directory

And the stdout file reads:

Traceback (most recent call last): File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 224, in run_shell_cmd p.returncode, cmd) subprocess.CalledProcessError: Command 'tar xvf /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inp uts/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar -C .' returned non-zero exit status 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_bowtie2.py", line 207, in main() File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_bowtie2.py", line 151, in main untar(tar, args.out_dir) File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 113, in untar run_shell_cmd(cmd) File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 230, in run_shell_cmd os.killpg(pgid, signal.SIGKILL) ProcessLookupError: [Errno 3] No such process ln: failed to access ‘.bam’: No such file or directory ln: failed to access ‘.bai’: No such file or directory ln: failed to access ‘.align.log’: No such file or directory ln: failed to access ‘.flagstat.qc’: No such file or directory [mdegorte@hppsl230s-rcf-412-02-l execution]$ more stdout [2018-07-09 15:54:22,907 INFO] ['/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_bowtie2.py', '/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449a ccb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar', '/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/crom well-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inputs/1145178294/merge_fastqs_R1_SCS47_merge_R1.trim.merged.fastq.gz', '/labs/smontgom/mdegorte/ATAC/ atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inputs/1145178294/merge_fastqs_R2_SCS47_merge_R2.trim.merged.fastq.gz', '--paire d-end', '--multimapping', '4', '--nth', '4'] [2018-07-09 15:54:22,907 INFO] Initializing and making output directory... [2018-07-09 15:54:22,907 INFO] Generating read length file... [2018-07-09 15:54:47,489 INFO] Unpacking bowtie2 index tar... [2018-07-09 15:54:47,493 INFO] run_shell_cmd: PID=164718, CMD=tar xvf /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/ca ll-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar -C . PID=164718: tar: /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_a nalysis_set_GCA_000001405.15.fasta.tar: Cannot open: Permission denied PID=164718: tar: Error is not recoverable: exiting now [2018-07-09 15:54:47,666 ERROR] Unknown exception caught. Killing process group 164718... Traceback (most recent call last): File "/labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/src/encode_common.py", line 224, in run_shell_cmd p.returncode, cmd) subprocess.CalledProcessError: Command 'tar xvf /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inp uts/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar -C .' returned non-zero exit status 2

Could this be related to how Cromwell handles links? (https://github.com/ENCODE-DCC/atac-seq-pipeline/issues/18) I'm using Cromwell-33.1.

Thanks for your help with this!

Adrien-Evo commented 6 years ago

Those errors : 

ln: failed to access ‘.bam’: No such file or directory
ln: failed to access ‘.bai’: No such file or directory
ln: failed to access ‘.align.log’: No such file or directory
ln: failed to access ‘.flagstat.qc’: No such file or directory

are coming from the file script in your call-bowtie2 . There is no output files so the "ln -l" comands that want to link the ouputs in the glob directory are failing.

This is because this command fails : 

tar xvf /labs/smontgom/mdegorte/ATAC/atac-seq-pipeline/cromwell-executions/atac/449accb2-e994-40e1-912f-753255616799/call-bowtie2/shard-0/inputs/-1037641694/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar -C

because the fiel might not be there due to a link problem here :

leepc12 commented 6 years ago

There was a permission problem and it's fixed now. Can you check if you can see this file (/ifs/scratch/leepc12/pipeline_genome_data/hg38/bowtie2_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar) on SCG?

mkdegorter commented 6 years ago

Yes, I can see the file, and the call-bowtie2 step completed. Thank you for your help with this!