ENCODE-DCC / chip-seq-pipeline2

ENCODE ChIP-seq pipeline

MIT License

244 stars 123 forks source link

Bad output 'macs2.bfilt_npeak': Failed to find index Success(WomInteger(0)) on array #31

Closed mkmwong closed 6 years ago

mkmwong commented 6 years ago

I have run this pipeline with deduplicated bam as input and that worked. However, when I ran with fastq.gz as input, I encounter following error:

[2018-10-10 15:34:55,23] [info] Running with database db.url = jdbc:hsqldb:mem:ecd30a73-9553-4e46-8424-5f4227afb55b;shutdown=false;hsqldb.tx=mvcc [2018-10-10 15:35:05,14] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000 [2018-10-10 15:35:05,16] [info] [RenameWorkflowOptionsInMetadata] 100% [2018-10-10 15:35:05,26] [info] Running with database db.url = jdbc:hsqldb:mem:1170c126-8f24-448b-99cb-8c3254c2f7dc;shutdown=false;hsqldb.tx=mvcc [2018-10-10 15:35:05,66] [[38;5;220mwarn[0m] This actor factory is deprecated. Please use cromwell.backend.google.pipelines.v1alpha2.PipelinesApiLifecycleActorFactory for PAPI v1 or cromwell.backend.google.pipelines.v2alpha1.PipelinesApiLifecycleActorFactory for PAPI v2 [2018-10-10 15:35:05,66] [[38;5;220mwarn[0m] Couldn't find a suitable DSN, defaulting to a Noop one. [2018-10-10 15:35:05,67] [info] Using noop to send events. [2018-10-10 15:35:05,97] [info] Slf4jLogger started [2018-10-10 15:35:06,27] [info] Workflow heartbeat configuration: { "cromwellId" : "cromid-bac7c2f", "heartbeatInterval" : "2 minutes", "ttl" : "10 minutes", "writeBatchSize" : 10000, "writeThreshold" : 10000 } [2018-10-10 15:35:06,42] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds. [2018-10-10 15:35:06,45] [info] Metadata summary refreshing every 2 seconds. [2018-10-10 15:35:06,47] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds. [2018-10-10 15:35:06,52] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds. [2018-10-10 15:35:07,62] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds. [2018-10-10 15:35:07,66] [info] SingleWorkflowRunnerActor: Version 34 [2018-10-10 15:35:07,68] [info] SingleWorkflowRunnerActor: Submitting workflow [2018-10-10 15:35:07,72] [info] PAPIQueryManager Running with 3 workers [2018-10-10 15:35:07,72] [info] JES batch polling interval is 33333 milliseconds [2018-10-10 15:35:07,72] [info] JES batch polling interval is 33333 milliseconds [2018-10-10 15:35:07,73] [info] JES batch polling interval is 33333 milliseconds [2018-10-10 15:35:07,82] [info] Unspecified type (Unspecified version) workflow f99b0390-cc7a-43a0-a7fa-294cf7d512af submitted [2018-10-10 15:35:07,88] [info] SingleWorkflowRunnerActor: Workflow submitted [38;5;2mf99b0390-cc7a-43a0-a7fa-294cf7d512af[0m [2018-10-10 15:35:07,88] [info] 1 new workflows fetched [2018-10-10 15:35:07,88] [info] WorkflowManagerActor Starting workflow [38;5;2mf99b0390-cc7a-43a0-a7fa-294cf7d512af[0m [2018-10-10 15:35:07,90] [info] WorkflowManagerActor Successfully started WorkflowActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [2018-10-10 15:35:07,90] [info] Retrieved 1 workflows from the WorkflowStoreActor [2018-10-10 15:35:07,91] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes. [2018-10-10 15:35:07,92] [[38;5;220mwarn[0m] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData [2018-10-10 15:35:08,03] [info] MaterializeWorkflowDescriptorActor [[38;5;2mf99b0390[0m]: Parsing workflow as WDL draft-2 [2018-10-10 15:35:39,69] [info] MaterializeWorkflowDescriptorActor [[38;5;2mf99b0390[0m]: Call-to-Backend assignments: chip.overlap_ppr -> slurm, chip.bam2ta -> slurm, chip.qc_report -> slurm, chip.filter_ctl -> slurm, chip.idr_ppr -> slurm, chip.macs2_ppr2 -> slurm, chip.choose_ctl -> slurm, chip.fingerprint -> slurm, chip.xcor -> slurm, chip.spp_pooled -> slurm, chip.overlap_pr -> slurm, chip.spp_ppr1 -> slurm, chip.bam2ta_no_filt -> slurm, chip.macs2 -> slurm, chip.pool_ta_ctl -> slurm, chip.spp -> slurm, chip.merge_fastq_ctl -> slurm, chip.fraglen_mean -> slurm, chip.idr -> slurm, chip.pool_ta -> slurm, chip.overlap -> slurm, chip.read_genome_tsv -> slurm, chip.reproducibility_overlap -> slurm, chip.filter -> slurm, chip.macs2_pr1 -> slurm, chip.merge_fastq -> slurm, chip.idr_pr -> slurm, chip.spr -> slurm, chip.bam2ta_ctl -> slurm, chip.macs2_pr2 -> slurm, chip.macs2_ppr1 -> slurm, chip.macs2_pooled -> slurm, chip.pool_ta_pr2 -> slurm, chip.spp_pr1 -> slurm, chip.bwa -> slurm, chip.bam2ta_no_filt_R1 -> slurm, chip.spp_ppr2 -> slurm, chip.bwa_R1 -> slurm, chip.trim_fastq -> slurm, chip.bwa_ctl -> slurm, chip.pool_ta_pr1 -> slurm, chip.spp_pr2 -> slurm, chip.reproducibility_idr -> slurm [2018-10-10 15:35:39,84] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,85] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,85] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,85] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,85] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,85] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,85] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,85] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,85] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,85] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,85] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,86] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:39,87] [[38;5;220mwarn[0m] slurm [[38;5;2mf99b0390[0m]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-10-10 15:35:42,12] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.read_genome_tsv [2018-10-10 15:35:42,13] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition met: '!align_only && !true_rep_only'. Running conditional section [2018-10-10 15:35:42,13] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition met: '!true_rep_only'. Running conditional section [2018-10-10 15:35:42,74] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.read_genome_tsv:NA:1]: Unrecognized runtime attribute keys: disks [2018-10-10 15:35:43,20] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: 'enable_idr'. Bypassing conditional section [2018-10-10 15:35:43,22] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: 'peakcaller == "spp"'. Bypassing conditional section [2018-10-10 15:35:43,22] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition met: 'peakcaller == "macs2"'. Running conditional section [2018-10-10 15:35:43,22] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: '!align_only && !true_rep_only && enable_idr'. Bypassing conditional section [2018-10-10 15:35:43,22] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: 'enable_idr'. Bypassing conditional section [2018-10-10 15:35:43,22] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: 'peakcaller == "spp"'. Bypassing conditional section [2018-10-10 15:35:43,41] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.read_genome_tsv:NA:1]: [38;5;5mcat /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-read_genome_tsv/inputs/58336912/hg19.tsv[0m [2018-10-10 15:35:43,49] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.read_genome_tsv:NA:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_read_genome_tsv \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-read_genome_tsv \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-read_genome_tsv/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-read_genome_tsv/execution/stderr \ -t 60 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=1 \ --mem=4000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-read_genome_tsv/execution/script" [2018-10-10 15:35:46,30] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.merge_fastq_ctl, chip.merge_fastq [2018-10-10 15:35:46,47] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.read_genome_tsv:NA:1]: job id: 28445729 [2018-10-10 15:35:46,47] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.read_genome_tsv:NA:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 15:35:46,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq:0:1]: Unrecognized runtime attribute keys: disks [2018-10-10 15:35:46,65] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq_ctl:0:1]: Unrecognized runtime attribute keys: disks [2018-10-10 15:35:46,72] [[38;5;220mwarn[0m] Localization via hard link has failed: /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq_ctl/shard-0/inputs/661839578/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.fastq.gz -> /scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.fastq.gz: Invalid cross-device link [2018-10-10 15:35:46,72] [[38;5;220mwarn[0m] Localization via hard link has failed: /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq/shard-0/inputs/661839578/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.fastq.gz -> /scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.fastq.gz: Invalid cross-device link [2018-10-10 15:35:46,74] [[38;5;220mwarn[0m] Localization via hard link has failed: /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq/shard-0/inputs/661839578/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R2.fastq.gz -> /scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R2.fastq.gz: Invalid cross-device link [2018-10-10 15:35:46,74] [[38;5;220mwarn[0m] Localization via hard link has failed: /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq_ctl/shard-0/inputs/661839578/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R2.fastq.gz -> /scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R2.fastq.gz: Invalid cross-device link [2018-10-10 15:35:46,79] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq:0:1]: [38;5;5mpython $(which encode_merge_fastq.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq/shard-0/execution/write_tsv_82e931bc7d1a525bafe0ce1331b4b615.tmp \ --paired-end \ --nth 2[0m [2018-10-10 15:35:46,80] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq_ctl:0:1]: [38;5;5mpython $(which encode_merge_fastq.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq_ctl/shard-0/execution/write_tsv_5318eadf4f1e11ea3b195ca44b819fac.tmp \ --paired-end \ --nth 2[0m [2018-10-10 15:35:46,81] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_merge_fastq \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq/shard-0/execution/stderr \ -t 360 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=2 \ --mem=12000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq/shard-0/execution/script" [2018-10-10 15:35:46,83] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq_ctl:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_merge_fastq_ctl \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq_ctl/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq_ctl/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq_ctl/shard-0/execution/stderr \ -t 360 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=2 \ --mem=12000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-merge_fastq_ctl/shard-0/execution/script" [2018-10-10 15:35:51,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq:0:1]: job id: 28445734 [2018-10-10 15:35:51,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq_ctl:0:1]: job id: 28445732 [2018-10-10 15:35:51,47] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 15:35:51,47] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq_ctl:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 15:37:11,58] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 15:37:17,02] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.trim_fastq [2018-10-10 15:37:17,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.trim_fastq:0:1]: Unrecognized runtime attribute keys: disks [2018-10-10 15:37:17,69] [[38;5;220mwarn[0m] Localization via hard link has failed: /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-trim_fastq/shard-0/inputs/-567840873/merge_fastqs_R1_SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.fastq.gz -> /scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.fastq.gz: Invalid cross-device link [2018-10-10 15:37:17,70] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.trim_fastq:0:1]: [38;5;5mpython $(which encode_trim_fastq.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-trim_fastq/shard-0/inputs/-567840873/merge_fastqs_R1_SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.fastq.gz \ --trim-bp 50[0m [2018-10-10 15:37:17,74] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.trim_fastq:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_trim_fastq \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-trim_fastq/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-trim_fastq/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-trim_fastq/shard-0/execution/stderr \ -t 60 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=1 \ --mem=8000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-trim_fastq/shard-0/execution/script" [2018-10-10 15:37:21,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.trim_fastq:0:1]: job id: 28445811 [2018-10-10 15:37:21,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.trim_fastq:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 15:37:21,71] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.merge_fastq_ctl:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 15:37:42,59] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.read_genome_tsv:NA:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 15:37:45,57] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.bwa, chip.bwa_ctl [2018-10-10 15:37:45,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa:0:1]: Unrecognized runtime attribute keys: preemptible, disks [2018-10-10 15:37:45,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_ctl:0:1]: Unrecognized runtime attribute keys: preemptible, disks [2018-10-10 15:37:45,72] [[38;5;220mwarn[0m] Localization via hard link has failed: /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_ctl/shard-0/inputs/44672371/merge_fastqs_R1_SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.merged.fastq.gz -> /scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.fastq.gz: Invalid cross-device link [2018-10-10 15:37:45,73] [[38;5;220mwarn[0m] Localization via hard link has failed: /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa/shard-0/inputs/-567840873/merge_fastqs_R1_SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.fastq.gz -> /scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.fastq.gz: Invalid cross-device link [2018-10-10 15:37:45,74] [[38;5;220mwarn[0m] Localization via hard link has failed: /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_ctl/shard-0/inputs/44672371/merge_fastqs_R2_SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R2.merged.fastq.gz -> /scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R2.fastq.gz: Invalid cross-device link [2018-10-10 15:37:45,74] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_ctl:0:1]: [38;5;5mpython $(which encode_bwa.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_ctl/shard-0/inputs/-1093316416/male.hg19.fa.tar \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_ctl/shard-0/inputs/44672371/merge_fastqs_R1_SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.merged.fastq.gz /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_ctl/shard-0/inputs/44672371/merge_fastqs_R2_SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R2.merged.fastq.gz \ --paired-end \ --nth 4[0m [2018-10-10 15:37:45,75] [[38;5;220mwarn[0m] Localization via hard link has failed: /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa/shard-0/inputs/-567840873/merge_fastqs_R2_SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R2.merged.fastq.gz -> /scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R2.fastq.gz: Invalid cross-device link [2018-10-10 15:37:45,76] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa:0:1]: [38;5;5mpython $(which encode_bwa.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa/shard-0/inputs/-1093316416/male.hg19.fa.tar \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa/shard-0/inputs/-567840873/merge_fastqs_R1_SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.fastq.gz /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa/shard-0/inputs/-567840873/merge_fastqs_R2_SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R2.merged.fastq.gz \ --paired-end \ --nth 4[0m [2018-10-10 15:37:45,78] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_ctl:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_bwa_ctl \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_ctl/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_ctl/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_ctl/shard-0/execution/stderr \ -t 2880 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=4 \ --mem=20000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_ctl/shard-0/execution/script" [2018-10-10 15:37:45,79] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_bwa \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa/shard-0/execution/stderr \ -t 2880 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=4 \ --mem=20000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa/shard-0/execution/script" [2018-10-10 15:37:51,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa:0:1]: job id: 28445834 [2018-10-10 15:37:51,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_ctl:0:1]: job id: 28445836 [2018-10-10 15:37:51,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_ctl:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 15:37:51,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 15:49:55,18] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.trim_fastq:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 15:50:00,71] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.bwa_R1 [2018-10-10 15:50:01,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_R1:0:1]: Unrecognized runtime attribute keys: preemptible, disks [2018-10-10 15:50:01,74] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_R1:0:1]: [38;5;5mpython $(which encode_bwa.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_R1/shard-0/inputs/-1093316416/male.hg19.fa.tar \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_R1/shard-0/inputs/913187384/merge_fastqs_R1_SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.trim_50bp.fastq.gz \ \ --nth 4[0m [2018-10-10 15:50:01,77] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_R1:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_bwa_R1 \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_R1/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_R1/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_R1/shard-0/execution/stderr \ -t 2880 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=4 \ --mem=20000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bwa_R1/shard-0/execution/script" [2018-10-10 15:50:06,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_R1:0:1]: job id: 28447299 [2018-10-10 15:50:06,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_R1:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 16:34:37,99] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_R1:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 16:34:40,06] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.bam2ta_no_filt_R1 [2018-10-10 16:34:40,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt_R1:0:1]: Unrecognized runtime attribute keys: disks [2018-10-10 16:34:40,71] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt_R1:0:1]: [38;5;5mpython $(which encode_bam2ta.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_no_filt_R1/shard-0/inputs/-55055628/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.trim_50bp.bam \ \ --disable-tn5-shift \ --regex-grep-v-ta 'chrM' \ --subsample 0 \[0m [2018-10-10 16:34:40,75] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt_R1:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_bam2ta_no_filt_R1 \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_no_filt_R1/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_no_filt_R1/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_no_filt_R1/shard-0/execution/stderr \ -t 360 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=2 \ --mem=10000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_no_filt_R1/shard-0/execution/script" [2018-10-10 16:34:41,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt_R1:0:1]: job id: 28451098 [2018-10-10 16:34:41,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt_R1:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 16:40:22,98] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt_R1:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 20:41:30,97] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 20:41:36,17] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.filter, chip.bam2ta_no_filt [2018-10-10 20:41:36,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt:0:1]: Unrecognized runtime attribute keys: disks [2018-10-10 20:41:36,69] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter:0:1]: Unrecognized runtime attribute keys: disks [2018-10-10 20:41:36,75] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter:0:1]: [38;5;5mpython $(which encode_filter.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-filter/shard-0/inputs/1119499152/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.bam \ --paired-end \ \ --dup-marker picard \ --mapq-thresh 30 \ \

ugly part to deal with optional outputs with Google JES backend

touch null[0m [2018-10-10 20:41:36,76] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt:0:1]: [38;5;5mpython $(which encode_bam2ta.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_no_filt/shard-0/inputs/1119499152/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.bam \ --paired-end \ --disable-tn5-shift \ --regex-grep-v-ta 'chrM' \ --subsample 0 \[0m [2018-10-10 20:41:36,79] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_bam2ta_no_filt \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_no_filt/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_no_filt/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_no_filt/shard-0/execution/stderr \ -t 360 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=2 \ --mem=10000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_no_filt/shard-0/execution/script" [2018-10-10 20:41:36,81] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_filter \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-filter/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-filter/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-filter/shard-0/execution/stderr \ -t 1440 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=2 \ --mem=20000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-filter/shard-0/execution/script" [2018-10-10 20:41:41,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt:0:1]: job id: 28469190 [2018-10-10 20:41:41,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter:0:1]: job id: 28469189 [2018-10-10 20:41:41,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 20:41:41,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 21:26:40,43] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_no_filt:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 22:34:18,23] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 22:34:23,63] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.bam2ta [2018-10-10 22:34:24,62] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta:0:1]: Unrecognized runtime attribute keys: disks [2018-10-10 22:34:24,70] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta:0:1]: [38;5;5mpython $(which encode_bam2ta.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta/shard-0/inputs/-1132106790/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.nodup.bam \ --paired-end \ --disable-tn5-shift \ --regex-grep-v-ta 'chrM' \ --subsample 0 \[0m [2018-10-10 22:34:24,73] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_bam2ta \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta/shard-0/execution/stderr \ -t 360 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=2 \ --mem=10000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta/shard-0/execution/script" [2018-10-10 22:34:26,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta:0:1]: job id: 28473079 [2018-10-10 22:34:26,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 23:00:31,64] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 23:00:34,78] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: '!true_reponly && length(tas) > 1 && peakcaller == "macs2"'. Bypassing conditional section [2018-10-10 23:00:34,78] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: '!true_reponly && length(tas) > 1'. Bypassing conditional section [2018-10-10 23:00:34,78] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: 'length(tas_) > 1 && peakcaller == "spp"'. Bypassing conditional section [2018-10-10 23:00:34,78] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: 'length(tas_) > 1'. Bypassing conditional section [2018-10-10 23:00:34,78] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: '!true_reponly && length(tas) > 1 && peakcaller == "spp"'. Bypassing conditional section [2018-10-10 23:00:34,78] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: '!true_reponly && length(tas) > 1 && peakcaller == "spp"'. Bypassing conditional section [2018-10-10 23:00:34,79] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: '!true_reponly && length(tas) > 1 && peakcaller == "macs2"'. Bypassing conditional section [2018-10-10 23:00:34,79] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: 'length(tas_) > 1'. Bypassing conditional section [2018-10-10 23:00:36,84] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.xcor, chip.spr [2018-10-10 23:00:37,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.spr:0:1]: Unrecognized runtime attribute keys: disks [2018-10-10 23:00:37,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.xcor:0:1]: Unrecognized runtime attribute keys: disks [2018-10-10 23:00:37,73] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.spr:0:1]: [38;5;5mpython $(which encode_spr.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-spr/shard-0/inputs/-922975244/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.nodup.tagAlign.gz \ --paired-end[0m [2018-10-10 23:00:37,73] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.xcor:0:1]: [38;5;5mpython $(which encode_xcor.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-xcor/shard-0/inputs/-155087896/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.trim_50bp.tagAlign.gz \ \ --subsample 15000000 \ --nth 2[0m [2018-10-10 23:00:37,76] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.spr:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_spr \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-spr/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-spr/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-spr/shard-0/execution/stderr \ -t 60 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=1 \ --mem=16000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-spr/shard-0/execution/script" [2018-10-10 23:00:37,77] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.xcor:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_xcor \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-xcor/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-xcor/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-xcor/shard-0/execution/stderr \ -t 360 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=2 \ --mem=16000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-xcor/shard-0/execution/script" [2018-10-10 23:00:41,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.xcor:0:1]: job id: 28475398 [2018-10-10 23:00:41,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.spr:0:1]: job id: 28475397 [2018-10-10 23:00:41,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.xcor:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 23:00:41,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.spr:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 23:06:01,11] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.spr:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 23:20:11,72] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.xcor:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-10 23:20:15,40] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.fraglen_mean [2018-10-10 23:20:15,63] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.fraglen_mean:NA:1]: Unrecognized runtime attribute keys: disks [2018-10-10 23:20:15,66] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.fraglenmean:NA:1]: [38;5;5mpython <<CODE arr = [-5] if len(arr): sum = sum(arr) mean = sum(arr)/float(len(arr)) print(int(round(mean))) else: print(0) CODE[0m [2018-10-10 23:20:15,69] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.fraglen_mean:NA:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_fraglen_mean \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-fraglen_mean \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-fraglen_mean/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-fraglen_mean/execution/stderr \ -t 60 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=1 \ --mem=4000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-fraglen_mean/execution/script" [2018-10-10 23:20:21,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.fraglen_mean:NA:1]: job id: 28476124 [2018-10-10 23:20:21,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.fraglen_mean:NA:1]: Status change from - to WaitingForReturnCodeFile [2018-10-10 23:22:22,12] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.fraglen_mean:NA:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-11 00:22:07,64] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bwa_ctl:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-11 00:22:13,31] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.filter_ctl [2018-10-11 00:22:13,63] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter_ctl:0:1]: Unrecognized runtime attribute keys: disks [2018-10-11 00:22:13,71] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter_ctl:0:1]: [38;5;5mpython $(which encode_filter.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-filter_ctl/shard-0/inputs/42388076/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.merged.bam \ --paired-end \ \ --dup-marker picard \ --mapq-thresh 30 \ \

ugly part to deal with optional outputs with Google JES backend

touch null[0m [2018-10-11 00:22:13,75] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter_ctl:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_filter_ctl \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-filter_ctl/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-filter_ctl/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-filter_ctl/shard-0/execution/stderr \ -t 1440 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=2 \ --mem=20000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-filter_ctl/shard-0/execution/script" [2018-10-11 00:22:16,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter_ctl:0:1]: job id: 28479287 [2018-10-11 00:22:16,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter_ctl:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-11 03:16:26,86] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.filter_ctl:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-11 03:16:30,91] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition NOT met: '!no_jsd && length(nodupbams) > 0 && length(ctl_nodupbams) > 0 && basename(blacklist) != "null"'. Bypassing conditional section [2018-10-11 03:16:32,95] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.bam2ta_ctl [2018-10-11 03:16:33,62] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_ctl:0:1]: Unrecognized runtime attribute keys: disks [2018-10-11 03:16:33,70] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_ctl:0:1]: [38;5;5mpython $(which encode_bam2ta.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_ctl/shard-0/inputs/-1207382602/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.merged.nodup.bam \ --paired-end \ --disable-tn5-shift \ --regex-grep-v-ta 'chrM' \ --subsample 0 \[0m [2018-10-11 03:16:33,73] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_ctl:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_bam2ta_ctl \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_ctl/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_ctl/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_ctl/shard-0/execution/stderr \ -t 360 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=2 \ --mem=10000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-bam2ta_ctl/shard-0/execution/script" [2018-10-11 03:16:36,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_ctl:0:1]: job id: 28483144 [2018-10-11 03:16:36,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2ta_ctl:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-11 03:47:39,16] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.bam2tactl:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-11 03:47:42,63] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition met: 'length(tas) > 0 && length(ctltas) > 0'. Running conditional section [2018-10-11 03:47:42,63] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Condition met: 'length(ctltas) > 0'. Running conditional section [2018-10-11 03:47:44,66] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.pool_ta_ctl [2018-10-11 03:47:45,62] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.pool_ta_ctl:NA:1]: Unrecognized runtime attribute keys: disks [2018-10-11 03:47:45,68] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.pool_ta_ctl:NA:1]: [38;5;5mpython $(which encode_pool_ta.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-pool_ta_ctl/inputs/1500631376/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.merged.nodup.tagAlign.gz[0m [2018-10-11 03:47:45,71] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.pool_ta_ctl:NA:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_pool_ta_ctl \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-pool_ta_ctl \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-pool_ta_ctl/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-pool_ta_ctl/execution/stderr \ -t 60 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=1 \ --mem=4000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-pool_ta_ctl/execution/script" [2018-10-11 03:47:46,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.pool_ta_ctl:NA:1]: job id: 28485165 [2018-10-11 03:47:46,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.pool_ta_ctl:NA:1]: Status change from - to WaitingForReturnCodeFile [2018-10-11 03:50:16,49] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.pool_ta_ctl:NA:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-11 03:50:18,63] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.choose_ctl [2018-10-11 03:50:19,62] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.choose_ctl:NA:1]: Unrecognized runtime attribute keys: disks [2018-10-11 03:50:19,74] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.choose_ctl:NA:1]: [38;5;5mpython $(which encode_choose_ctl.py) \ --tas /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-choose_ctl/inputs/-922975244/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.nodup.tagAlign.gz \ --ctl-tas /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-choose_ctl/inputs/1500631376/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.merged.nodup.tagAlign.gz \ \ --ctl-ta-pooled /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-choose_ctl/inputs/311943167/SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.merged.nodup.tagAlign.gz \ --always-use-pooled-ctl \ --ctl-depth-ratio 1.2[0m [2018-10-11 03:50:19,78] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.choose_ctl:NA:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_choose_ctl \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-choose_ctl \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-choose_ctl/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-choose_ctl/execution/stderr \ -t 60 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=1 \ --mem=8000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-choose_ctl/execution/script" [2018-10-11 03:50:21,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.choose_ctl:NA:1]: job id: 28485274 [2018-10-11 03:50:21,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.choose_ctl:NA:1]: Status change from - to WaitingForReturnCodeFile [2018-10-11 03:52:47,47] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.choose_ctl:NA:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-11 03:52:51,57] [info] WorkflowExecutionActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af [[38;5;2mf99b0390[0m]: Starting chip.macs2, chip.macs2_pr2, chip.macs2_pr1 [2018-10-11 03:52:51,63] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2:0:1]: Unrecognized runtime attribute keys: disks [2018-10-11 03:52:51,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr2:0:1]: Unrecognized runtime attribute keys: disks [2018-10-11 03:52:51,64] [[38;5;220mwarn[0m] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr1:0:1]: Unrecognized runtime attribute keys: disks [2018-10-11 03:52:52,44] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2:0:1]: [38;5;5mpython $(which encode_macs2_chip.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2/shard-0/inputs/-922975244/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.nodup.tagAlign.gz /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2/shard-0/inputs/-1900987009/ctl_for_rep1.tagAlign.gz \ --gensz hs \ --chrsz /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2/shard-0/inputs/58336912/hg19.chrom.sizes \ --fraglen -5 \ --cap-num-peak 500000 \ --pval-thresh 0.01 \ --make-signal \ --blacklist /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2/shard-0/inputs/58336912/wgEncodeDacMapabilityConsensusExcludable.bed.gz

touch null # ugly part to deal with optional outputs[0m [2018-10-11 03:52:52,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr2:0:1]: [38;5;5mpython $(which encode_macs2_chip.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr2/shard-0/inputs/-303265875/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.nodup.pr2.tagAlign.gz /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr2/shard-0/inputs/-1900987009/ctl_for_rep1.tagAlign.gz \ --gensz hs \ --chrsz /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr2/shard-0/inputs/58336912/hg19.chrom.sizes \ --fraglen -5 \ --cap-num-peak 500000 \ --pval-thresh 0.01 \ \ --blacklist /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr2/shard-0/inputs/58336912/wgEncodeDacMapabilityConsensusExcludable.bed.gz

touch null.pval.signal.bigwig null.fc.signal.bigwig touch null # ugly part to deal with optional outputs[0m [2018-10-11 03:52:52,47] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr1:0:1]: [38;5;5mpython $(which encode_macs2_chip.py) \ /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr1/shard-0/inputs/1660355428/SCGPM_RbWTK9K27_HTYFT_L7_TAGCTT_R1.merged.nodup.pr1.tagAlign.gz /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr1/shard-0/inputs/-1900987009/ctl_for_rep1.tagAlign.gz \ --gensz hs \ --chrsz /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr1/shard-0/inputs/58336912/hg19.chrom.sizes \ --fraglen -5 \ --cap-num-peak 500000 \ --pval-thresh 0.01 \ \ --blacklist /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr1/shard-0/inputs/58336912/wgEncodeDacMapabilityConsensusExcludable.bed.gz

touch null.pval.signal.bigwig null.fc.signal.bigwig touch null # ugly part to deal with optional outputs[0m [2018-10-11 03:52:52,48] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_macs2 \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2/shard-0/execution/stderr \ -t 1440 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=1 \ --mem=16000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2/shard-0/execution/script" [2018-10-11 03:52:52,50] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr2:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_macs2_pr2 \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr2/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr2/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr2/shard-0/execution/stderr \ -t 1440 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=1 \ --mem=16000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr2/shard-0/execution/script" [2018-10-11 03:52:52,52] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr1:0:1]: executing: sbatch \ --export=ALL \ -J cromwell_f99b0390_macs2_pr1 \ -D /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr1/shard-0 \ -o /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr1/shard-0/execution/stdout \ -e /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr1/shard-0/execution/stderr \ -t 1440 \ -n 1 \ --ntasks-per-node=1 \ --cpus-per-task=1 \ --mem=16000 \ \ \ \ \ --wrap "/bin/bash /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/f99b0390-cc7a-43a0-a7fa-294cf7d512af/call-macs2_pr1/shard-0/execution/script" [2018-10-11 03:52:56,45] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr1:0:1]: job id: 28485366 [2018-10-11 03:52:56,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr2:0:1]: job id: 28485368 [2018-10-11 03:52:56,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2:0:1]: job id: 28485367 [2018-10-11 03:52:56,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr1:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-11 03:52:56,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr2:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-11 03:52:56,46] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2:0:1]: Status change from - to WaitingForReturnCodeFile [2018-10-11 03:54:05,40] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr1:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-11 03:54:19,71] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-11 03:54:30,01] [info] DispatchedConfigAsyncJobExecutionActor [[38;5;2mf99b0390[0mchip.macs2_pr2:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-10-11 03:54:30,47] [[38;5;1merror[0m] WorkflowManagerActor Workflow f99b0390-cc7a-43a0-a7fa-294cf7d512af failed (during ExecutingWorkflowState): cromwell.backend.standard.StandardAsyncExecutionActor$$anon$2: Failed to evaluate job outputs: Bad output 'macs2_pr1.npeak': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2_pr1.bfilt_npeak': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2_pr1.bfilt_npeak_bb': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2_pr1.frip_qc': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$handleExecutionSuccess$1(StandardAsyncExecutionActor.scala:839) at scala.util.Success.$anonfun$map$1(Try.scala:251) at scala.util.Success.map(Try.scala:209) at scala.concurrent.Future.$anonfun$map$1(Future.scala:288) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

cromwell.backend.standard.StandardAsyncExecutionActor$$anon$2: Failed to evaluate job outputs: Bad output 'macs2.npeak': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2.bfilt_npeak': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2.bfilt_npeak_bb': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2.sig_pval': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2.sig_fc': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2.frip_qc': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$handleExecutionSuccess$1(StandardAsyncExecutionActor.scala:839) at scala.util.Success.$anonfun$map$1(Try.scala:251) at scala.util.Success.map(Try.scala:209) at scala.concurrent.Future.$anonfun$map$1(Future.scala:288) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

cromwell.backend.standard.StandardAsyncExecutionActor$$anon$2: Failed to evaluate job outputs: Bad output 'macs2_pr2.npeak': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2_pr2.bfilt_npeak': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2_pr2.bfilt_npeak_bb': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 Bad output 'macs2_pr2.frip_qc': Failed to find index Success(WomInteger(0)) on array:

Success([])

0 at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$handleExecutionSuccess$1(StandardAsyncExecutionActor.scala:839) at scala.util.Success.$anonfun$map$1(Try.scala:251) at scala.util.Success.map(Try.scala:209) at scala.concurrent.Future.$anonfun$map$1(Future.scala:288) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[2018-10-11 03:54:30,48] [info] WorkflowManagerActor WorkflowActor-f99b0390-cc7a-43a0-a7fa-294cf7d512af is in a terminal state: WorkflowFailedState [2018-10-11 03:54:32,98] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'. [2018-10-11 03:54:36,49] [info] Workflow polling stopped [2018-10-11 03:54:36,50] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds [2018-10-11 03:54:36,51] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds [2018-10-11 03:54:36,51] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds [2018-10-11 03:54:36,51] [info] JobExecutionTokenDispenser stopped [2018-10-11 03:54:36,51] [info] Aborting all running workflows. [2018-10-11 03:54:36,51] [info] WorkflowStoreActor stopped [2018-10-11 03:54:36,52] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds [2018-10-11 03:54:36,52] [info] WorkflowLogCopyRouter stopped [2018-10-11 03:54:36,52] [info] Connection pools shut down [2018-10-11 03:54:36,52] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds [2018-10-11 03:54:36,52] [info] Shutting down JobStoreActor - Timeout = 1800 seconds [2018-10-11 03:54:36,52] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds [2018-10-11 03:54:36,52] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds [2018-10-11 03:54:36,52] [info] Shutting down DockerHashActor - Timeout = 1800 seconds [2018-10-11 03:54:36,52] [info] Shutting down IoProxy - Timeout = 1800 seconds [2018-10-11 03:54:36,52] [info] WorkflowManagerActor stopped [2018-10-11 03:54:36,52] [info] DockerHashActor stopped [2018-10-11 03:54:36,52] [info] IoProxy stopped [2018-10-11 03:54:36,52] [info] CallCacheWriteActor Shutting down: 0 queued messages to process [2018-10-11 03:54:36,52] [info] SubWorkflowStoreActor stopped [2018-10-11 03:54:36,52] [info] CallCacheWriteActor stopped [2018-10-11 03:54:36,52] [info] WorkflowManagerActor All workflows finished [2018-10-11 03:54:36,52] [info] JobStoreActor stopped [2018-10-11 03:54:36,53] [info] WriteMetadataActor Shutting down: 0 queued messages to process [2018-10-11 03:54:36,53] [info] KvWriteActor Shutting down: 0 queued messages to process [2018-10-11 03:54:36,54] [info] ServiceRegistryActor stopped [2018-10-11 03:54:36,54] [info] Database closed [2018-10-11 03:54:36,54] [info] Stream materializer shut down Workflow f99b0390-cc7a-43a0-a7fa-294cf7d512af transitioned to state Failed [2018-10-11 03:54:36,57] [info] Automatic shutdown of the async connection [2018-10-11 03:54:36,57] [info] Gracefully shutdown sentry threads. [2018-10-11 03:54:36,57] [info] Shutdown finished.

And here's the path after I activated conda environment: /home/groups/ashbym/mandy/anaconda3/envs/encode-chip-seq-pipeline/bin:/home/users/mkmwong/perl5/bin:/share/software/user/open/perl/5.26.0/bin:/home/groups/ashbym/mandy/anaconda3/bin:/share/software/user/srcc/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/iSM/bin:/home/groups/ashbym/mandy/.bds:/home/groups/ashbym/mandy/anaconda3/bin/samtools:/home/groups/ashbym/mandy/bowtie2-2.3.4.3-linux-x86_64:/home/groups/ashbym/mandy/bedtools2/bin:/home/groups/ashbym/mandy/sratoolkit.2.9.2-centos_linux64/bin:/home/users/mkmwong/bin:/home/users/mkmwong/localperl/bin

Thank you !

leepc12 commented 6 years ago

Thanks for reporting errors. Please make a tar ball of logs. You should be able to find the instruction for it when you create an issue.

mkmwong commented 6 years ago

Hi,

I think i found something - I had the same problem before when I ran the old version of pipeline. This is the stderr from call-macs2 folder:

[2018-10-11 01:09:02,147 INFO] ['/home/groups/ashbym/mandy/anaconda3/envs/encode-chip-seq-pipeline/bin/encode_macs2_chip.py', '/home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/a04b663b-6769-41ed-8f95-97c153ba38cb/call-macs2/shard-0/inputs/-209809900/SCGPM_RbWTK9K27_HTYFT_L7_TGACCA_R1.merged.nodup.tagAlign.gz', '/home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/a04b663b-6769-41ed-8f95-97c153ba38cb/call-macs2/shard-0/inputs/-686991457/ctl_for_rep1.tagAlign.gz', '--gensz', 'hs', '--chrsz', '/home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/a04b663b-6769-41ed-8f95-97c153ba38cb/call-macs2/shard-0/inputs/58336912/hg19.chrom.sizes', '--fraglen', '-5', '--cap-num-peak', '500000', '--pval-thresh', '0.01', '--make-signal', '--blacklist', '/home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/a04b663b-6769-41ed-8f95-97c153ba38cb/call-macs2/shard-0/inputs/58336912/wgEncodeDacMapabilityConsensusExcludable.bed.gz'] [2018-10-11 01:09:02,148 INFO] Initializing and making output directory... [2018-10-11 01:09:02,148 INFO] Calling peaks and generating signal tracks with MACS2... [2018-10-11 01:09:02,161 INFO] run_shell_cmd: PID=23840, CMD= macs2 callpeak -t /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/a04b663b-6769-41ed-8f95-97c153ba38cb/call-macs2/shard-0/inputs/-209809900/SCGPM_RbWTK9K27_HTYFT_L7_TGACCA_R1.merged.nodup.tagAlign.gz -c /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/a04b663b-6769-41ed-8f95-97c153ba38cb/call-macs2/shard-0/inputs/-686991457/ctl_for_rep1.tagAlign.gz -f BED -n SCGPM_RbWTK9K27_HTYFT_L7_TGACCA_R1.merged.nodup_x_ctl_for_rep1 -g hs -p 0.01 --nomodel --shift 0 --extsize -5 --keep-dup all -B --SPMR PID=23840: ERROR:root:--extsize must >= 1! [2018-10-11 01:09:04,219 ERROR] Unknown exception caught. Killing process group 23840... Traceback (most recent call last): File "/home/groups/ashbym/mandy/anaconda3/envs/encode-chip-seq-pipeline/bin/encode_common.py", line 224, in run_shell_cmd p.returncode, cmd) CalledProcessError: Command ' macs2 callpeak -t /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/a04b663b-6769-41ed-8f95-97c153ba38cb/call-macs2/shard-0/inputs/-209809900/SCGPM_RbWTK9K27_HTYFT_L7_TGACCA_R1.merged.nodup.tagAlign.gz -c /home/groups/ashbym/mandy/chip-seq-pipeline2/cromwell-executions/chip/a04b663b-6769-41ed-8f95-97c153ba38cb/call-macs2/shard-0/inputs/-686991457/ctl_for_rep1.tagAlign.gz -f BED -n SCGPM_RbWTK9K27_HTYFT_L7_TGACCA_R1.merged.nodup_x_ctl_for_rep1 -g hs -p 0.01 --nomodel --shift 0 --extsize -5 --keep-dup all -B --SPMR' returned non-zero exit status 1

Is there anywhere we can manually change --extsize?

leepc12 commented 6 years ago

No, --extsize is determined by a fragment length calculated from the cross-correlation analysis. Please make a tar ball and attach it. I think this has something to do with the quality of your data.

akundaje commented 6 years ago

Jin,

It will be good to have an option to over-ride ext-size. Because if the cross-correlation estimation of fragment length fails as in this case, the user should be allowed to set an --ext-size (and the --shift should be set accordingly).

-Anshul.

On Thu, Oct 11, 2018 at 1:54 PM Jin Lee notifications@github.com wrote:

No, --extsize is determined by a fragment length calculated from the cross-correlation analysis. Please make a tar ball and attach it. I think this has something to do with the quality of your data.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ENCODE-DCC/chip-seq-pipeline2/issues/31#issuecomment-429114966, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI7Edp_-ACCpTfCYAQn4AZXdgp-gxM2ks5uj6_2gaJpZM4XYDor .

leepc12 commented 6 years ago

@akundaje: Sure, we will add it in the next release.

leepc12 commented 6 years ago

@akundaje: oh.. I forgot that it's already implemented in this pipeline.

@mkmwong: to manually specify --extsize (fraglen) add chip.fraglen to your input JSON. It's an integer array and the length of it should match with the number of replicates.

      "chip.fraglen" : [100,100],

mkmwong commented 6 years ago

@akundaje: oh.. I forgot that it's already implemented in this pipeline.

@mkmwong: to manually specify --extsize (fraglen) add chip.fraglen to your input JSON. It's an integer array and the length of it should match with the number of replicates.
      "chip.fraglen" : [100,100],

Thank you Jin! One extra question, what would be a reasonable estimate for --ext-size? Would the average size of fragment be a good approximate for this?

akundaje commented 6 years ago

100-150 should be work well.

On Thu, Oct 11, 2018, 6:05 PM Mandy Wong notifications@github.com wrote:

@akundaje https://github.com/akundaje: oh.. I forgot that it's already implemented in this pipeline.

@mkmwong https://github.com/mkmwong: to manually specify --extsize (fraglen) add chip.fraglen to your input JSON. It's an integer array and the length of it should match with the number of replicates.
  "chip.fraglen" : [100,100],
Thank you Jin! One extra question, what would be a reasonable estimate for --ext-size? Would the average size of fragment be a good approximate for this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ENCODE-DCC/chip-seq-pipeline2/issues/31#issuecomment-429170381, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI7EZ5arLxcWpFwR83VorOOZAxi8p7Zks5uj-rygaJpZM4XYDor .

mkmwong commented 6 years ago

Hi,

I am not sure why after I used the fraglen setting, I encountered the following problem very early on in the pipeline: subprocess.CalledProcessError: Command 'samtools view -Su SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.merged.sam | samtools sort - SCGPM_RbWTK9K27_HTYFT_L7_CTTGTA_R1.merged' returned non-zero exit s. I attached the tar ball here. Thanks! debug_31.tar.gz

Upon closer examination I've found a typo. I'm going to rerun the pipeline and see if thats the cause.

leepc12 commented 6 years ago

It's weird. Adding chip.fraglen to the input JSON does not affect early steps of the pipeline. Did you change something else?

mkmwong commented 6 years ago

I don't think I changed anything else. I thought it was weird too. I rerun the pipeline after I corrected the typo and now it failed at a different step. I attached the tarball here. Please let me know if you have any leads on what's going on. Thanks! debug_31_new.tar.gz

mkmwong commented 6 years ago

Hi,

The closest I could find was this. where it suggested that the command was used to sort the sam file and covert it into a bam file is malformed.

2593 [2018-10-12 15:12:35,343 INFO] run_shell_cmd: PID=17948, CMD=samtools view -Su SCGPM_RbWTK9K27_HTYFT_L7_GATCAG_R1.merged.sam | samtools sort - SCGPM_R bWTK9K27_HTYFT_L7_GATCAG_R1.merged 2594 PID=17948: [bam_sort] Use -T PREFIX / -o FILE to specify temporary and final output files 2595 PID=17948: Usage: samtools sort [options...] [in.bam] 2596 PID=17948: Options: 2597 PID=17948: -l INT Set compression level, from 0 (uncompressed) to 9 (best) 2598 PID=17948: -m INT Set maximum memory per thread; suffix K/M/G recognized [768M] 2599 PID=17948: -n Sort by read name 2600 PID=17948: -t TAG Sort by value of TAG. Uses position as secondary index (or read name if -n is set) 2601 PID=17948: -o FILE Write final output to FILE rather than standard output 2602 PID=17948: -T PREFIX Write temporary files to PREFIX.nnnn.bam 2603 PID=17948: --input-fmt-option OPT[=VAL] 2604 PID=17948: Specify a single input file format option in the form 2605 PID=17948: of OPTION or OPTION=VALUE 2606 PID=17948: -O, --output-fmt FORMAT[,OPT[=VAL]]... 2607 PID=17948: Specify output format (SAM, BAM, CRAM) 2608 PID=17948: --output-fmt-option OPT[=VAL] 2609 PID=17948: Specify a single output file format option in the form 2610 PID=17948: of OPTION or OPTION=VALUE 2611 PID=17948: --reference FILE 2612 PID=17948: Reference sequence FASTA FILE [null] 2613 PID=17948: -@, --threads INT 2614 PID=17948: Number of additional threads to use [0] 2615 [2018-10-12 15:12:35,796 ERROR] Unknown exception caught. Killing process group 17948... 2616 Traceback (most recent call last): 2617 File "/home/groups/ashbym/mandy/anaconda3/envs/encode-chip-seq-pipeline/bin/encode_common.py", line 224, in run_shell_cmd 2618 p.returncode, cmd) 2619 subprocess.CalledProcessError: Command 'samtools view -Su SCGPM_RbWTK9K27_HTYFT_L7_GATCAG_R1.merged.sam | samtools sort - SCGPM_RbWTK9K27_HTYFT_L7_GAT CAG_R1.merged' returned non-zero exit status 1.

However, when I run the command directly, it runs just fine. Any idea on why that is? Thanks!

leepc12 commented 6 years ago

Can you run again without chip.fraglen?

mkmwong commented 6 years ago

Can you run again without chip.fraglen?

Got the exact same error as the original post

leepc12 commented 6 years ago

It's very hard to figure out. Can you try with docker or singularity?

mkmwong commented 6 years ago

It's very hard to figure out. Can you try with docker or singularity?

I will try. The original error is from how the fraglen is estimated incorrectly. I just have one question about using chip.fraglen. Here's what's in my input file:

{ "chip.pipeline_type" : "histone", "chip.genome_tsv" : "/home/groups/ashbym/mandy/chip-seq-pipeline2/genome/hg19.tsv", "chip.fastqs_rep1_R1" : [ "/scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_TTAGGC_R1.fastq.gz" ], "chip.fastqs_rep1_R2" : [ "/scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_TTAGGC_R2.fastq.gz" ], "chip.fastqs_rep2_R1" : [ "/scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_TGACCA_R1.fastq.gz" ], "chip.fastqs_rep2_R2" : [ "/scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_TGACCA_R2.fastq.gz" ], "chip.ctl_fastqs_rep1_R1" : [ "/scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_ACAGTG_R1.fastq.gz" ], "chip.ctl_fastqs_rep1_R2" : [ "/scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_ACAGTG_R2.fastq.gz" ], "chip.ctl_fastqs_rep2_R1" : [ "/scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_GCCAAT_R1.fastq.gz" ], "chip.ctl_fastqs_rep2_R2" : [ "/scratch/users/mkmwong/RbKO_WT_ChIP/allFastq/original/SCGPM_RbWTK9K27_HTYFT_L7_GCCAAT_R2.fastq.gz" ], "chip.paired_end" : true, "chip.no_jsd" : true, "chip.choose_ctl.always_use_pooled_ctl" : true, "chip.qc_report.name" : "WT_H3K27me3", "chip.qc_report.desc" : "H3K27me3 ChIP-Seq on WT IMR-90" }

when I use chip.fraglen, do I only use an array of length of 2? or do I need to use 8 because I have 8 files in total?

leepc12 commented 6 years ago

You have to use an array of length 2 [rep1, rep2]. You don't need fraglen for controls.