spp. chip.call_peak RetryableFailure

[yatianyang2019]

Describe the bug Hi, i was running the pipeline for TF with error. And the error was pasted in the bottom.

.... [Caper] troubleshooting 49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74 ... Found failures: [ { "causedBy": [ { "message": "Job chip.call_peak_pooled:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] }, { "message": "Job chip.call_peak:1:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] }, { "message": "Job chip.call_peak_pr2:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] }, { "message": "Job chip.call_peak:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] }, { "message": "Job chip.call_peak_ppr2:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] } ], "message": "Workflow failed" } ]

chip.call_peak RetryableFailure. SHARD_IDX=0, RC=1, JOB_ID=20508, RUN_START=2020-02-28T19:20:16.079Z, RUN_END=2020-02-28T23:39:13.988Z, STDOUT=/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/execution/stdout, STDERR=/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/execution/stderr STDERR_CONTENTS= Traceback (most recent call last): File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in main() File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 91, in main args.fraglen, args.cap_num_peak, args.nth, args.out_dir) File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 66, in spp run_shell_cmd(cmd0) File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd raise Exception(err_str) Exception: PID=16, PGID=16, RC=1 STDERR=Loading required package: caTools There were 24 warnings (use warnings() to see them) Error in data.frame(..., check.names = FALSE) : arguments imply differing number of rows: 1, 0 Calls: find.binding.positions ... calculate.enrichment.estimates -> lapply -> FUN -> cbind -> cbind -> data.frame In addition: There were 50 or more warnings (use warnings() to see the first 50) Execution halted STDOUT=################ ChIP data: /cromwell-executions/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/inputs/-623737054/reverb_SR8278.merged.nodup.tagAlign.gz Control data: /cromwell-executions/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/inputs/812910509/ctl_for_rep1.tagAlign.gz :

---

• OS/Platform:Ubuntu 18.01
• Pipeline version: 1.3.3
• Caper version: 0.6.1

Caper configuration file backend=local tmp-dir=temp

Input JSON file "chip.title" : "reverb", "chip.description" : "reverb chip seq",

"chip.pipeline_type" : "tf",
"chip.aligner" : "bwa",
"chip.align_only" : false,
"chip.true_rep_only" : false,

"chip.genome_tsv" : "/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/hg38_caper.tsv",

"chip.paired_end" : false,
"chip.ctl_paired_end" : false,

"chip.always_use_pooled_ctl" : false,

"chip.fastqs_rep1_R1" : [ "/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/reverb_SR8278.fq.gz" ],
"chip.fastqs_rep2_R1" : [ "/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/reverb_Vehicle.fq.gz" ],

"chip.ctl_fastqs_rep1_R1" : [ "/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/Input_SR8278.fq.gz" ],
"chip.ctl_fastqs_rep2_R1" : [ "/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/Input_Vehicle.fq.gz" ],

"chip.align_cpu": 16,
"chip.align_mem_mb": 32768,

"chip.filter_cpu": 4,
"chip.filter_mem_mb": 24576,

"chip.bam2ta_cpu": 2,
"chip.bam2ta_mem_mb": 24576,

"chip.spr_mem_mb": 24576,

"chip.jsd_cpu": 4,
"chip.jsd_mem_mb": 24576,

"chip.xcor_cpu": 4,
"chip.xcor_mem_mb": 24576,

"chip.call_peak_cpu": 4,
"chip.call_peak_mem_mb": 24576,

"chip.macs2_signal_track_mem_mb": 24576

}

Error log [Caper] troubleshooting 49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74 ... Found failures: [ { "causedBy": [ { "message": "Job chip.call_peak_pooled:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] }, { "message": "Job chip.call_peak:1:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] }, { "message": "Job chip.call_peak_pr2:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] }, { "message": "Job chip.call_peak:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] }, { "message": "Job chip.call_peak_ppr2:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] } ], "message": "Workflow failed" } ]

chip.call_peak RetryableFailure. SHARD_IDX=0, RC=1, JOB_ID=20508, RUN_START=2020-02-28T19:20:16.079Z, RUN_END=2020-02-28T23:39:13.988Z, STDOUT=/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_Eva nsLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/execution/stdout, STDERR=/home/hwclab/NGS_Data/reverb-project/201904_DCai_R EV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/execution/stderr STDERR_CONTENTS= Traceback (most recent call last): File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in main() File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 91, in main args.fraglen, args.cap_num_peak, args.nth, args.out_dir) File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 66, in spp run_shell_cmd(cmd0) File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd raise Exception(err_str) Exception: PID=16, PGID=16, RC=1 STDERR=Loading required package: caTools There were 24 warnings (use warnings() to see them) Error in data.frame(..., check.names = FALSE) : arguments imply differing number of rows: 1, 0 Calls: find.binding.positions ... calculate.enrichment.estimates -> lapply -> FUN -> cbind -> cbind -> data.frame In addition: There were 50 or more warnings (use warnings() to see the first 50) Execution halted STDOUT=################ ChIP data: /cromwell-executions/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/inputs/-623737054/reverb_SR8278.merged.nodup.tagAlign.gz Control data: /cromwell-executions/chip/49bc38e3-3a8d-4c65-8d22-5c4dc2f37f74/call-call_peak/shard-0/inputs/812910509/ctl_for_rep1.tagAlign.gz :

[leepc12]

What is your pipeline version?

[yatianyang2019]

1.3.3

[leepc12]

Did you use Conda? Then check spp version in pipeline's Conda env. I think this is already fixed in v1.3.6. What is your r-spp version?

$ conda activate encode-chip-seq-pipeline
$ conda list r-spp

[yatianyang2019]

I was using the docker image. I will give a try on the v1.3.6. Thanks!

[yatianyang2019]

Hi, I was using the docker on the V1.3.6. But it still have same error as below: ....... chip.call_peak_pooled RetryableFailure. SHARD_IDX=-1, RC=1, JOB_ID=11888, RUN_START=2020-03-03T02:24:04.408Z, RUN_END=2020-03-03T07:07:19.256Z, STDOUT=/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker_1.3.6/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/execution/stdout, STDERR=/home/hwclab/NGS_Data/reverb-project/201904_DCai_REV-ERBa_EvansLba_abs_SR8278_7.5_ENZ_20_24hrs-ing/Clean/caper_output_docker_1.3.6/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/execution/stderr STDERR_CONTENTS= Traceback (most recent call last): File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 109, in main() File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 95, in main args.nth, args.out_dir) File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 69, in spp run_shell_cmd(cmd0) File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd raise Exception(err_str) Exception: PID=16, PGID=16, RC=1 STDERR=Loading required package: caTools There were 24 warnings (use warnings() to see them) Error in data.frame(..., check.names = FALSE) : arguments imply differing number of rows: 1, 0 Calls: find.binding.positions ... calculate.enrichment.estimates -> lapply -> FUN -> cbind -> cbind -> data.frame In addition: There were 50 or more warnings (use warnings() to see the first 50) Execution halted STDOUT=################ ChIP data: /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/inputs/1096230463/basename_prefix.pooled.tagAlign.gz Control data: /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/inputs/-832788197/basename_prefix.pooled.tagAlign.gz strandshift(min): -500 strandshift(step): 5 strandshift(max) 1500 user-defined peak shift 148 exclusion(min): 10 exclusion(max): NaN num parallel nodes: NA FDR threshold: 0.01 NumPeaks Threshold: 3e+05 Output Directory: /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/execution narrowPeak output file name: NA regionPeak output file name: basename_prefix.pooled_x_basename_prefix.pooled.300K.regionPeak.gz.tmp Rdata filename: NA plot pdf filename: NA result filename: NA Overwrite files?: TRUE

Decompressing ChIP file Decompressing control file Reading ChIP tagAlign/BAM file /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/inputs/1096230463/basename_prefix.pooled.tagAlign.gz opened /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/tmp.7e7cdd0d/Rtmpo99IrN/basename_prefix.pooled.tagAlign1256eec687 done. read 40532677 fragments ChIP data read length 50 [1] TRUE Reading Control tagAlign/BAM file /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/inputs/-832788197/basename_prefix.pooled.tagAlign.gz opened /cromwell-executions/chip/17049ef2-f215-4dd5-b0db-d40d6b240544/call-call_peak_pooled/tmp.7e7cdd0d/Rtmpo99IrN/basename_prefix.pooled.tagAlign1251b8815d done. read 36009580 fragments Control data read length 50 Calculating peak characteristics Minimum cross-correlation value 0.365511 Minimum cross-correlation shift 1500 Top 3 cross-correlation values 0.431022635744594 Top 3 estimates for fragment length 148 Window half size 455 Phantom peak location 55 Phantom peak Correlation 0.4120462 Normalized Strand cross-correlation coefficient (NSC) 1.179233 Relative Strand cross-correlation Coefficient (RSC) 1.407788 Phantom Peak Quality Tag 1 Removing read stacks Finding peaks finding background exclusion regions ... done determining peaks on provided 1 control datasets: using reversed signal for FDR calculations ....

[leepc12]

I was able to reproduce the problem. I will look into it and let you know.

[yatianyang2019]

thanks, I tried to change to use reference genome hg19.tsv on the docker v 1.3.6. It run successfully. But there is still the same error with hg38. So I have no idea about this issue.

[snewberry]

I have run into this issue as well.

I removed everything except chr1-22, X, Y, and M from the rep bams and that seemed to fix the issue. This isn't ideal as I had to edit the bams and then re-run from the bams on; however, perhaps the underlying issue has something to do with calling on the contigs.