shirleytemples
commented
2 years ago The issue is solved and pipeline ran successfully after I changed the json file "chip.ctl_paired_ends" from true to false.
"chip.ctl_paired_ends" : [false]
Open shirleytemples opened 2 years ago
shirleytemples
commented
2 years ago The issue is solved and pipeline ran successfully after I changed the json file "chip.ctl_paired_ends" from true to false.
"chip.ctl_paired_ends" : [false]
shirleytemples
commented
2 years ago Even though I was able to run the all-CTCF.bam file (tf) from above, however, I am having a similar issue with the tf file all-POLR2A.bam
I ran caper run chip.wdl -i /mnt/c/users/shirl/Desktop/guttman_lab/POLR2A.json --conda
I made the same change in the JSON file for all-POLR2A.bam that helped me to get the tf file all-CTCF.bam to run:
{
"chip.title" : "Running all-POLR2A.bam through ENCODE",
"chip.description" : "This is an template input JSON for POLR2A.",
"chip.pipeline_type" : "tf",
"chip.nodup_bams" : ["/mnt/c/Users/shirl/Desktop/guttman_lab/all-POLR2A.bam"],
"chip.ctl_nodup_bams" : ["/mnt/c/Users/shirl/Desktop/guttman_lab/Control_ChIP-seq_hg38_K562.bam"],
"chip.genome_tsv" : "https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v4/hg38.tsv",
"chip.enable_gc_bias" : false,
"chip.paired_end" : false,
"chip.ctl_paired_ends" : [false]
}However, I still got the error message:
2022-03-07 04:08:01,451|caper.cli|INFO| Cromwell stdout: /home/shirleytemples/chip-seq-pipeline2/cromwell.out.3
2022-03-07 04:08:01,454|caper.caper_base|INFO| Creating a timestamped temporary directory. /home/shirleytemples/chip-seq-pipeline2/.caper_tmp/chip/20220307_040801_453090
2022-03-07 04:08:01,454|caper.caper_runner|INFO| Localizing files on work_dir. /home/shirleytemples/chip-seq-pipeline2/.caper_tmp/chip/20220307_040801_453090
2022-03-07 04:08:01,700|autouri.autouri|INFO| cp: (1e4ed2c7) started. src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/caf534ed3cf684406e731d19be272b4a/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz
2022-03-07 04:08:03,362|autouri.autouri|INFO| cp: (1e4ed2c7) skipped due to name_size_match. size=872949833, mt=1549739698.0
2022-03-07 04:08:03,363|autouri.autouri|INFO| cp: (dbdf3dbd) started. src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only.fasta.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/f43b63a83784d3ec8055f1a22168ed89/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only.fasta.gz
2022-03-07 04:08:03,987|autouri.autouri|INFO| cp: (dbdf3dbd) skipped due to md5_match. md5=05297d96dd1f7cfb45a7b637d6dd7036
2022-03-07 04:08:03,988|autouri.autouri|INFO| cp: (771e91e5) started. src=https://www.encodeproject.org/files/ENCFF356LFX/@@download/ENCFF356LFX.bed.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/f183dcba5d34f959d8b55ed438ee2e22/ENCFF356LFX.bed.gz
2022-03-07 04:08:04,723|autouri.autouri|INFO| cp: (771e91e5) skipped due to md5_match. md5=393688b4f06c9ce26165d47433dd8c37
2022-03-07 04:08:05,326|autouri.autouri|INFO| cp: (5798a8ee) started. src=https://www.encodeproject.org/files/GRCh38_EBV.chrom.sizes/@@download/GRCh38_EBV.chrom.sizes.tsv, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/c52f52c7bfa357f55a39b1de7e4d0b0c/GRCh38_EBV.chrom.sizes.tsv
2022-03-07 04:08:05,809|autouri.autouri|INFO| cp: (5798a8ee) skipped due to md5_match. md5=c95303fb77cc3e11d50e3c3a4b93b3fb
2022-03-07 04:08:05,811|autouri.autouri|INFO| cp: (81a35cb0) started. src=https://www.encodeproject.org/files/ENCFF110MCL/@@download/ENCFF110MCL.tar.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/3ff4ac4c3f59d096b1a3842a182072ae/ENCFF110MCL.tar.gz
2022-03-07 04:08:07,433|autouri.autouri|INFO| cp: (81a35cb0) skipped due to name_size_match. size=3749246230, mt=1571469011.0
2022-03-07 04:08:07,435|autouri.autouri|INFO| cp: (808b9fa1) started. src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index.tar.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/df5193e07055d13c48be59bacd0f56b8/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index.tar.gz
2022-03-07 04:08:08,282|autouri.autouri|INFO| cp: (808b9fa1) skipped due to md5_match. md5=80b263f6ea6ff65d547eef07102535db
2022-03-07 04:08:08,283|autouri.autouri|INFO| cp: (1d3e41ad) started. src=https://www.encodeproject.org/files/ENCFF643CGH/@@download/ENCFF643CGH.tar.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/8c692fba4640609720272154ab0faa30/ENCFF643CGH.tar.gz
2022-03-07 04:08:09,778|autouri.autouri|INFO| cp: (1d3e41ad) skipped due to name_size_match. size=4318261891, mt=1549723866.0
2022-03-07 04:08:09,780|autouri.autouri|INFO| cp: (a4d03689) started. src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bwa_index/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bwa_index.tar.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/d3dff25534e93d893902540d81e4f475/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bwa_index.tar.gz
2022-03-07 04:08:10,452|autouri.autouri|INFO| cp: (a4d03689) skipped due to md5_match. md5=7e088c24a017a43b1db5e8f50060eec1
2022-03-07 04:08:10,454|autouri.autouri|INFO| cp: (94e2b8f6) started. src=https://www.encodeproject.org/files/ENCFF766FGL/@@download/ENCFF766FGL.bed.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/f16cc7892540d7a9bc3207cc44eb8288/ENCFF766FGL.bed.gz
2022-03-07 04:08:11,212|autouri.autouri|INFO| cp: (94e2b8f6) skipped due to md5_match. md5=d108aa2a392ce5dd207fb8769a6e8240
2022-03-07 04:08:11,214|autouri.autouri|INFO| cp: (616b519e) started. src=https://www.encodeproject.org/files/ENCFF304XEX/@@download/ENCFF304XEX.bed.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/805e179275a9c0fb7a37def40c4312d1/ENCFF304XEX.bed.gz
2022-03-07 04:08:12,578|autouri.autouri|INFO| cp: (616b519e) skipped due to name_size_match. size=14377496, mt=1592463730.0
2022-03-07 04:08:12,579|autouri.autouri|INFO| cp: (afefbc93) started. src=https://www.encodeproject.org/files/ENCFF140XLU/@@download/ENCFF140XLU.bed.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/0cbd2c602ddad252bc39729fc8a29286/ENCFF140XLU.bed.gz
2022-03-07 04:08:13,316|autouri.autouri|INFO| cp: (afefbc93) skipped due to md5_match. md5=91047588129069ff91ec1b0664179f8e
2022-03-07 04:08:13,318|autouri.autouri|INFO| cp: (c81021fa) started. src=https://www.encodeproject.org/files/ENCFF212UAV/@@download/ENCFF212UAV.bed.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/1d3aa436b05f16a509edb94789c061d3/ENCFF212UAV.bed.gz
2022-03-07 04:08:14,933|autouri.autouri|INFO| cp: (c81021fa) skipped due to name_size_match. size=18381891, mt=1592463727.0
2022-03-07 04:08:14,935|autouri.autouri|INFO| cp: (a00d24dc) started. src=https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_formatted.txt.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/3b39284516e676ea52238f0636c0bbbf/hg38_dnase_avg_fseq_signal_formatted.txt.gz
2022-03-07 04:08:15,341|autouri.autouri|INFO| cp: (a00d24dc) skipped due to md5_match. md5=df624401f76fbd4d651e736068c43a1a
2022-03-07 04:08:15,343|autouri.autouri|INFO| cp: (18128f68) started. src=https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_celltype_compare_subsample.bed.gz, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/c73f434c3fa4f3f54bc2ecad09c065c2/hg38_celltype_compare_subsample.bed.gz
2022-03-07 04:08:15,617|autouri.autouri|INFO| cp: (18128f68) skipped due to md5_match. md5=ced0c653d28628654288f7a8ab052590
2022-03-07 04:08:15,618|autouri.autouri|INFO| cp: (e061ae42) started. src=https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_metadata.txt, dest=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/a9745b33b4ffdd83d7d2c5a7d3c8036a/hg38_dnase_avg_fseq_signal_metadata.txt
2022-03-07 04:08:15,872|autouri.autouri|INFO| cp: (e061ae42) skipped due to md5_match. md5=3f7fd85ab9a4c6274f28c3e82a79c10d
2022-03-07 04:08:16,515|caper.wdl_parser|ERROR| Failed to parse WDL with miniwdl.
2022-03-07 04:08:16,516|caper.caper_workflow_opts|INFO| Conda environment name not found in WDL metadata. wdl=/home/shirleytemples/chip-seq-pipeline2/chip.wdl
2022-03-07 04:08:16,526|caper.cromwell|INFO| Validating WDL/inputs/imports with Womtool...
2022-03-07 04:08:20,148|caper.cromwell|INFO| Passed Womtool validation.
2022-03-07 04:08:20,151|caper.caper_runner|INFO| launching run: wdl=/home/shirleytemples/chip-seq-pipeline2/chip.wdl, inputs=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/mnt/c/users/shirl/Desktop/guttman_lab/POLR2A.local.json, backend_conf=/home/shirleytemples/chip-seq-pipeline2/.caper_tmp/chip/20220307_040801_453090/backend.conf
2022-03-07 04:08:30,793|caper.cromwell_workflow_monitor|INFO| Workflow: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, status=Submitted
2022-03-07 04:08:30,838|caper.cromwell_workflow_monitor|INFO| Workflow: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, status=Running
2022-03-07 04:08:45,581|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.read_genome_tsv:-1, retry=0, status=Started, job_id=16849
2022-03-07 04:08:45,585|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.read_genome_tsv:-1, retry=0, status=WaitingForReturnCode
2022-03-07 04:08:47,507|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.read_genome_tsv:-1, retry=0, status=Done
2022-03-07 04:09:00,575|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.bam2ta_ctl:0, retry=0, status=Started, job_id=16915
2022-03-07 04:09:00,577|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.bam2ta:0, retry=0, status=Started, job_id=16913
2022-03-07 04:09:00,578|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.bam2ta:0, retry=0, status=WaitingForReturnCode
2022-03-07 04:09:00,578|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.bam2ta_ctl:0, retry=0, status=WaitingForReturnCode
2022-03-07 04:09:05,570|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.jsd:-1, retry=0, status=Started, job_id=17009
2022-03-07 04:09:05,571|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.jsd:-1, retry=0, status=WaitingForReturnCode
2022-03-07 04:09:10,300|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.bam2ta:0, retry=0, status=Done
2022-03-07 04:09:20,579|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.spr:0, retry=0, status=Started, job_id=17075
2022-03-07 04:09:20,581|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.spr:0, retry=0, status=WaitingForReturnCode
2022-03-07 04:09:25,567|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.xcor:0, retry=0, status=Started, job_id=17125
2022-03-07 04:09:25,568|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.xcor:0, retry=0, status=WaitingForReturnCode
2022-03-07 04:09:31,938|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.spr:0, retry=0, status=Done
2022-03-07 04:10:01,799|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.bam2ta_ctl:0, retry=0, status=Done
2022-03-07 04:10:15,568|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.choose_ctl:-1, retry=0, status=Started, job_id=17268
2022-03-07 04:10:15,569|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.choose_ctl:-1, retry=0, status=WaitingForReturnCode
2022-03-07 04:10:26,088|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.choose_ctl:-1, retry=0, status=Done
2022-03-07 04:10:54,423|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.xcor:0, retry=0, status=Done
2022-03-07 04:11:05,585|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak_pr1:0, retry=0, status=Started, job_id=17419
2022-03-07 04:11:05,586|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.fraglen_mean:-1, retry=0, status=Started, job_id=17376
2022-03-07 04:11:05,587|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak_pr2:0, retry=0, status=Started, job_id=17467
2022-03-07 04:11:05,593|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak_pr2:0, retry=0, status=WaitingForReturnCode
2022-03-07 04:11:05,597|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.fraglen_mean:-1, retry=0, status=WaitingForReturnCode
2022-03-07 04:11:05,607|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak_pr1:0, retry=0, status=WaitingForReturnCode
2022-03-07 04:11:07,806|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.fraglen_mean:-1, retry=0, status=Done
2022-03-07 04:11:10,613|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak:0, retry=0, status=Started, job_id=17520
2022-03-07 04:11:10,615|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.macs2_signal_track:0, retry=0, status=Started, job_id=17567
2022-03-07 04:11:10,616|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.macs2_signal_track:0, retry=0, status=WaitingForReturnCode
2022-03-07 04:11:10,617|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak:0, retry=0, status=WaitingForReturnCode
2022-03-07 04:18:50,862|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.jsd:-1, retry=0, status=Done
2022-03-07 07:12:41,978|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.macs2_signal_track:0, retry=0, status=Done
Exception in thread Thread-5:
2022-03-07 11:08:12,864|caper.cromwell_workflow_monitor|INFO| Workflow: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, status=Failed
Traceback (most recent call last):
File "/home/shirleytemples/miniconda3/lib/python3.8/threading.py", line 932, in _bootstrap_inner
2022-03-07 11:55:28,395|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak_pr2:0, retry=0, status=Done
2022-03-07 11:55:28,398|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak_pr2:0, retry=1, status=Started, job_id=18629
2022-03-07 11:55:28,401|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak_pr2:0, retry=1, status=WaitingForReturnCode
2022-03-07 11:55:28,401|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak_pr1:0, retry=0, status=Done
2022-03-07 11:55:28,401|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak:0, retry=0, status=Done
2022-03-07 11:55:28,401|caper.cromwell_workflow_monitor|INFO| Task: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, task=chip.call_peak_pr2:0, retry=1, status=Done
self.run()
File "/home/shirleytemples/miniconda3/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/shirleytemples/miniconda3/lib/python3.8/site-packages/caper/nb_subproc_thread.py", line 186, in read_from_stdout_obj
for line in iter(stdout.readline, b''):
ValueError: readline of closed file
2022-03-07 11:55:28,438|caper.cromwell_metadata|INFO| Wrote metadata file. /home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/metadata.json
2022-03-07 11:55:28,438|caper.cromwell|INFO| Workflow failed. Auto-troubleshooting...
* Started troubleshooting workflow: id=c2323035-49ad-4622-8ba9-c1e5687bb0d8, status=Failed
* Found failures JSON object.
[
{
"causedBy": [
{
"causedBy": [],
"message": "Job chip.call_peak_pr2:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."
}
],
"message": "Workflow failed"
}
]
* Recursively finding failures in calls (tasks)...
==== NAME=chip.call_peak_pr2, STATUS=RetryableFailure, PARENT=
SHARD_IDX=0, RC=1, JOB_ID=17467
START=2022-03-07T12:11:04.711Z, END=2022-03-07T17:03:05.623Z
STDOUT=/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/stdout
STDERR=/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_task_spp.py", line 133, in <module>
main()
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_task_spp.py", line 123, in main
'No peaks found. FDR threshold (fdr_thresh in your input JSON) '
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_lib_common.py", line 218, in assert_file_not_empty
raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (all-POLR2A.pr2_x_Control_ChIP-seq_hg38_K562.300K.regionPeak.gz). Help: No peaks found. FDR threshold (fdr_thresh in your input JSON) might be too stringent or poor quality sample?
STDERR_BACKGROUND_CONTENTS=
Traceback (most recent call last):
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_task_spp.py", line 133, in <module>
main()
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_task_spp.py", line 123, in main
'No peaks found. FDR threshold (fdr_thresh in your input JSON) '
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_lib_common.py", line 218, in assert_file_not_empty
raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (all-POLR2A.pr2_x_Control_ChIP-seq_hg38_K562.300K.regionPeak.gz). Help: No peaks found. FDR threshold (fdr_thresh in your input JSON) might be too stringent or poor quality sample?
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/*.num_peak.qc': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/*.peak_region_size.qc': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/*.peak_region_size.png': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/*.frip.qc': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/*.bfilt.regionPeak.hammock.gz*': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/*.bfilt.regionPeak.bb': No such file or directory
mkdir: cannot create directory ‘/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/glob-c71d4cca8627aab27dc3503b7db7a39d’: File exists
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/*.bfilt.regionPeak.hammock.gz*': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/*.bfilt.regionPeak.gz': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/execution/*.bfilt.regionPeak.starch': No such file or directory
==== NAME=chip.call_peak_pr2, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=1, JOB_ID=18629
START=2022-03-07T17:03:06.760Z, END=2022-03-07T19:07:40.511Z
STDOUT=/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/stdout
STDERR=/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_task_spp.py", line 133, in <module>
main()
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_task_spp.py", line 123, in main
'No peaks found. FDR threshold (fdr_thresh in your input JSON) '
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_lib_common.py", line 218, in assert_file_not_empty
raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (all-POLR2A.pr2_x_Control_ChIP-seq_hg38_K562.300K.regionPeak.gz). Help: No peaks found. FDR threshold (fdr_thresh in your input JSON) might be too stringent or poor quality sample?
STDERR_BACKGROUND_CONTENTS=
Traceback (most recent call last):
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_task_spp.py", line 133, in <module>
main()
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_task_spp.py", line 123, in main
'No peaks found. FDR threshold (fdr_thresh in your input JSON) '
File "/home/shirleytemples/miniconda3/envs/encode-chip-seq-pipeline-spp/bin/encode_lib_common.py", line 218, in assert_file_not_empty
raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (all-POLR2A.pr2_x_Control_ChIP-seq_hg38_K562.300K.regionPeak.gz). Help: No peaks found. FDR threshold (fdr_thresh in your input JSON) might be too stringent or poor quality sample?
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/*.num_peak.qc': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/*.peak_region_size.qc': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/*.peak_region_size.png': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/*.frip.qc': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/*.bfilt.regionPeak.hammock.gz*': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/*.bfilt.regionPeak.bb': No such file or directory
mkdir: cannot create directory ‘/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/glob-c71d4cca8627aab27dc3503b7db7a39d’: File exists
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/*.bfilt.regionPeak.hammock.gz*': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/*.bfilt.regionPeak.gz': No such file or directory
ln: failed to access '/home/shirleytemples/chip-seq-pipeline2/chip/c2323035-49ad-4622-8ba9-c1e5687bb0d8/call-call_peak_pr2/shard-0/attempt-2/execution/*.bfilt.regionPeak.starch': No such file or directory
2022-03-07 11:55:28,515|caper.nb_subproc_thread|ERROR| Cromwell failed. returncode=1
2022-03-07 11:55:28,519|caper.cli|ERROR| Check stdout in /home/shirleytemples/chip-seq-pipeline2/cromwell.out.3I am not sure how to further debug this, any suggestions would be helpful!
leepc12
commented
2 years ago How did you align your FASTQs into BAM files (all-CTCF.bam and the control BAM)?
shirleytemples
commented
2 years ago For all-CTCF.bam, the FASTQs were aligned using bowtie2 and I got the control BAM from [ENCFF178EJY] (https://www.encodeproject.org/files/ENCFF178EJY/). The same for all-POLR2A.bam and its control bam.
leepc12
commented
2 years ago What reference genome files did you use when you align FASTQs? They should match with files (e.g. FASTA, chromsizes, ...) in the TSV file https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v4/hg38.tsv defined in your input JSON.
$ curl https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v4/hg38.tsv
genome_name hg38
ref_fa https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz
ref_mito_fa https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only.fasta.gz
mito_chr_name chrM
blacklist https://www.encodeproject.org/files/ENCFF356LFX/@@download/ENCFF356LFX.bed.gz
chrsz https://www.encodeproject.org/files/GRCh38_EBV.chrom.sizes/@@download/GRCh38_EBV.chrom.sizes.tsv
gensz hs
bowtie2_idx_tar https://www.encodeproject.org/files/ENCFF110MCL/@@download/ENCFF110MCL.tar.gz
bowtie2_mito_idx_tar https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index.tar.gz
bwa_idx_tar https://www.encodeproject.org/files/ENCFF643CGH/@@download/ENCFF643CGH.tar.gz
bwa_mito_idx_tar https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bwa_index/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bwa_index.tar.gz
tss https://www.encodeproject.org/files/ENCFF766FGL/@@download/ENCFF766FGL.bed.gz
dnase https://www.encodeproject.org/files/ENCFF304XEX/@@download/ENCFF304XEX.bed.gz
prom https://www.encodeproject.org/files/ENCFF140XLU/@@download/ENCFF140XLU.bed.gz
enh https://www.encodeproject.org/files/ENCFF212UAV/@@download/ENCFF212UAV.bed.gz
reg2map https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_formatted.txt.gz
reg2map_bed https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_celltype_compare_subsample.bed.gz
roadmap_meta https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_metadata.txtAlso the control https://www.encodeproject.org/files/ENCFF178EJY/ is a single-ended one but chip.ctl_paired_ends : [false] defined in your input JSON. Please change it to chip.ctl_paired_end : false and try again.
I have been able to successfully run the chip-seq pipeline for the example json and my own histone bam files. However, while passing in a custom tf .bam file, I am running into errors that I am not too sure how to debug. Any suggestions are much appreciated!
Here are more info about the issue:
I ran
caper run chip.wdl -i /mnt/c/users/shirl/Desktop/guttman_lab/CTCF.json --condaCaper configuration file Locally run
Input JSON file
Error message: