search

ENCODE-DCC / chip-seq-pipeline2

ENCODE ChIP-seq pipeline
MIT License
234 stars 123 forks source link

call-call_peak error, file is empty #117

Closed zhuchcn closed 4 years ago

zhuchcn commented 4 years ago

Describe the bug Hi, I was running the pipeline for TF today with an error. The error message was pasted in the bottom. When I dig deeper, I found this message from the file chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/execution/stdout

...
processing chrUn_KI270754v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270755v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270756v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270757v1 in 1 steps [.] done ( 1 positions)
processing chrUn_GL000214v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270742v1 in 1 steps [.] done ( 2 positions)
processing chrUn_GL000216v2 in 1 steps [.] done ( 1 positions)
processing chrUn_GL000218v1 in 1 steps [.] done ( 1 positions)
excluding systematic background anomalies ... done
calculating statistical thresholds
FDR 0.99 threshold= Inf 
Detected 0 peaks
[2019-12-19 22:59:22,514 INFO] run_shell_cmd: PID=32, PGID=32, CMD=zcat -f RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz.tmp | awk 'BEGIN{OFS="\t"}{if ($2<0) $2=0; print $1,int($2),int($3),$4,$5,$6,$7,$8,$9,$10;}' | gzip -f -nc > RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz
[2019-12-19 22:59:22,520 INFO] PID=32, PGID=32, RC=0
STDERR=
STDOUT=
[2019-12-19 22:59:22,523 INFO] run_shell_cmd: PID=36, PGID=36, CMD=rm -f RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz.tmp RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz.tmp.gz
[2019-12-19 22:59:22,527 INFO] PID=36, PGID=36, RC=0
STDERR=
STDOUT=
[2019-12-19 22:59:22,529 INFO] Checking if output is empty...
[2019-12-19 22:59:22,531 INFO] run_shell_cmd: PID=38, PGID=38, CMD=zcat -f RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz | wc -l
[2019-12-19 22:59:22,537 INFO] PID=38, PGID=38, RC=0
STDERR=
STDOUT=0

So it looks like the threshold calculated run_spp.R is somehow Inf thus no peak was kept.

OS/Platform

Caper configuration file

backend=local

tmp-dir=temp

Input JSON file

{
    "chip.title" : "RORgamma",
    "chip.description" : "RORgamma chip seq",

    "chip.pipeline_type" : "tf",
    "chip.aligner" : "bwa",
    "chip.align_only" : false,
    "chip.true_rep_only" : false,

    "chip.genome_tsv" : "/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/hg38_caper.tsv",

    "chip.paired_end" : false,
    "chip.ctl_paired_end" : false,

    "chip.always_use_pooled_ctl" : false,

    "chip.fastqs_rep1_R1" : [ "/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/RORgamma_SR8278.fq.gz" ],
    "chip.fastqs_rep2_R1" : [ "/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/RORgamma_SR2211.fq.gz" ],
    "chip.fastqs_rep3_R1" : [ "/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/RORgamma_Vehicle.fq.gz" ],

    "chip.ctl_fastqs_rep1_R1" : [ "/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/Input-RORgamma_SR8278.fq.gz" ],
    "chip.ctl_fastqs_rep2_R1" : [ "/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/Input-RORgamma_SR2211.fq.gz" ],
    "chip.ctl_fastqs_rep3_R1" : [ "/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/Input-RORgamma_Vehicle.fq.gz" ],

    "chip.align_cpu": 32,
    "chip.align_mem_mb": 98304,

    "chip.filter_cpu": 8,
    "chip.filter_mem_mb": 49152,

    "chip.bam2ta_cpu": 4,
    "chip.bam2ta_mem_mb": 49152,

    "chip.spr_mem_mb": 49152,

    "chip.jsd_cpu": 8,
    "chip.jsd_mem_mb": 49152,

    "chip.xcor_cpu": 8,
    "chip.xcor_mem_mb": 49152,

    "chip.call_peak_cpu": 8,
    "chip.call_peak_mem_mb": 49152,

    "chip.macs2_signal_track_mem_mb": 49152
}

Error log

[Caper] troubleshooting 3516233a-e8eb-4cca-9c61-f05ce8c6881e ...
Found failures:
[
    {
        "message": "Workflow failed",
        "causedBy": [
            {
                "causedBy": [],
                "message": "Job chip.call_peak_ppr2:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."
            },
            {
                "causedBy": [],
                "message": "Job chip.call_peak:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."
            },
            {
                "causedBy": [],
                "message": "Job chip.call_peak:1:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."
            },
            {
                "message": "Job chip.call_peak_ppr1:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
                "causedBy": []
            }
        ]
    }
]

chip.call_peak_ppr1 RetryableFailure. SHARD_IDX=-1, RC=1, JOB_ID=376, RUN_START=2019-12-19T21:19:12.647Z, RUN_END=2019-12-19T23:19:54.449Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak_ppr1/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak_ppr1/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 94, in main
    assert_file_not_empty(rpeak)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 212, in assert_file_not_empty
    raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (RORgamma_SR8278.merged.nodup.pr1.pooled_x_Input-RORgamma_SR8278.merged.nodup.pooled.300K.regionPeak.gz). Help: 

chip.call_peak_ppr1 Failed. SHARD_IDX=-1, RC=1, JOB_ID=24440, RUN_START=2019-12-19T23:19:54.643Z, RUN_END=2019-12-20T01:16:32.414Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak_ppr1/attempt-2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak_ppr1/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 94, in main
    assert_file_not_empty(rpeak)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 212, in assert_file_not_empty
    raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (RORgamma_SR8278.merged.nodup.pr1.pooled_x_Input-RORgamma_SR8278.merged.nodup.pooled.300K.regionPeak.gz). Help: 

chip.call_peak RetryableFailure. SHARD_IDX=0, RC=1, JOB_ID=2177, RUN_START=2019-12-19T21:20:22.643Z, RUN_END=2019-12-19T23:01:24.449Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 94, in main
    assert_file_not_empty(rpeak)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 212, in assert_file_not_empty
    raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz). Help: 

chip.call_peak Failed. SHARD_IDX=0, RC=1, JOB_ID=22683, RUN_START=2019-12-19T23:01:26.643Z, RUN_END=2019-12-20T00:34:51.143Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/attempt-2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 94, in main
    assert_file_not_empty(rpeak)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 212, in assert_file_not_empty
    raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz). Help: 

chip.call_peak RetryableFailure. SHARD_IDX=1, RC=1, JOB_ID=1207, RUN_START=2019-12-19T21:20:10.646Z, RUN_END=2019-12-19T23:06:09.449Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-1/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-1/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 94, in main
    assert_file_not_empty(rpeak)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 212, in assert_file_not_empty
    raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (RORgamma_SR2211.merged.nodup_x_ctl_for_rep2.300K.regionPeak.gz). Help: 

chip.call_peak Failed. SHARD_IDX=1, RC=1, JOB_ID=23352, RUN_START=2019-12-19T23:06:10.647Z, RUN_END=2019-12-20T00:49:40.991Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-1/attempt-2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-1/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 94, in main
    assert_file_not_empty(rpeak)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 212, in assert_file_not_empty
    raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (RORgamma_SR2211.merged.nodup_x_ctl_for_rep2.300K.regionPeak.gz). Help: 

chip.call_peak_ppr2 RetryableFailure. SHARD_IDX=-1, RC=1, JOB_ID=723, RUN_START=2019-12-19T21:19:16.646Z, RUN_END=2019-12-19T22:52:09.486Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak_ppr2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak_ppr2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 94, in main
    assert_file_not_empty(rpeak)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 212, in assert_file_not_empty
    raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (RORgamma_SR8278.merged.nodup.pr2.pooled_x_Input-RORgamma_SR8278.merged.nodup.pooled.300K.regionPeak.gz). Help: 

chip.call_peak_ppr2 Failed. SHARD_IDX=-1, RC=1, JOB_ID=22098, RUN_START=2019-12-19T22:52:10.643Z, RUN_END=2019-12-20T00:17:59.830Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak_ppr2/attempt-2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak_ppr2/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 94, in main
    assert_file_not_empty(rpeak)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 212, in assert_file_not_empty
    raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (RORgamma_SR8278.merged.nodup.pr2.pooled_x_Input-RORgamma_SR8278.merged.nodup.pooled.300K.regionPeak.gz). Help:
leepc12 commented 4 years ago

Pipeline failed at peak-calling stage. No peaks were found. Find *.tagAlign.gz (BED version of your BAM file) and check reads in them.

zhuchcn commented 4 years ago

@leepc12 *.tagAlign.gz is not empty, at least. See below.

$ zcat inputs/1934196154/ctl_for_rep1.tagAlign.gz | head
chr1    10206   10256   N   1000    +
chr1    10206   10257   N   1000    -
chr1    10209   10259   N   1000    -
chr1    10215   10265   N   1000    +
chr1    10226   10276   N   1000    -
chr1    10299   10349   N   1000    -
chr1    10305   10355   N   1000    -
chr1    10396   10446   N   1000    +
chr1    50469   50519   N   1000    +
chr1    69531   69581   N   1000    -

$ zcat inputs/1934196154/ctl_for_rep1.tagAlign.gz | wc -l
12301614

According to the logging message in call-call_peak/shard-0/execution/stdout , it seems like spp is able to find peak, but it somehow used a threshold of infinity, so none of the peaks were saved to file.

leepc12 commented 4 years ago

Can you upload that stdout here?

zhuchcn commented 4 years ago

Sure! I attached it here. It's a little bit long.

[2019-12-19 21:20:25,007 INFO] ['/software/chip-seq-pipeline/src/encode_task_spp.py', '/cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/inputs/716451503/RORgamma_SR8278.merged.nodup.tagAlign.gz', '/cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/inputs/1934196154/ctl_for_rep1.tagAlign.gz', '--fraglen', '75', '--cap-num-peak', '300000', '--nth', '8']
[2019-12-19 21:20:25,007 INFO] Initializing and making output directory...
[2019-12-19 21:20:25,007 INFO] Calling peaks with spp...
[2019-12-19 21:20:25,009 INFO] run_shell_cmd: PID=16, PGID=16, CMD=Rscript --max-ppsize=500000 $(which run_spp.R) -c=/cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/inputs/716451503/RORgamma_SR8278.merged.nodup.tagAlign.gz -i=/cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/inputs/1934196154/ctl_for_rep1.tagAlign.gz -npeak=300000 -odir=/cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/execution -speak=75 -savr=RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz.tmp -rf 
[2019-12-19 22:59:22,459 INFO] PID=16, PGID=16, RC=0
STDERR=Loading required package: caTools
There were 32 warnings (use warnings() to see them)
There were 50 or more warnings (use warnings() to see the first 50)
STDOUT=################
ChIP data: /cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/inputs/716451503/RORgamma_SR8278.merged.nodup.tagAlign.gz 
Control data: /cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/inputs/1934196154/ctl_for_rep1.tagAlign.gz 
strandshift(min): -500 
strandshift(step): 5 
strandshift(max) 1500 
user-defined peak shift 75 
exclusion(min): 10 
exclusion(max): NaN 
num parallel nodes: NA 
FDR threshold: 0.01 
NumPeaks Threshold: 3e+05 
Output Directory: /cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/execution 
narrowPeak output file name: NA 
regionPeak output file name: RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz.tmp 
Rdata filename: NA 
plot pdf filename: NA 
result filename: NA 
Overwrite files?: TRUE

Decompressing ChIP file
Decompressing control file
Reading ChIP tagAlign/BAM file /cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/inputs/716451503/RORgamma_SR8278.merged.nodup.tagAlign.gz 
opened /cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/tmp.bb4fac37/Rtmp5UQhg0/RORgamma_SR8278.merged.nodup.tagAlign126d5c0a89
done. read 10520742 fragments
ChIP data read length 50 
[1] TRUE
Reading Control tagAlign/BAM file /cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/inputs/1934196154/ctl_for_rep1.tagAlign.gz 
opened /cromwell-executions/chip/3516233a-e8eb-4cca-9c61-f05ce8c6881e/call-call_peak/shard-0/tmp.bb4fac37/Rtmp5UQhg0/ctl_for_rep1.tagAlign125b39d679
done. read 12301614 fragments
Control data read length 50 
Calculating peak characteristics
Minimum cross-correlation value 0.1269051 
Minimum cross-correlation shift 1500 
Top 3 cross-correlation values 0.1275339867587 
Top 3 estimates for fragment length 75 
Window half size 185 
Phantom peak location 50 
Phantom peak Correlation 0.1280596 
Normalized Strand cross-correlation coefficient (NSC) 1.004955 
Relative Strand cross-correlation Coefficient (RSC) 0.5447293 
Phantom Peak Quality Tag 0 
Removing read stacks
Finding peaks
finding background exclusion regions ... done
determining peaks on provided 1 control datasets:
using reversed signal for FDR calculations
bg.weight= 1.155196  processing chr1 in 7 steps [.......] done ( 59320 positions)
processing chr2 in 7 steps [.......] done ( 37850 positions)
processing chr3 in 5 steps [.....] done ( 46030 positions)
processing chr4 in 5 steps [.....] done ( 39100 positions)
processing chr5 in 5 steps [.....] done ( 40251 positions)
processing chr6 in 5 steps [.....] done ( 33673 positions)
processing chr7 in 4 steps [....] done ( 40978 positions)
processing chr8 in 4 steps [....] done ( 34131 positions)
processing chr9 in 4 steps [....] done ( 28746 positions)
processing chr10 in 4 steps [....] done ( 33877 positions)
processing chr11 in 4 steps [....] done ( 32916 positions)
processing chr12 in 4 steps [....] done ( 37219 positions)
processing chr13 in 3 steps [...] done ( 12926 positions)
processing chr14 in 3 steps [...] done ( 23462 positions)
processing chr15 in 3 steps [...] done ( 21972 positions)
processing chr16 in 3 steps [...] done ( 26324 positions)
processing chr17 in 3 steps [...] done ( 29718 positions)
processing chr18 in 3 steps [...] done ( 16316 positions)
processing chr19 in 2 steps [..] done ( 27030 positions)
processing chr20 in 2 steps [..] done ( 18267 positions)
processing chr21 in 2 steps [..] done ( 7370 positions)
processing chr22 in 2 steps [..] done ( 12352 positions)
processing chrX in 4 steps [....] done ( 15787 positions)
processing chrY in 2 steps [..] done ( 1893 positions)
processing chrM in 1 steps [.] done ( 44 positions)
processing chr1_KI270706v1_random in 1 steps [.] done ( 8 positions)
processing chr1_KI270707v1_random in 1 steps [.] done ( 1 positions)
processing chr1_KI270708v1_random in 1 steps [.] done ( 5 positions)
processing chr1_KI270709v1_random in 1 steps [.] done ( 20 positions)
processing chr1_KI270710v1_random in 1 steps [.] done ( 3 positions)
processing chr1_KI270711v1_random in 1 steps [.] done ( 0 positions)
processing chr1_KI270712v1_random in 1 steps [.] done ( 11 positions)
processing chr1_KI270713v1_random in 1 steps [.] done ( 3 positions)
processing chr1_KI270714v1_random in 1 steps [.] done ( 0 positions)
processing chr2_KI270716v1_random in 1 steps [.] done ( 11 positions)
processing chr3_GL000221v1_random in 1 steps [.] done ( 1 positions)
processing chr4_GL000008v2_random in 1 steps [.] done ( 15 positions)
processing chr5_GL000208v1_random in 1 steps [.] done ( 41 positions)
processing chr9_KI270717v1_random in 1 steps [.] done ( 1 positions)
processing chr9_KI270718v1_random in 1 steps [.] done ( 1 positions)
processing chr9_KI270719v1_random in 1 steps [.] done ( 5 positions)
processing chr9_KI270720v1_random in 1 steps [.] done ( 1 positions)
processing chr14_GL000009v2_random in 1 steps [.] done ( 7 positions)
processing chr14_GL000225v1_random in 1 steps [.] done ( 140 positions)
processing chr14_KI270722v1_random in 1 steps [.] done ( 20 positions)
processing chr14_GL000194v1_random in 1 steps [.] done ( 4 positions)
processing chr14_KI270723v1_random in 1 steps [.] done ( 5 positions)
processing chr14_KI270724v1_random in 1 steps [.] done ( 3 positions)
processing chr14_KI270725v1_random in 1 steps [.] done ( 2 positions)
processing chr14_KI270726v1_random in 1 steps [.] done ( 3 positions)
processing chr15_KI270727v1_random in 1 steps [.] done ( 0 positions)
processing chr16_KI270728v1_random in 1 steps [.] done ( 31 positions)
processing chr17_GL000205v2_random in 1 steps [.] done ( 80 positions)
processing chr17_KI270729v1_random in 1 steps [.] done ( 162 positions)
processing chr17_KI270730v1_random in 1 steps [.] done ( 33 positions)
processing chr22_KI270731v1_random in 1 steps [.] done ( 0 positions)
processing chr22_KI270732v1_random in 1 steps [.] done ( 9 positions)
processing chr22_KI270733v1_random in 1 steps [.] done ( 31 positions)
processing chr22_KI270734v1_random in 1 steps [.] done ( 0 positions)
processing chr22_KI270735v1_random in 1 steps [.] done ( 5 positions)
processing chr22_KI270736v1_random in 1 steps [.] done ( 164 positions)
processing chr22_KI270737v1_random in 1 steps [.] done ( 5 positions)
processing chr22_KI270738v1_random in 1 steps [.] done ( 92 positions)
processing chr22_KI270739v1_random in 1 steps [.] done ( 1 positions)
processing chrY_KI270740v1_random in 1 steps [.] done ( 0 positions)
processing chrUn_KI270322v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270320v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270311v1 in 1 steps [.] done ( 1 positions)
processing chrUn_KI270420v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270424v1 in 1 steps [.] done (  positions)
processing chrUn_KI270429v1 in 1 steps [.] done ( 2 positions)
processing chrUn_KI270442v1 in 1 steps [.] done ( 318 positions)
processing chrUn_KI270466v1 in 1 steps [.] done ( 3 positions)
processing chrUn_KI270465v1 in 1 steps [.] done ( 1 positions)
processing chrUn_KI270467v1 in 1 steps [.] done ( 9 positions)
processing chrUn_KI270435v1 in 1 steps [.] done ( 34 positions)
processing chrUn_KI270438v1 in 1 steps [.] done ( 99 positions)
processing chrUn_KI270468v1 in 1 steps [.] done ( 3 positions)
processing chrUn_KI270510v1 in 1 steps [.] done ( 2 positions)
processing chrUn_KI270509v1 in 1 steps [.] done ( 2 positions)
processing chrUn_KI270518v1 in 1 steps [.] done ( 1 positions)
processing chrUn_KI270508v1 in 1 steps [.] done ( 2 positions)
processing chrUn_KI270516v1 in 1 steps [.] done ( 2 positions)
processing chrUn_KI270512v1 in 1 steps [.] done ( 9 positions)
processing chrUn_KI270519v1 in 1 steps [.] done ( 77 positions)
processing chrUn_KI270522v1 in 1 steps [.] done ( 3 positions)
processing chrUn_KI270511v1 in 1 steps [.] done ( 8 positions)
processing chrUn_KI270515v1 in 1 steps [.] done ( 9 positions)
processing chrUn_KI270507v1 in 1 steps [.] done ( 4 positions)
processing chrUn_KI270517v1 in 1 steps [.] done ( 4 positions)
processing chrUn_KI270538v1 in 1 steps [.] done ( 56 positions)
processing chrUn_KI270583v1 in 1 steps [.] done ( 1 positions)
processing chrUn_KI270587v1 in 1 steps [.] done ( 4 positions)
processing chrUn_KI270580v1 in 1 steps [.] done ( 1 positions)
processing chrUn_KI270589v1 in 1 steps [.] done ( 5 positions)
processing chrUn_KI270590v1 in 1 steps [.] done ( 10 positions)
processing chrUn_KI270584v1 in 1 steps [.] done ( 8 positions)
processing chrUn_KI270588v1 in 1 steps [.] done ( 6 positions)
processing chrUn_KI270593v1 in 1 steps [.] done ( 2 positions)
processing chrUn_KI270591v1 in 1 steps [.] done ( 7 positions)
processing chrUn_KI270330v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270333v1 in 1 steps [.] done ( 4 positions)
processing chrUn_KI270336v1 in 1 steps [.] done ( 3 positions)
processing chrUn_KI270337v1 in 1 steps [.] done ( 3 positions)
processing chrUn_KI270362v1 in 1 steps [.] done ( 1 positions)
processing chrUn_KI270448v1 in 1 steps [.] done ( 6 positions)
processing chrUn_KI270521v1 in 1 steps [.] done ( 1 positions)
processing chrUn_GL000195v1 in 1 steps [.] done ( 10 positions)
processing chrUn_GL000219v1 in 1 steps [.] done ( 25 positions)
processing chrUn_GL000220v1 in 1 steps [.] done ( 4 positions)
processing chrUn_GL000224v1 in 1 steps [.] done ( 23 positions)
processing chrUn_KI270741v1 in 1 steps [.] done ( 0 positions)
processing chrUn_GL000213v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270743v1 in 1 steps [.] done ( 6 positions)
processing chrUn_KI270744v1 in 1 steps [.] done ( 11 positions)
processing chrUn_KI270746v1 in 1 steps [.] done ( 6 positions)
processing chrUn_KI270747v1 in 1 steps [.] done ( 4 positions)
processing chrUn_KI270749v1 in 1 steps [.] done ( 9 positions)
processing chrUn_KI270750v1 in 1 steps [.] done ( 4 positions)
processing chrUn_KI270751v1 in 1 steps [.] done ( 14 positions)
processing chrUn_KI270753v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270754v1 in 1 steps [.] done ( 1 positions)
processing chrUn_KI270755v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270756v1 in 1 steps [.] done ( 15 positions)
processing chrUn_KI270757v1 in 1 steps [.] done ( 31 positions)
processing chrUn_GL000214v1 in 1 steps [.] done ( 4 positions)
processing chrUn_KI270742v1 in 1 steps [.] done ( 4 positions)
processing chrUn_GL000216v2 in 1 steps [.] done ( 129 positions)
processing chrUn_GL000218v1 in 1 steps [.] done ( 13 positions)
excluding systematic background anomalies ... done
determining peaks on real data:
bg.weight= 0.8656538  processing chr1 in 7 steps [.......] done ( 25654 positions)
processing chr2 in 7 steps [.......] done ( 14878 positions)
processing chr3 in 5 steps [.....] done ( 24370 positions)
processing chr4 in 5 steps [.....] done ( 20644 positions)
processing chr5 in 5 steps [.....] done ( 21525 positions)
processing chr6 in 5 steps [.....] done ( 15697 positions)
processing chr7 in 4 steps [....] done ( 17469 positions)
processing chr8 in 4 steps [....] done ( 17182 positions)
processing chr9 in 4 steps [....] done ( 12905 positions)
processing chr10 in 4 steps [....] done ( 14196 positions)
processing chr11 in 4 steps [....] done ( 15323 positions)
processing chr12 in 4 steps [....] done ( 15205 positions)
processing chr13 in 3 steps [...] done ( 5878 positions)
processing chr14 in 3 steps [...] done ( 10662 positions)
processing chr15 in 3 steps [...] done ( 9339 positions)
processing chr16 in 3 steps [...] done ( 7141 positions)
processing chr17 in 3 steps [...] done ( 7800 positions)
processing chr18 in 3 steps [...] done ( 9296 positions)
processing chr19 in 2 steps [..] done ( 4157 positions)
processing chr20 in 2 steps [..] done ( 6856 positions)
processing chr21 in 2 steps [..] done ( 3005 positions)
processing chr22 in 2 steps [..] done ( 2994 positions)
processing chrX in 4 steps [....] done ( 4000 positions)
processing chrY in 2 steps [..] done ( 219 positions)
processing chrM in 1 steps [.] done ( 0 positions)
processing chr1_KI270706v1_random in 1 steps [.] done ( 0 positions)
processing chr1_KI270707v1_random in 1 steps [.] done (  positions)
processing chr1_KI270708v1_random in 1 steps [.] done ( 0 positions)
processing chr1_KI270709v1_random in 1 steps [.] done ( 1 positions)
processing chr1_KI270710v1_random in 1 steps [.] done ( 0 positions)
processing chr1_KI270711v1_random in 1 steps [.] done (  positions)
processing chr1_KI270712v1_random in 1 steps [.] done ( 0 positions)
processing chr1_KI270713v1_random in 1 steps [.] done ( 0 positions)
processing chr2_KI270716v1_random in 1 steps [.] done ( 0 positions)
processing chr3_GL000221v1_random in 1 steps [.] done ( 0 positions)
processing chr4_GL000008v2_random in 1 steps [.] done ( 0 positions)
processing chr5_GL000208v1_random in 1 steps [.] done ( 1 positions)
processing chr9_KI270717v1_random in 1 steps [.] done ( 0 positions)
processing chr9_KI270718v1_random in 1 steps [.] done ( 0 positions)
processing chr9_KI270719v1_random in 1 steps [.] done ( 1 positions)
processing chr9_KI270720v1_random in 1 steps [.] done ( 0 positions)
processing chr14_GL000009v2_random in 1 steps [.] done ( 0 positions)
processing chr14_GL000225v1_random in 1 steps [.] done ( 3 positions)
processing chr14_KI270722v1_random in 1 steps [.] done ( 11 positions)
processing chr14_GL000194v1_random in 1 steps [.] done ( 0 positions)
processing chr14_KI270723v1_random in 1 steps [.] done ( 0 positions)
processing chr14_KI270724v1_random in 1 steps [.] done ( 1 positions)
processing chr14_KI270725v1_random in 1 steps [.] done ( 1 positions)
processing chr14_KI270726v1_random in 1 steps [.] done ( 0 positions)
processing chr15_KI270727v1_random in 1 steps [.] done ( 0 positions)
processing chr16_KI270728v1_random in 1 steps [.] done ( 1 positions)
processing chr17_GL000205v2_random in 1 steps [.] done ( 34 positions)
processing chr17_KI270729v1_random in 1 steps [.] done ( 1 positions)
processing chr17_KI270730v1_random in 1 steps [.] done ( 0 positions)
processing chr22_KI270731v1_random in 1 steps [.] done ( 0 positions)
processing chr22_KI270732v1_random in 1 steps [.] done ( 1 positions)
processing chr22_KI270733v1_random in 1 steps [.] done ( 0 positions)
processing chr22_KI270735v1_random in 1 steps [.] done ( 1 positions)
processing chr22_KI270736v1_random in 1 steps [.] done ( 3 positions)
processing chr22_KI270737v1_random in 1 steps [.] done ( 0 positions)
processing chr22_KI270738v1_random in 1 steps [.] done ( 1 positions)
processing chr22_KI270739v1_random in 1 steps [.] done ( 0 positions)
processing chrY_KI270740v1_random in 1 steps [.] done ( 0 positions)
processing chrUn_KI270302v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270322v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270320v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270311v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270420v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270429v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270442v1 in 1 steps [.] done ( 9 positions)
processing chrUn_KI270466v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270465v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270467v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270435v1 in 1 steps [.] done ( 2 positions)
processing chrUn_KI270438v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270468v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270510v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270509v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270518v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270508v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270516v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270512v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270519v1 in 1 steps [.] done ( 2 positions)
processing chrUn_KI270522v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270511v1 in 1 steps [.] done ( 1 positions)
processing chrUn_KI270515v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270507v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270517v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270538v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270583v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270587v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270580v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270589v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270590v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270584v1 in 1 steps [.] done ( 1 positions)
processing chrUn_KI270588v1 in 1 steps [.] done ( 1 positions)
processing chrUn_KI270593v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270591v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270330v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270333v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270336v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270337v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270448v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270521v1 in 1 steps [.] done ( 0 positions)
processing chrUn_GL000195v1 in 1 steps [.] done ( 3 positions)
processing chrUn_GL000219v1 in 1 steps [.] done ( 3 positions)
processing chrUn_GL000220v1 in 1 steps [.] done ( 0 positions)
processing chrUn_GL000224v1 in 1 steps [.] done ( 4 positions)
processing chrUn_GL000213v1 in 1 steps [.] done (  positions)
processing chrUn_KI270743v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270744v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270745v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270746v1 in 1 steps [.] done ( 9 positions)
processing chrUn_KI270747v1 in 1 steps [.] done (  positions)
processing chrUn_KI270749v1 in 1 steps [.] done ( 2 positions)
processing chrUn_KI270750v1 in 1 steps [.] done ( 13 positions)
processing chrUn_KI270751v1 in 1 steps [.] done ( 3 positions)
processing chrUn_KI270753v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270754v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270755v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270756v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270757v1 in 1 steps [.] done ( 1 positions)
processing chrUn_GL000214v1 in 1 steps [.] done ( 0 positions)
processing chrUn_KI270742v1 in 1 steps [.] done ( 2 positions)
processing chrUn_GL000216v2 in 1 steps [.] done ( 1 positions)
processing chrUn_GL000218v1 in 1 steps [.] done ( 1 positions)
excluding systematic background anomalies ... done
calculating statistical thresholds
FDR 0.99 threshold= Inf 
Detected 0 peaks
[2019-12-19 22:59:22,514 INFO] run_shell_cmd: PID=32, PGID=32, CMD=zcat -f RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz.tmp | awk 'BEGIN{OFS="\t"}{if ($2<0) $2=0; print $1,int($2),int($3),$4,$5,$6,$7,$8,$9,$10;}' | gzip -f -nc > RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz
[2019-12-19 22:59:22,520 INFO] PID=32, PGID=32, RC=0
STDERR=
STDOUT=
[2019-12-19 22:59:22,523 INFO] run_shell_cmd: PID=36, PGID=36, CMD=rm -f RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz.tmp RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz.tmp.gz
[2019-12-19 22:59:22,527 INFO] PID=36, PGID=36, RC=0
STDERR=
STDOUT=
[2019-12-19 22:59:22,529 INFO] Checking if output is empty...
[2019-12-19 22:59:22,531 INFO] run_shell_cmd: PID=38, PGID=38, CMD=zcat -f RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz | wc -l
[2019-12-19 22:59:22,537 INFO] PID=38, PGID=38, RC=0
STDERR=
STDOUT=0

And here is the stderr which also appears in metadata.json.

Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 103, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_spp.py", line 94, in main
    assert_file_not_empty(rpeak)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 212, in assert_file_not_empty
    raise Exception('File is empty ({}). Help: {}'.format(f, help))
Exception: File is empty (RORgamma_SR8278.merged.nodup_x_ctl_for_rep1.300K.regionPeak.gz). Help:
leepc12 commented 4 years ago

Did you choose a correct genome (and its genome_tsv file)?

zhuchcn commented 4 years ago

The experiment was done with human prostate cancer cell lines so we used the hg38. The hg38.tsv is at the correct location. I pasted it here below. I didn't see anything strange.


genome_name     hg38
ref_fa  https://storage.googleapis.com/encode-pipeline-genome-data/hg38/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz
ref_mito_fa     https://storage.googleapis.com/encode-pipeline-genome-data/hg38/GRCh38_no_alt_analysis_set_GCA_000001405.15.chrM.fa.gz
mito_chr_name   chrM
blacklist       https://storage.googleapis.com/encode-pipeline-genome-data/hg38/hg38.blacklist.bed.gz
chrsz   https://storage.googleapis.com/encode-pipeline-genome-data/hg38/hg38.chrom.sizes
gensz   hs
bowtie2_idx_tar https://storage.googleapis.com/encode-pipeline-genome-data/hg38/bowtie2_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar
bowtie2_mito_idx_tar    https://storage.googleapis.com/encode-pipeline-genome-data/hg38/bowtie2_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.chrM.fa.tar
bwa_idx_tar     https://storage.googleapis.com/encode-pipeline-genome-data/hg38/bwa_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar
bwa_mito_idx_tar        https://storage.googleapis.com/encode-pipeline-genome-data/hg38/bwa_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.chrM.fa.tar
tss     https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_gencode_tss_unique.bed.gz
dnase   https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_all_p10_ucsc.hg19_to_hg38.bed.gz
prom    https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_prom_p2.hg19_to_hg38.bed.gz
enh     https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_enh_p2.hg19_to_hg38.bed.gz
reg2map https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_formatted.txt.gz
reg2map_bed     https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_celltype_compare_subsample.bed.gz
roadmap_meta    https://storage.googleapis.com/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_metadata.txt
leepc12 commented 4 years ago

For debugging, can you add the following to your input JSON and try again? This will change the peak caller from SPP to MACS2

    "chip.peak_caller": "macs2",
zhuchcn commented 4 years ago

I added the line that you suggested to the JSON file and did it again. It gave me a different error this time, and it looks like that the call_peak steps went through with macs2. The probably now appears to be the idr.

[Caper] troubleshooting 817defca-2726-48de-9ea0-78371ec6e672 ...
Found failures:
[
    {
        "message": "Workflow failed",
        "causedBy": [
            {
                "causedBy": [],
                "message": "Job chip.idr:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."
            },
            {
                "causedBy": [],
                "message": "Job chip.idr:2:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."
            },
            {
                "causedBy": [],
                "message": "Job chip.idr_pr:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."
            },
            {
                "causedBy": [],
                "message": "Job chip.idr:1:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."
            },
            {
                "message": "Job chip.idr_ppr:NA:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
                "causedBy": []
            }
        ]
    }
]

chip.idr_ppr RetryableFailure. SHARD_IDX=-1, RC=1, JOB_ID=44744, RUN_START=2019-12-27T08:12:05.631Z, RUN_END=2019-12-27T08:12:41.568Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr_ppr/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr_ppr/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 180, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 152, in main
    idr_peak, args.blacklist, args.regex_bfilt_peak_chr_name, args.out_dir)
  File "/software/chip-seq-pipeline/src/encode_lib_blacklist_filter.py", line 54, in blacklist_filter
    run_shell_cmd(cmd)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=41, PGID=41, RC=1
STDERR=
STDOUT=

chip.idr_ppr Failed. SHARD_IDX=-1, RC=1, JOB_ID=45884, RUN_START=2019-12-27T08:12:45.628Z, RUN_END=2019-12-27T08:13:28.008Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr_ppr/attempt-2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr_ppr/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 180, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 152, in main
    idr_peak, args.blacklist, args.regex_bfilt_peak_chr_name, args.out_dir)
  File "/software/chip-seq-pipeline/src/encode_lib_blacklist_filter.py", line 54, in blacklist_filter
    run_shell_cmd(cmd)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=41, PGID=41, RC=1
STDERR=
STDOUT=

chip.idr RetryableFailure. SHARD_IDX=0, RC=1, JOB_ID=43769, RUN_START=2019-12-27T08:11:55.631Z, RUN_END=2019-12-27T08:12:41.568Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-0/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-0/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 180, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 152, in main
    idr_peak, args.blacklist, args.regex_bfilt_peak_chr_name, args.out_dir)
  File "/software/chip-seq-pipeline/src/encode_lib_blacklist_filter.py", line 54, in blacklist_filter
    run_shell_cmd(cmd)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=41, PGID=41, RC=1
STDERR=
STDOUT=

chip.idr Failed. SHARD_IDX=0, RC=1, JOB_ID=45736, RUN_START=2019-12-27T08:12:43.629Z, RUN_END=2019-12-27T08:13:29.959Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-0/attempt-2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-0/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 180, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 152, in main
    idr_peak, args.blacklist, args.regex_bfilt_peak_chr_name, args.out_dir)
  File "/software/chip-seq-pipeline/src/encode_lib_blacklist_filter.py", line 54, in blacklist_filter
    run_shell_cmd(cmd)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=41, PGID=41, RC=1
STDERR=
STDOUT=

chip.idr RetryableFailure. SHARD_IDX=1, RC=1, JOB_ID=44537, RUN_START=2019-12-27T08:12:03.631Z, RUN_END=2019-12-27T08:12:41.568Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-1/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-1/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 180, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 152, in main
    idr_peak, args.blacklist, args.regex_bfilt_peak_chr_name, args.out_dir)
  File "/software/chip-seq-pipeline/src/encode_lib_blacklist_filter.py", line 54, in blacklist_filter
    run_shell_cmd(cmd)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=41, PGID=41, RC=1
STDERR=
STDOUT=

chip.idr Failed. SHARD_IDX=1, RC=1, JOB_ID=46031, RUN_START=2019-12-27T08:12:47.628Z, RUN_END=2019-12-27T08:13:26.872Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-1/attempt-2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-1/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 180, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 152, in main
    idr_peak, args.blacklist, args.regex_bfilt_peak_chr_name, args.out_dir)
  File "/software/chip-seq-pipeline/src/encode_lib_blacklist_filter.py", line 54, in blacklist_filter
    run_shell_cmd(cmd)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=41, PGID=41, RC=1
STDERR=
STDOUT=

chip.idr RetryableFailure. SHARD_IDX=2, RC=1, JOB_ID=43610, RUN_START=2019-12-27T08:11:53.634Z, RUN_END=2019-12-27T08:12:36.567Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 180, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 152, in main
    idr_peak, args.blacklist, args.regex_bfilt_peak_chr_name, args.out_dir)
  File "/software/chip-seq-pipeline/src/encode_lib_blacklist_filter.py", line 54, in blacklist_filter
    run_shell_cmd(cmd)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=41, PGID=41, RC=1
STDERR=
STDOUT=

chip.idr Failed. SHARD_IDX=2, RC=1, JOB_ID=45486, RUN_START=2019-12-27T08:12:37.628Z, RUN_END=2019-12-27T08:13:42.659Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-2/attempt-2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr/shard-2/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 180, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 152, in main
    idr_peak, args.blacklist, args.regex_bfilt_peak_chr_name, args.out_dir)
  File "/software/chip-seq-pipeline/src/encode_lib_blacklist_filter.py", line 54, in blacklist_filter
    run_shell_cmd(cmd)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=41, PGID=41, RC=1
STDERR=
STDOUT=

chip.idr_pr RetryableFailure. SHARD_IDX=0, RC=1, JOB_ID=41521, RUN_START=2019-12-27T08:08:55.636Z, RUN_END=2019-12-27T08:11:11.574Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr_pr/shard-0/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr_pr/shard-0/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 180, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 152, in main
    idr_peak, args.blacklist, args.regex_bfilt_peak_chr_name, args.out_dir)
  File "/software/chip-seq-pipeline/src/encode_lib_blacklist_filter.py", line 54, in blacklist_filter
    run_shell_cmd(cmd)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=41, PGID=41, RC=1
STDERR=
STDOUT=

chip.idr_pr Failed. SHARD_IDX=0, RC=1, JOB_ID=43237, RUN_START=2019-12-27T08:11:13.632Z, RUN_END=2019-12-27T08:13:38.112Z, STDOUT=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr_pr/shard-0/attempt-2/execution/stdout, STDERR=/home/hwclab/NGS_Data/ChIPseq_Rawdata_RORgamma_Lncap/caper_output_docker/chip/817defca-2726-48de-9ea0-78371ec6e672/call-idr_pr/shard-0/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 180, in <module>
    main()
  File "/software/chip-seq-pipeline/src/encode_task_idr.py", line 152, in main
    idr_peak, args.blacklist, args.regex_bfilt_peak_chr_name, args.out_dir)
  File "/software/chip-seq-pipeline/src/encode_lib_blacklist_filter.py", line 54, in blacklist_filter
    run_shell_cmd(cmd)
  File "/software/chip-seq-pipeline/src/encode_lib_common.py", line 319, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=41, PGID=41, RC=1
STDERR=
STDOUT=
leepc12 commented 4 years ago

1) MACS2 Some peaks were called with MACS2 and stored in peak-files (*.narrowPeak.gz). But those peaks didn't pass the IDR threshold. Try with a more relaxed idr_thresh.

2) SPP No peaks were called with SPP, which resulted in an empty peak-files (*.regionPeak.gz).

BTW did the small demo sample (ENCSR936KRQ_subsampled) work? If so, please check the quality of your sample.

zhuchcn commented 4 years ago

We did not run with the demo data but we have been using your pipeline several times before with other data and those all worked out fine as far as I can tell. I just checked the quality of our data with fastqc. The quality is great, the means are ~35 throughout the reads for all of the samples.

zhuchcn commented 4 years ago

I haven't solved my problem yet, bu I think it is somehow because of the data quality. So I'm closing it. Many thanks for your help!