search

ENCODE-DCC / chip-seq-pipeline2

ENCODE ChIP-seq pipeline
MIT License
247 stars 123 forks source link

Error in chip.align tasks (Failed to evaluate job outputs) #226

Open sqreb opened 3 years ago

sqreb commented 3 years ago

Hi, I tried to analyze the DAP-seq data using the chip-seq-pipeline2 workflow, but encountered an error (Failed to evaluate job outputs). The task has been completed, the file has been generated, but the file cannot be found by caper.

-rw-r--r-- 1 wangdehe ias_admin         3  4月 12 22:28 MYB25_like.rep2.trim.R1.trim_50bp.read_length.txt
-rw-r--r-- 1 wangdehe ias_admin 784580536  4月 12 22:28 MYB25_like.rep2.trim.R1.trim_50bp.srt.bam
-rw-r--r-- 1 wangdehe ias_admin   1524248  4月 12 22:29 MYB25_like.rep2.trim.R1.trim_50bp.srt.bam.bai
-rw-r--r-- 1 wangdehe ias_admin       342  4月 12 22:32 MYB25_like.rep2.trim.R1.trim_50bp.srt.samstats.qc
drwxr-xr-x 2 wangdehe ias_admin         1  4月 12 21:51 R1
drwxr-xr-x 2 wangdehe ias_admin         1  4月 12 21:59 R1_trimmed
-rw-r--r-- 1 wangdehe ias_admin     15022  4月 12 21:51 script
-rw-r--r-- 1 wangdehe ias_admin       471  4月 12 21:51 script.background
-rw-r--r-- 1 wangdehe ias_admin       350  4月 12 21:51 script.submit
-rw-r--r-- 1 wangdehe ias_admin         0  4月 12 21:51 stderr
-rw-r--r-- 1 wangdehe ias_admin         0  4月 12 21:51 stderr.background
-rw-r--r-- 1 wangdehe ias_admin     10326  4月 12 22:32 stdout
-rw-r--r-- 1 wangdehe ias_admin     10333  4月 12 22:32 stdout.background
-rw-r--r-- 1 wangdehe ias_admin       207  4月 12 21:51 write_tsv_b9349cb0e3563b467dcaa20e89670ada.tmp

{
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does 
not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0
 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an ind
ex value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an ind
ex value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            }

can you please help? Best,

Installation

git clone https://github.com/ENCODE-DCC/chip-seq-pipeline2
cd chip-seq-pipeline2
scripts/install_conda_env.sh mamba
source activate encode-chip-seq-pipeline
pip install caper croo
caper init local

Caper file

backend=local

# Hashing strategy for call-caching (3 choices)
# This parameter is for local (local/slurm/sge/pbs) backend only.
# This is important for call-caching,
# which means re-using outputs from previous/failed workflows.
# Cache will miss if different strategy is used.
# "file" method has been default for all old versions of Caper<1.0.
# "path+modtime" is a new default for Caper>=1.0,
#   file: use md5sum hash (slow).
#   path: use path.
#   path+modtime: use path and modification time.
local-hash-strat=path+modtime

# Local directory for localized files and Cromwell's intermediate files
# If not defined, Caper will make .caper_tmp/ on local-out-dir or CWD.
# /tmp is not recommended here since Caper store all localized data files
# on this directory (e.g. input FASTQs defined as URLs in input JSON).
local-loc-dir=/home/wangdehe/project/caper_loc_dir

cromwell=/home/wangdehe/.caper/cromwell_jar/cromwell-59.jar
womtool=/home/wangdehe/.caper/womtool_jar/womtool-59.jar

Input JSON file

Error in chip.align tasks (Failed to evaluate job outputs)
{
    "chip.pipeline_type" : "tf",
    "chip.genome_tsv" : "/home/wangdehe/project/CottonSingleCell/data/encode_chip_indx/gosHir_HUA.tsv",
    "chip.fastqs_rep1_R1" : ["/home/wangdehe/project/CottonSingleCell/analysis_v3/DAP_seq/trim/MYB25_like.rep1.trim.R1.fastq.gz"
    ],
    "chip.fastqs_rep1_R2" : ["/home/wangdehe/project/CottonSingleCell/analysis_v3/DAP_seq/trim/MYB25_like.rep1.trim.R2.fastq.gz"
    ],
    "chip.fastqs_rep2_R1" : ["/home/wangdehe/project/CottonSingleCell/analysis_v3/DAP_seq/trim/MYB25_like.rep2.trim.R1.fastq.gz"
    ],
    "chip.fastqs_rep2_R2" : ["/home/wangdehe/project/CottonSingleCell/analysis_v3/DAP_seq/trim/MYB25_like.rep2.trim.R2.fastq.gz"
    ],
    "chip.ctl_fastqs_rep1_R1" : ["/home/wangdehe/project/CottonSingleCell/analysis_v3/DAP_seq/trim/MYB25.Input.trim.R1.fastq.gz"
    ],
    "chip.ctl_fastqs_rep1_R2" : ["/home/wangdehe/project/CottonSingleCell/analysis_v3/DAP_seq/trim/MYB25.Input.trim.R2.fastq.gz"
    ],
    "chip.ctl_fastqs_rep2_R1" : ["/home/wangdehe/project/CottonSingleCell/analysis_v3/DAP_seq/trim/MYB25_like.Input.trim.R1.fastq.gz"
    ],
    "chip.ctl_fastqs_rep2_R2" : ["/home/wangdehe/project/CottonSingleCell/analysis_v3/DAP_seq/trim/MYB25_like.Input.trim.R2.fastq.gz"
    ],
    "chip.paired_end" : true,
    "chip.ctl_paired_end" : true,
    "chip.title" : "MYB25-like DAP-seq",
    "chip.description" : "MYB25-like DAP-seq",

    "chip.aligner" : "bowtie2",
    "chip.align_only" : false,
    "chip.true_rep_only" : false,
}

caper debug metadata.json

* Started troubleshooting workflow: id=22177161-660e-4365-9d49-bded5aee4ebf, status=Failed
* Found failures JSON object.
[
    {
        "causedBy": [
            {
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            },
            {
                "causedBy": [
                    {
                        "message": "Bad output 'align_R1.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'",
                        "causedBy": []
                    },
                    {
                        "message": "Bad output 'align_R1.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'",
                        "causedBy": []
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            },
            {
                "message": "Failed to evaluate job outputs",
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"Error in chip.align tasks (Failed to evaluate job outputs)
                    },
                    {
                        "message": "Bad output 'align.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'",
                        "causedBy": []
                    }
                ]
            },
            {
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            },
            {
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            },
            {
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            },
            {
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            }
        ],
        "message": "Workflow failed"
    }
]
* Recursively finding failures in calls (tasks)...

==== NAME=chip.align, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=None, JOB_ID=280536
START=2021-04-12T13:51:13.920Z, END=2021-04-12T15:55:29.372Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align/shard-0/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align/shard-0/execution/stderr
STDERR_CONTENTS=

==== NAME=chip.align, STATUS=Failed, PARENT=
SHARD_IDX=1, RC=None, JOB_ID=280615
START=2021-04-12T13:51:15.912Z, END=2021-04-12T16:07:19.780Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align/shard-1/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align/shard-1/execution/stderr
STDERR_CONTENTS=

==== NAME=chip.align_ctl, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=None, JOB_ID=280465
START=2021-04-12T13:51:11.941Z, END=2021-04-12T15:52:12.966Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_ctl/shard-0/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_ctl/shard-0/execution/stderr
STDERR_CONTENTS=

==== NAME=chip.align_ctl, STATUS=Failed, PARENT=
SHARD_IDX=1, RC=None, JOB_ID=280758
START=2021-04-12T13:51:19.910Z, END=2021-04-12T15:34:12.712Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee* Started troubleshooting workflow: id=22177161-660e-4365-9d49-bded5aee4ebf, status=Failed
* Found failures JSON object.
[
    {
        "causedBy": [
            {
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            },
            {
                "causedBy": [
                    {
                        "message": "Bad output 'align_R1.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'",
                        "causedBy": []
                    },
                    {
                        "message": "Bad output 'align_R1.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'",
                        "causedBy": []
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            },
            {
                "message": "Failed to evaluate job outputs",
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "message": "Bad output 'align.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'",
                        "causedBy": []
                    }
                ]
            },
            {
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            },
            {
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            },
            {
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_ctl.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            },
            {
                "causedBy": [
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.samstat_qc': Bad array access glob(\"*.samstats.qc\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.read_len_log': Bad array access glob(\"*.read_length.txt\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.bai': Bad array access glob(\"*.bai\")[0]: Array size 0 does not have an index value '0'"
                    },
                    {
                        "causedBy": [],
                        "message": "Bad output 'align_R1.bam': Bad array access glob(\"*.bam\")[0]: Array size 0 does not have an index value '0'"
                    }
                ],
                "message": "Failed to evaluate job outputs"
            }
        ],
        "message": "Workflow failed"
    }
]
* Recursively finding failures in calls (tasks)...

==== NAME=chip.align, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=None, JOB_ID=280536
START=2021-04-12T13:51:13.920Z, END=2021-04-12T15:55:29.372Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align/shard-0/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align/shard-0/execution/stderr
STDERR_CONTENTS=

==== NAME=chip.align, STATUS=Failed, PARENT=
SHARD_IDX=1, RC=None, JOB_ID=280615
START=2021-04-12T13:51:15.912Z, END=2021-04-12T16:07:19.780Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align/shard-1/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align/shard-1/execution/stderr
STDERR_CONTENTS=

==== NAME=chip.align_ctl, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=None, JOB_ID=280465
START=2021-04-12T13:51:11.941Z, END=2021-04-12T15:52:12.966Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_ctl/shard-0/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_ctl/shard-0/execution/stderr
STDERR_CONTENTS=

==== NAME=chip.align_ctl, STATUS=Failed, PARENT=
SHARD_IDX=1, RC=None, JOB_ID=280758
START=2021-04-12T13:51:19.910Z, END=2021-04-12T15:34:12.712Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_ctl/shard-1/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_ctl/shard-1/execution/stderr
STDERR_CONTENTS=

==== NAME=chip.align_R1, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=None, JOB_ID=280803
START=2021-04-12T13:51:21.912Z, END=2021-04-12T14:27:07.619Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_R1/shard-0/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_R1/shard-0/execution/stderr
STDERR_CONTENTS=

==== NAME=chip.align_R1, STATUS=Failed, PARENT=
SHARD_IDX=1, RC=None, JOB_ID=280689
START=2021-04-12T13:51:17.910Z, END=2021-04-12T14:34:40.620Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_R1/shard-1/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_R1/shard-1/execution/stderr
STDERR_CONTENTS=

4ebf/call-align_ctl/shard-1/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_ctl/shard-1/execution/stderr
STDERR_CONTENTS=

==== NAME=chip.align_R1, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=None, JOB_ID=280803
START=2021-04-12T13:51:21.912Z, END=2021-04-12T14:27:07.619Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_R1/shard-0/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_R1/shard-0/execution/stderr
STDERR_CONTENTS=

==== NAME=chip.align_R1, STATUS=Failed, PARENT=
SHARD_IDX=1, RC=None, JOB_ID=280689
START=2021-04-12T13:51:17.910Z, END=2021-04-12T14:34:40.620Z
STDOUT=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_R1/shard-1/execution/stdout
STDERR=/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_R1/shard-1/execution/stderr
STDERR_CONTENTS=

STDOUT and STDERR file (chip.align_R1)

tail -n 20 /beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25_like/chip/22177161-660e-4365-9d49-bded5aee4ebf/call-align_R1/shard-1/execution/stdout 
[2021-04-12 22:32:02,007 INFO] List all files in output directory...
[2021-04-12 22:32:02,012 INFO] run_shell_cmd: PID=68066, PGID=68066, CMD=ls -l 
[2021-04-12 22:32:02,039 INFO] PID=68066, PGID=68066, RC=0, DURATION_SEC=0.0
STDERR=
STDOUT=total 767721
-rw-r--r-- 1 wangdehe ias_admin         3  4月 12 22:28 MYB25_like.rep2.trim.R1.trim_50bp.read_length.txt
-rw-r--r-- 1 wangdehe ias_admin 784580536  4月 12 22:28 MYB25_like.rep2.trim.R1.trim_50bp.srt.bam
-rw-r--r-- 1 wangdehe ias_admin   1524248  4月 12 22:29 MYB25_like.rep2.trim.R1.trim_50bp.srt.bam.bai
-rw-r--r-- 1 wangdehe ias_admin       342  4月 12 22:32 MYB25_like.rep2.trim.R1.trim_50bp.srt.samstats.qc
drwxr-xr-x 2 wangdehe ias_admin         1  4月 12 21:51 R1
drwxr-xr-x 2 wangdehe ias_admin         1  4月 12 21:59 R1_trimmed
-rw-r--r-- 1 wangdehe ias_admin     15022  4月 12 21:51 script
-rw-r--r-- 1 wangdehe ias_admin       471  4月 12 21:51 script.background
-rw-r--r-- 1 wangdehe ias_admin       350  4月 12 21:51 script.submit
-rw-r--r-- 1 wangdehe ias_admin         0  4月 12 21:51 stderr
-rw-r--r-- 1 wangdehe ias_admin         0  4月 12 21:51 stderr.background
-rw-r--r-- 1 wangdehe ias_admin     10326  4月 12 22:32 stdout
-rw-r--r-- 1 wangdehe ias_admin     10333  4月 12 22:32 stdout.background
-rw-r--r-- 1 wangdehe ias_admin       207  4月 12 21:51 write_tsv_b9349cb0e3563b467dcaa20e89670ada.tmp
[2021-04-12 22:32:02,039 INFO] All done.

STDERR file is empty.

leepc12 commented 3 years ago

Can you post a STDOUT file in any of the task (e.g. align_ctl)?

sqreb commented 3 years ago

Can you post a STDOUT file in any of the task (e.g. align_ctl)?

STDOUT file for align_ctl task

[2021-04-12 21:50:15,109 INFO] ['/home/wangdehe/software/miniconda3/envs/encode-chip-seq-pipeline/bin/encode_task_merge_fastq.py', '/
beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25/chip/d59b6acc-4361-477e-8ed4-9b8054e51c9b/call
-align_ctl/shard-0/execution/write_tsv_d93a107b21131706aa52e75496aa597c.tmp', '--paired-end', '--nth', '6']
[2021-04-12 21:50:15,109 INFO] Initializing and making output directory...
[2021-04-12 21:50:15,109 INFO] Merging fastqs...
[2021-04-12 21:50:15,109 INFO] R1 to be merged: ['/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/M
YB25/chip/d59b6acc-4361-477e-8ed4-9b8054e51c9b/call-align_ctl/shard-0/inputs/254997668/MYB25.Input.trim.R1.fastq.gz']
[2021-04-12 21:50:15,117 INFO] run_shell_cmd: PID=278289, PGID=278289, CMD=zcat -f /beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v
3/DAP_seq/encode_chip2_pipeline/MYB25/chip/d59b6acc-4361-477e-8ed4-9b8054e51c9b/call-align_ctl/shard-0/inputs/254997668/MYB25.Input.t
rim.R1.fastq.gz | gzip -nc > R1/MYB25.Input.trim.R1.fastq.gz
[2021-04-12 21:57:35,843 INFO] PID=278289, PGID=278289, RC=0, DURATION_SEC=440.7
STDERR=
STDOUT=
[2021-04-12 21:57:35,843 INFO] R2 to be merged: ['/beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/M
YB25/chip/d59b6acc-4361-477e-8ed4-9b8054e51c9b/call-align_ctl/shard-0/inputs/254997668/MYB25.Input.trim.R2.fastq.gz']
[2021-04-12 21:57:35,852 INFO] run_shell_cmd: PID=291939, PGID=291939, CMD=zcat -f /beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v
3/DAP_seq/encode_chip2_pipeline/MYB25/chip/d59b6acc-4361-477e-8ed4-9b8054e51c9b/call-align_ctl/shard-0/inputs/254997668/MYB25.Input.t
rim.R2.fastq.gz | gzip -nc > R2/MYB25.Input.trim.R2.fastq.gz
[2021-04-12 22:05:36,123 INFO] PID=291939, PGID=291939, RC=0, DURATION_SEC=480.3
STDERR=
STDOUT=
[2021-04-12 22:05:36,124 INFO] List all files in output directory...
[2021-04-12 22:05:36,129 INFO] run_shell_cmd: PID=14201, PGID=14201, CMD=ls -l 
[2021-04-12 22:05:36,150 INFO] PID=14201, PGID=14201, RC=0, DURATION_SEC=0.0
STDERR=
STDOUT=total 22
drwxr-xr-x 2 wangdehe ias_admin     1  4月 12 21:50 R1
drwxr-xr-x 2 wangdehe ias_admin     1  4月 12 21:57 R2
-rw-r--r-- 1 wangdehe ias_admin 14999  4月 12 21:50 script
-rw-r--r-- 1 wangdehe ias_admin   463  4月 12 21:50 script.background
-rw-r--r-- 1 wangdehe ias_admin   346  4月 12 21:50 script.submit
-rw-r--r-- 1 wangdehe ias_admin     0  4月 12 21:50 stderr
-rw-r--r-- 1 wangdehe ias_admin     0  4月 12 21:50 stderr.background
-rw-r--r-- 1 wangdehe ias_admin  1997  4月 12 22:05 stdout
-rw-r--r-- 1 wangdehe ias_admin  2004  4月 12 22:05 stdout.background
-rw-r--r-- 1 wangdehe ias_admin   398  4月 12 21:50 write_tsv_d93a107b21131706aa52e75496aa597c.tmp
[2021-04-12 22:05:36,150 INFO] All done.
[2021-04-12 22:05:36,254 INFO] ['/home/wangdehe/software/miniconda3/envs/encode-chip-seq-pipeline/bin/encode_task_bowtie2.py', '/beeg
fs/zhoulab/dhwang/CottonSingleCell/analysis_v3/DAP_seq/encode_chip2_pipeline/MYB25/chip/d59b6acc-4361-477e-8ed4-9b8054e51c9b/call-ali
gn_ctl/shard-0/inputs/456323938/Ghirsutum_genome.chrM.fasta.tar.gz', 'R1/MYB25.Input.trim.R1.fastq.gz', 'R2/MYB25.Input.trim.R2.fastq
.gz', '--paired-end', '--mem-gb', '6.179190651997924', '--nth', '6']
[2021-04-12 22:05:36,254 INFO] Initializing and making output directory...
[2021-04-12 22:05:36,254 INFO] Unpacking bowtie2 index tar...
[2021-04-12 22:05:36,258 INFO] run_shell_cmd: PID=14211, PGID=14211, CMD=tar zxvf /beegfs/zhoulab/dhwang/CottonSingleCell/analysis_v3
/DAP_seq/encode_chip2_pipeline/MYB25/chip/d59b6acc-4361-477e-8ed4-9b8054e51c9b/call-align_ctl/shard-0/inputs/456323938/Ghirsutum_geno
me.chrM.fasta.tar.gz --no-same-owner -C .
[2021-04-12 22:06:34,681 INFO] PID=14211, PGID=14211, RC=0, DURATION_SEC=58.4
STDERR=
STDOUT=Ghirsutum_genome.chrM.fasta.1.bt2
Ghirsutum_genome.chrM.fasta.2.bt2
Ghirsutum_genome.chrM.fasta.3.bt2
Ghirsutum_genome.chrM.fasta.4.bt2
Ghirsutum_genome.chrM.fasta.rev.1.bt2
Ghirsutum_genome.chrM.fasta.rev.2.bt2
[2021-04-12 22:06:34,683 INFO] Running bowtie2...
[2021-04-12 22:06:34,690 INFO] run_shell_cmd: PID=16297, PGID=16297, CMD=bowtie2  -X2000  --mm --threads 6 -x Ghirsutum_genome.chrM.f
asta -1 R1/MYB25.Input.trim.R1.fastq.gz -2 R2/MYB25.Input.trim.R2.fastq.gz | samtools view -1 -S /dev/stdin > MYB25.Input.trim.R1.bam
[2021-04-12 23:45:10,300 INFO] PID=16297, PGID=16297, RC=0, DURATION_SEC=5915.6
STDERR=9407957 reads; of these:
  9407957 (100.00%) were paired; of these:
    366983 (3.90%) aligned concordantly 0 times
    2969937 (31.57%) aligned concordantly exactly 1 time
    6071037 (64.53%) aligned concordantly >1 times
    ----
    366983 pairs aligned concordantly 0 times; of these:
      13173 (3.59%) aligned discordantly 1 time
    ----
    353810 pairs aligned 0 times concordantly or discordantly; of these:
      707620 mates make up the pairs; of these:
        197083 (27.85%) aligned 0 times
        105409 (14.90%) aligned exactly 1 time
        405128 (57.25%) aligned >1 times
98.95% overall alignment rate
STDOUT=
[2021-04-12 23:45:10,309 INFO] run_shell_cmd: PID=14711, PGID=14711, CMD=samtools sort MYB25.Input.trim.R1.bam -o MYB25.Input.trim.R1.srt.bam -T MYB25.Input.trim.R1 -@ 5 -m 768M 
[2021-04-12 23:46:12,525 INFO] PID=14711, PGID=14711, RC=0, DURATION_SEC=62.2
STDERR=[bam_sort_core] merging from 5 files and 5 in-memory blocks...
STDOUT=
[2021-04-12 23:46:12,531 INFO] run_shell_cmd: PID=16049, PGID=16049, CMD=rm -f MYB25.Input.trim.R1.bam
[2021-04-12 23:46:12,747 INFO] PID=16049, PGID=16049, RC=0, DURATION_SEC=0.2
STDERR=
STDOUT=
[2021-04-12 23:46:12,747 INFO] Removing temporary files...
['Ghirsutum_genome.chrM.fasta*']
[2021-04-12 23:46:12,752 INFO] run_shell_cmd: PID=16051, PGID=16051, CMD=rm -f Ghirsutum_genome.chrM.fasta*
[2021-04-12 23:46:13,959 INFO] PID=16051, PGID=16051, RC=0, DURATION_SEC=1.2
STDERR=
STDOUT=
[2021-04-12 23:46:13,959 INFO] Checking if BAM file is empty...
[2021-04-12 23:46:13,964 INFO] run_shell_cmd: PID=16109, PGID=16109, CMD=samtools view -c MYB25.Input.trim.R1.srt.bam -@ 5 
[2021-04-12 23:46:18,877 INFO] PID=16109, PGID=16109, RC=0, DURATION_SEC=4.9
STDERR=
STDOUT=18815914
[2021-04-12 23:46:18,877 INFO] List all files in output directory...
[2021-04-12 23:46:18,882 INFO] run_shell_cmd: PID=16239, PGID=16239, CMD=ls -l 
[2021-04-12 23:46:18,900 INFO] PID=16239, PGID=16239, RC=0, DURATION_SEC=0.0
STDERR=
STDOUT=total 1381867
-rw-r--r-- 1 wangdehe ias_admin 1415000458  4月 12 23:46 MYB25.Input.trim.R1.srt.bam
drwxr-xr-x 2 wangdehe ias_admin          1  4月 12 21:50 R1
drwxr-xr-x 2 wangdehe ias_admin          1  4月 12 21:57 R2
-rw-r--r-- 1 wangdehe ias_admin      14999  4月 12 21:50 script
-rw-r--r-- 1 wangdehe ias_admin        463  4月 12 21:50 script.background
-rw-r--r-- 1 wangdehe ias_admin        346  4月 12 21:50 script.submit
-rw-r--r-- 1 wangdehe ias_admin          0  4月 12 21:50 stderr
-rw-r--r-- 1 wangdehe ias_admin          0  4月 12 21:50 stderr.background
-rw-r--r-- 1 wangdehe ias_admin       6329  4月 12 23:46 stdout
-rw-r--r-- 1 wangdehe ias_admin       6336  4月 12 23:46 stdout.background
-rw-r--r-- 1 wangdehe ias_admin        398  4月 12 21:50 write_tsv_d93a107b21131706aa52e75496aa597c.tmp
[2021-04-12 23:46:18,901 INFO] All done.
[2021-04-12 23:46:19,011 INFO] ['/home/wangdehe/software/miniconda3/envs/encode-chip-seq-pipeline/bin/encode_task_post_align.py', 'R1/MYB25.Input.trim.R1.fastq.gz', 'MYB25.Input.trim.R1.srt.bam', '--mito-chr-name', 'chrM', '--mem-gb', '6.179190651997924', '--nth', '6']
[2021-04-12 23:46:19,011 INFO] Initializing and making output directory...
[2021-04-12 23:46:19,011 INFO] Generating read length file...
[2021-04-12 23:46:26,394 INFO] Running samtools index...
[2021-04-12 23:46:26,399 INFO] run_shell_cmd: PID=16415, PGID=16415, CMD=samtools index MYB25.Input.trim.R1.srt.bam -@ 6 
[2021-04-12 23:47:07,210 INFO] PID=16415, PGID=16415, RC=0, DURATION_SEC=40.8
STDERR=
STDOUT=
[2021-04-12 23:47:07,211 INFO] SAMstats on raw BAM...
[2021-04-12 23:47:07,217 INFO] run_shell_cmd: PID=17494, PGID=17494, CMD=samtools sort -n MYB25.Input.trim.R1.srt.bam -T MYB25.Input.trim.R1.srt.tmp -@ 5 -m 768M  -O sam | SAMstats --sorted_sam_file - --outf MYB25.Input.trim.R1.srt.samstats.qc
[2021-04-12 23:53:00,744 INFO] PID=17494, PGID=17494, RC=0, DURATION_SEC=353.5
STDERR=[bam_sort_core] merging from 5 files and 5 in-memory blocks...
STDOUT=starting flag calculation...
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
1000000
1100000
1200000
1300000
1400000
1500000
1600000
1700000
1800000
1900000
2000000
2100000
2200000
2300000
2400000
2500000
2600000
2700000
2800000
2900000
3000000
3100000
3200000
3300000
3400000
3500000
3600000
3700000
3800000
3900000
4000000
4100000
4200000
4300000
4400000
4500000
4600000
4700000
4800000
4900000
5000000
5100000
5200000
5300000
5400000
5500000
5600000
5700000
5800000
5900000
6000000
6100000
6200000
6300000
6400000
6500000
6600000
6700000
6800000
6900000
7000000
7100000
7200000
7300000
7400000
7500000
7600000
7700000
7800000
7900000
8000000
8100000
8200000
8300000
8400000
8500000
8600000
8700000
8800000
8900000
9000000
9100000
9200000
9300000
9400000
9500000
9600000
9700000
9800000
9900000
10000000
10100000
10200000
10300000
10400000
10500000
10600000
10700000
10800000
10900000
11000000
11100000
11200000
11300000
11400000
11500000
11600000
11700000
11800000
11900000
12000000
12100000
12200000
12300000
12400000
12500000
12600000
12700000
12800000
12900000
13000000
13100000
13200000
13300000
13400000
13500000
13600000
13700000
13800000
13900000
14000000
14100000
14200000
14300000
14400000
14500000
14600000
14700000
14800000
14900000
15000000
15100000
15200000
15300000
15400000
15500000
15600000
15700000
15800000
15900000
16000000
16100000
16200000
16300000
16400000
16500000
16600000
16700000
16800000
16900000
17000000
17100000
17200000
17300000
17400000
17500000
17600000
17700000
17800000
17900000
18000000
18100000
18200000
18300000
18400000
18500000
18600000
18700000
18800000
finished parsing lines, summarizing...
[2021-04-12 23:53:00,744 INFO] List all files in output directory...
[2021-04-12 23:53:00,751 INFO] run_shell_cmd: PID=30715, PGID=30715, CMD=ls -l 
[2021-04-12 23:53:00,778 INFO] PID=30715, PGID=30715, RC=0, DURATION_SEC=0.0
STDERR=
STDOUT=total 1383727
-rw-r--r-- 1 wangdehe ias_admin          3  4月 12 23:46 MYB25.Input.trim.R1.read_length.txt
-rw-r--r-- 1 wangdehe ias_admin 1415000458  4月 12 23:46 MYB25.Input.trim.R1.srt.bam
-rw-r--r-- 1 wangdehe ias_admin    1895608  4月 12 23:47 MYB25.Input.trim.R1.srt.bam.bai
-rw-r--r-- 1 wangdehe ias_admin        393  4月 12 23:52 MYB25.Input.trim.R1.srt.samstats.qc
drwxr-xr-x 2 wangdehe ias_admin          1  4月 12 21:50 R1
drwxr-xr-x 2 wangdehe ias_admin          1  4月 12 21:57 R2
-rw-r--r-- 1 wangdehe ias_admin      14999  4月 12 21:50 script
-rw-r--r-- 1 wangdehe ias_admin        463  4月 12 21:50 script.background
-rw-r--r-- 1 wangdehe ias_admin        346  4月 12 21:50 script.submit
-rw-r--r-- 1 wangdehe ias_admin          0  4月 12 21:50 stderr
-rw-r--r-- 1 wangdehe ias_admin          0  4月 12 21:50 stderr.background
-rw-r--r-- 1 wangdehe ias_admin      10221  4月 12 23:53 stdout
-rw-r--r-- 1 wangdehe ias_admin      10228  4月 12 23:53 stdout.background
-rw-r--r-- 1 wangdehe ias_admin        398  4月 12 21:50 write_tsv_d93a107b21131706aa52e75496aa597c.tmp
[2021-04-12 23:53:00,778 INFO] All done.
dcheng1 commented 3 years ago

I had the same error. Did you figure out a way to solve it?

paulineauffret commented 3 years ago

Hi, I encountered a similar error with my ChiP-seq data. The alignment step failed. It keeps running for days, caper output saying 2021-09-19 10:58:26,309 cromwell-system-akka.dispatchers.backend-dispatcher-3577 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(15cfae78)chip.align:1:1]: Status change from WaitingForReturnCode to Done But every STDOUT file for the align tasks looks fine (ending with All done.). Surprinsingly, this problem only occurs with one of my dataset. If you figured out a trail to solve that issue I'm really interested in hearing it. Thank you very much !

"submission": "2021-09-09T10:31:33.778Z",
    "status": "Failed",
    "failures": [
        {
            "message": "Workflow failed",
            "causedBy": [
                {
                    "message": "Call input and runtime attributes evaluation failed for align_ctl",
                    "causedBy": [
                        {
                            "message": "Failed to evaluate input 'tmp_fastqs' (reason 1 of 1): transpose requires all collections have the same size",
                            "causedBy": []
                        }
                    ]
                },
                {
                    "causedBy": [
                        {
                            "message": "Failed to evaluate input 'tmp_fastqs' (reason 1 of 1): transpose requires all collections have the same size",
                            "causedBy": []
                        }
                    ],
                    "message": "Call input and runtime attributes evaluation failed for align_ctl"
                }
            ]
        }
    ],
    "end": "2021-09-19T08:58:28.169Z",
    "start": "2021-09-09T10:31:34.367Z"
leepc12 commented 3 years ago

I will release new Caper + pipeline today to fix all Conda and SLURM issues. Make sure that you update both Caper and pipeline.

If you use Conda, then uninstall and reinstall pipeilne's Conda environment. Explicitly define --conda in the command line arguments: caper run ... --conda

I recommend using Singularity caper run ... --singularity if you have it installed on your system.

Please try with new Caper + pipeline and let me know if it works.