Closed Barbarak17 closed 4 years ago
leepc12
commented
4 years ago If you mix up different sequential endedness (SE or PE) between exp reps and controls, then you need to define control's endedness. Add the following to your input JSON.
"chip.ctl_paired_end" : false,
Barbarak17
commented
4 years ago Thanks, I fixed that but now I am getting a different error:
"submission": "2020-07-15T13:12:49.709Z",
"status": "Failed",
"failures": [
{
"causedBy": [
{
"message": "Job chip.filter_ctl:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
}
],
"message": "Workflow failed"
}
],
"end": "2020-07-15T20:21:08.306Z",
"start": "2020-07-15T13:12:49.821Z"
}this is the full metadata.json
{
"workflowName": "chip",
"workflowProcessingEvents": [
{
"cromwellId": "cromid-7739c3c",
"description": "Finished",
"timestamp": "2020-07-15T20:21:08.307Z",
"cromwellVersion": "47"
},
{
"timestamp": "2020-07-15T13:12:49.757Z",
"cromwellVersion": "47",
"description": "PickedUp",
"cromwellId": "cromid-7739c3c"
}
],
"actualWorkflowLanguageVersion": "1.0",
"submittedFiles": {
"workflow": "version 1.0\n\nworkflow chip {\n String pipeline_ver = 'v1.5.0'\n\n meta {\n author: 'Jin wook Lee (leepc12@gmail.com) at ENCODE-DCC'\n description: 'ENCODE TF/Histone ChIP-Seq pipeline'\n specification_document: 'https://docs.google.com/document/d/1lG_Rd7fnYgRpSIqrIfuVlAz2dW1VaSQThzk836Db99c/edit?usp=sharing'\n\n caper_docker: 'encodedcc/chip-seq-pipeline:v1.5.0'\n caper_singularity: 'docker://encodedcc/chip-seq-pipeline:v1.5.0'\n croo_out_def: 'https://storage.googleapis.com/encode-pipeline-output-definition/chip.croo.v4.json'\n\n parameter_group: {\n pipeline_metadata: {\n title: 'Pipeline metadata',\n description: 'Metadata for a pipeline (e.g. title and description).'\n },\n reference_genome: {\n title: 'Reference genome',\n description: 'Genome specific files. e.g. reference FASTA, bowtie2 index, chromosome sizes file.',\n help: 'Choose one chip.genome_tsv file that defines all genome specific parameters in it or define each genome specific parameter in input JSON to override those defined in genome TSV file. If you use Caper then use https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v1/[GENOME]_caper.tsv. Caper will automatically download/install all files defined in such TSV. Otherwise download genome TSV file by using a shell script (scripts/download_genome_data.sh [GENOME] [DEST_DIR]). Supported genomes are hg38, hg19, mm10 and mm9. See pipeline documentation if you want to build genome database from your own FASTA file. If some genome data are missing then analyses using such data will be skipped.'\n },\n input_genomic_data: {\n title: 'Input genomic data',\n description: 'Genomic input files for experiment.',\n help: 'Pipeline can start with any types of experiment data (e.g. FASTQ, BAM, NODUP_BAM, TAG-ALIGN, PEAK). Choose one type and leave others empty. FASTQs have a variable for each biological replicate. e.g. chip.fastqs_rep1_R1 and chip.fastqs_rep2_R1. You can define up to 10 experiment replicates. For other types, there is an array to define file for each biological replicate. e.g. chip.bams: [\"rep1.bam\", \"rep1.bam\"]. Define sequential endedness with chip.paired_end, if you have mixed SE and PE replicates then define chip.paired_ends instead for each replicate. e.g. chip.paired_ends: [false, true].'\n },\n input_genomic_data_control: {\n title: 'Input genomic data (control)',\n description: 'Genomic input files for control. TF ChIP-seq requires control for peak calling but histone ChIP-seq does not.',\n help: 'Pipeline can start with any types of control data (e.g. FASTQ, BAM, NODUP_BAM, TAG-ALIGN). Choose one type and leave others empty. FASTQs have a variable for each control replicate. e.g. chip.ctl_fastqs_rep1_R1 and chip.ctl_fastqs_rep2_R1. You can define up to 10 control replicates. For other types, there is an array to define file for each control replicate. e.g. chip.ctl_bams: [\"ctl1.bam\", \"ctl1.bam\"]. Define sequential endedness with chip.ctl_paired_end, if you have mixed SE and PE control replicates then define chip.ctl_paired_ends instead for each replicate. e.g. chip.ctl_paired_ends: [false, true]. If none of these are defined, pipeline will use chip.paired_end for controls.'\n },\n pipeline_parameter: {\n title: 'Pipeline parameter',\n description: 'Pipeline type and flags to turn on/off analyses.',\n help: 'Use chip.align_only to align FASTQs without peak calling.'\n },\n alignment: {\n title: 'Alignment',\n description: 'Parameters for alignment.',\n help: 'Pipeline can crop FASTQs (chip.crop_length > 0) with tolerance (chip.crop_length_tol) before mapping.'\n },\n peak_calling: {\n title: 'Peak calling',\n description: 'Parameters for peak calling.',\n help: 'This group includes statistical thresholds for peak-calling or post-peak-calling analyses: p-val, FDR, IDR. It also include parameters for control choosing/subsampling. All control replicates are pooled and pooled control is used for peak calling against each experiment replicate by default (see chip.always_use_pooled_ctl). Pipeline compares read depth of experiment replicate and a chosen control. It also compare read depth of controls. If control is too deep then it is subsampled.'\n },\n resource_parameter: {\n title: 'Resource parameter',\n description: 'Number of CPUs (threads), max. memory and walltime for tasks.',\n help: 'Resource settings are used for determining an instance type on cloud backends (e.g. GCP, AWS) and used for submitting tasks to a cluster engine (e.g. SLURM, SGE, ...). Walltime (chip.*_time_hr) is only used for cluster engines. Other tasks default to use 1 CPU and 4GB of memory.'\n }\n }\n }\n input {\n # group: pipeline_metadata\n String title = 'Untitled'\n String description = 'No description'\n\n # group: reference_genome\n File? genome_tsv\n String? genome_name\n File? ref_fa\n File? bwa_idx_tar\n File? bowtie2_idx_tar\n File? chrsz\n File? blacklist\n File? blacklist2\n String? mito_chr_name\n String? regex_bfilt_peak_chr_name\n String? gensz\n\n # group: input_genomic_data\n Boolean? paired_end\n Array[Boolean] paired_ends = []\n Array[File] fastqs_rep1_R1 = []\n Array[File] fastqs_rep1_R2 = []\n Array[File] fastqs_rep2_R1 = []\n Array[File] fastqs_rep2_R2 = []\n Array[File] fastqs_rep3_R1 = []\n Array[File] fastqs_rep3_R2 = []\n Array[File] fastqs_rep4_R1 = []\n Array[File] fastqs_rep4_R2 = []\n Array[File] fastqs_rep5_R1 = []\n Array[File] fastqs_rep5_R2 = []\n Array[File] fastqs_rep6_R1 = []\n Array[File] fastqs_rep6_R2 = []\n Array[File] fastqs_rep7_R1 = []\n Array[File] fastqs_rep7_R2 = []\n Array[File] fastqs_rep8_R1 = []\n Array[File] fastqs_rep8_R2 = []\n Array[File] fastqs_rep9_R1 = []\n Array[File] fastqs_rep9_R2 = []\n Array[File] fastqs_rep10_R1 = []\n Array[File] fastqs_rep10_R2 = []\n Array[File?] bams = []\n Array[File?] nodup_bams = []\n Array[File?] tas = []\n Array[File?] peaks = []\n Array[File?] peaks_pr1 = []\n Array[File?] peaks_pr2 = []\n File? peak_ppr1\n File? peak_ppr2\n File? peak_pooled\n\n Boolean? ctl_paired_end\n Array[Boolean] ctl_paired_ends = []\n Array[File] ctl_fastqs_rep1_R1 = []\n Array[File] ctl_fastqs_rep1_R2 = []\n Array[File] ctl_fastqs_rep2_R1 = []\n Array[File] ctl_fastqs_rep2_R2 = []\n Array[File] ctl_fastqs_rep3_R1 = []\n Array[File] ctl_fastqs_rep3_R2 = []\n Array[File] ctl_fastqs_rep4_R1 = []\n Array[File] ctl_fastqs_rep4_R2 = []\n Array[File] ctl_fastqs_rep5_R1 = []\n Array[File] ctl_fastqs_rep5_R2 = []\n Array[File] ctl_fastqs_rep6_R1 = []\n Array[File] ctl_fastqs_rep6_R2 = []\n Array[File] ctl_fastqs_rep7_R1 = []\n Array[File] ctl_fastqs_rep7_R2 = []\n Array[File] ctl_fastqs_rep8_R1 = []\n Array[File] ctl_fastqs_rep8_R2 = []\n Array[File] ctl_fastqs_rep9_R1 = []\n Array[File] ctl_fastqs_rep9_R2 = []\n Array[File] ctl_fastqs_rep10_R1 = []\n Array[File] ctl_fastqs_rep10_R2 = []\n Array[File?] ctl_bams = []\n Array[File?] ctl_nodup_bams = []\n Array[File?] ctl_tas = []\n\n # group: pipeline_parameter\n String pipeline_type\n Boolean align_only = false\n Boolean true_rep_only = false\n Boolean enable_count_signal_track = false\n Boolean enable_jsd = true\n Boolean enable_gc_bias = true\n\n # group: alignment\n String aligner = 'bowtie2'\n Boolean use_bwa_mem_for_pe = false\n Int crop_length = 0\n Int crop_length_tol = 2\n Int xcor_trim_bp = 50\n Boolean use_filt_pe_ta_for_xcor = false\n String dup_marker = 'picard'\n Boolean no_dup_removal = false\n Int mapq_thresh = 30\n Array[String] filter_chrs = []\n Int subsample_reads = 0\n Int ctl_subsample_reads = 0\n Int xcor_subsample_reads = 15000000\n Int xcor_exclusion_range_min = -500\n Int? xcor_exclusion_range_max\n\n # group: peak_calling\n Int ctl_depth_limit = 200000000\n Float exp_ctl_depth_ratio_limit = 5.0\n Array[Int?] fraglen = []\n String? peak_caller\n Boolean always_use_pooled_ctl = true\n Float ctl_depth_ratio = 1.2\n Int? cap_num_peak\n Float pval_thresh = 0.01\n Float fdr_thresh = 0.01\n Float idr_thresh = 0.05\n\n # group: resource_parameter\n Int align_cpu = 4\n Int align_mem_mb = 20000\n Int align_time_hr = 48\n String align_disks = 'local-disk 400 HDD'\n\n Int filter_cpu = 2\n Int filter_mem_mb = 20000\n Int filter_time_hr = 24\n String filter_disks = 'local-disk 400 HDD'\n\n Int bam2ta_cpu = 2\n Int bam2ta_mem_mb = 10000\n Int bam2ta_time_hr = 6\n String bam2ta_disks = 'local-disk 100 HDD'\n\n Int spr_mem_mb = 16000\n\n Int jsd_cpu = 2\n Int jsd_mem_mb = 12000\n Int jsd_time_hr = 6\n String jsd_disks = 'local-disk 200 HDD'\n\n Int xcor_cpu = 2\n Int xcor_mem_mb = 16000 \n Int xcor_time_hr = 24\n String xcor_disks = 'local-disk 100 HDD'\n\n Int macs2_signal_track_mem_mb = 16000\n Int macs2_signal_track_time_hr = 24\n String macs2_signal_track_disks = 'local-disk 400 HDD'\n\n Int call_peak_cpu = 2\n Int call_peak_mem_mb = 16000\n Int call_peak_time_hr = 72\n String call_peak_disks = 'local-disk 200 HDD'\n\n String? align_trimmomatic_java_heap\n String? filter_picard_java_heap\n String? gc_bias_picard_java_heap\n }\n\n parameter_meta {\n title: {\n description: 'Experiment title.',\n group: 'pipeline_metadata',\n example: 'ENCSR936XTK (subsampled 1/50)'\n }\n description: {\n description: 'Experiment description.',\n group: 'pipeline_metadata',\n example: 'ZNF143 ChIP-seq on human GM12878 (subsampled 1/50)'\n }\n genome_tsv: {\n description: 'Reference genome database TSV.',\n group: 'reference_genome',\n help: 'This TSV files includes all genome specific parameters (e.g. reference FASTA, bowtie2 index). You can still invidiaully define any parameters in it. Parameters defined in input JSON will override those defined in genome TSV.',\n example: 'https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v1/hg38_caper.tsv'\n }\n genome_name: {\n description: 'Genome name.',\n group: 'reference_genome'\n }\n ref_fa: {\n description: 'Reference FASTA file.',\n group: 'reference_genome'\n }\n ref_mito_fa: {\n description: 'Reference FASTA file (mitochondrial reads only).',\n group: 'reference_genome'\n }\n bowtie2_idx_tar: {\n description: 'BWA index TAR file.',\n group: 'reference_genome'\n }\n bowtie2_mito_idx_tar: {\n description: 'BWA index TAR file (mitochondrial reads only).',\n group: 'reference_genome'\n }\n chrsz: {\n description: '2-col chromosome sizes file.',\n group: 'reference_genome'\n }\n blacklist: {\n description: 'Blacklist file in BED format.',\n group: 'reference_genome',\n help: 'Peaks will be filtered with this file.'\n }\n blacklist2: {\n description: 'Secondary blacklist file in BED format.',\n group: 'reference_genome',\n help: 'If it is defined, it will be merged with chip.blacklist. Peaks will be filtered with merged blacklist.'\n }\n mito_chr_name: {\n description: 'Mitochondrial chromosome name.',\n group: 'reference_genome',\n help: 'e.g. chrM, MT. Mitochondrial reads defined here will be filtered out during filtering BAMs in \"filter\" task.'\n }\n regex_bfilt_peak_chr_name: {\n description: 'Reg-ex for chromosomes to keep while filtering peaks.',\n group: 'reference_genome',\n help: 'Chromosomes defined here will be kept. All other chromosomes will be filtered out in .bfilt. peak file. This is done along with blacklist filtering peak file.'\n }\n gensz: {\n description: 'Genome sizes. \"hs\" for human, \"mm\" for mouse or sum of 2nd columnin chromosome sizes file.',\n group: 'reference_genome'\n }\n paired_end: {\n description: 'Sequencing endedness.',\n group: 'input_genomic_data',\n help: 'Setting this on means that all replicates are paired ended. For mixed samples, use chip.paired_ends array instead.',\n example: true\n }\n paired_ends: {\n description: 'Sequencing endedness array (for mixed SE/PE datasets).',\n group: 'input_genomic_data',\n help: 'Whether each biological replicate is paired ended or not.'\n }\n fastqs_rep1_R1: {\n description: 'Read1 FASTQs to be merged for a biological replicate 1.',\n group: 'input_genomic_data',\n help: 'Define if you want to start pipeline from FASTQs files. Pipeline can start from any type of inputs (e.g. FASTQs, BAMs, ...). Choose one type and fill paramters for that type and leave other undefined. Especially for FASTQs, we have individual variable for each biological replicate to allow FASTQs of technical replicates can be merged. Make sure that they are consistent with read2 FASTQs (chip.fastqs_rep1_R2). These FASTQs are usually technical replicates to be merged.',\n example: [\n 'https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R1.subsampled.50.fastq.gz'\n ]\n }\n fastqs_rep1_R2: {\n description: 'Read2 FASTQs to be merged for a biological replicate 1.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.fastqs_rep1_R1). These FASTQs are usually technical replicates to be merged.',\n example: [\n 'https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R2.subsampled.50.fastq.gz'\n ]\n }\n fastqs_rep2_R1: {\n description: 'Read1 FASTQs to be merged for a biological replicate 2.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.fastqs_rep2_R2). These FASTQs are usually technical replicates to be merged.',\n example: [\n 'https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R1.subsampled.50.fastq.gz'\n ]\n }\n fastqs_rep2_R2: {\n description: 'Read2 FASTQs to be merged for a biological replicate 2.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.fastqs_rep2_R1). These FASTQs are usually technical replicates to be merged.',\n example: [\n 'https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R2.subsampled.50.fastq.gz'\n ]\n }\n fastqs_rep3_R1: {\n description: 'Read1 FASTQs to be merged for a biological replicate 3.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.fastqs_rep3_R2). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep3_R2: {\n description: 'Read2 FASTQs to be merged for a biological replicate 3.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.fastqs_rep3_R1). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep4_R1: {\n description: 'Read1 FASTQs to be merged for a biological replicate 4.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.fastqs_rep4_R2). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep4_R2: {\n description: 'Read2 FASTQs to be merged for a biological replicate 4.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.fastqs_rep4_R1). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep5_R1: {\n description: 'Read1 FASTQs to be merged for a biological replicate 5.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.fastqs_rep5_R2). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep5_R2: {\n description: 'Read2 FASTQs to be merged for a biological replicate 5.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.fastqs_rep5_R1). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep6_R1: {\n description: 'Read1 FASTQs to be merged for a biological replicate 6.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.fastqs_rep6_R2). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep6_R2: {\n description: 'Read2 FASTQs to be merged for a biological replicate 6.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.fastqs_rep6_R1). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep7_R1: {\n description: 'Read1 FASTQs to be merged for a biological replicate 7.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.fastqs_rep7_R2). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep7_R2: {\n description: 'Read2 FASTQs to be merged for a biological replicate 7.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.fastqs_rep7_R1). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep8_R1: {\n description: 'Read1 FASTQs to be merged for a biological replicate 8.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.fastqs_rep8_R2). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep8_R2: {\n description: 'Read2 FASTQs to be merged for a biological replicate 8.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.fastqs_rep8_R1). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep9_R1: {\n description: 'Read1 FASTQs to be merged for a biological replicate 9.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.fastqs_rep9_R2). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep9_R2: {\n description: 'Read2 FASTQs to be merged for a biological replicate 9.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.fastqs_rep9_R1). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep10_R1: {\n description: 'Read1 FASTQs to be merged for a biological replicate 10.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.fastqs_rep10_R2). These FASTQs are usually technical replicates to be merged.'\n }\n fastqs_rep10_R2: {\n description: 'Read2 FASTQs to be merged for a biological replicate 10.',\n group: 'input_genomic_data',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.fastqs_rep10_R1). These FASTQs are usually technical replicates to be merged.'\n }\n bams: {\n description: 'List of unfiltered/raw BAM files for each biological replicate.',\n group: 'input_genomic_data',\n help: 'Define if you want to start pipeline from BAM files. Unfiltered/raw BAM file generated from aligner (e.g. bowtie2). Each entry for each biological replicate. e.g. [rep1.bam, rep2.bam, rep3.bam, ...].'\n }\n nodup_bams: {\n description: 'List of filtered/deduped BAM files for each biological replicate',\n group: 'input_genomic_data',\n help: 'Define if you want to start pipeline from filtered BAM files. Filtered/deduped BAM file. Each entry for each biological replicate. e.g. [rep1.nodup.bam, rep2.nodup.bam, rep3.nodup.bam, ...].'\n }\n tas: {\n description: 'List of TAG-ALIGN files for each biological replicate.',\n group: 'input_genomic_data',\n help: 'Define if you want to start pipeline from TAG-ALIGN files. TAG-ALIGN is in a 6-col BED format. It is a simplified version of BAM. Each entry for each biological replicate. e.g. [rep1.tagAlign.gz, rep2.tagAlign.gz, ...].'\n }\n peaks: {\n description: 'List of NARROWPEAK files (not blacklist filtered) for each biological replicate.',\n group: 'input_genomic_data',\n help: 'Define if you want to start pipeline from PEAK files. Each entry for each biological replicate. e.g. [rep1.narrowPeak.gz, rep2.narrowPeak.gz, ...]. Define other PEAK parameters (e.g. chip.peaks_pr1, chip.peak_pooled) according to your flag settings (e.g. chip.true_rep_only) and number of replicates. If you have more than one replicate then define chip.peak_pooled, chip.peak_ppr1 and chip.peak_ppr2. If chip.true_rep_only flag is on then do not define any parameters (chip.peaks_pr1, chip.peaks_pr2, chip.peak_ppr1 and chip.peak_ppr2) related to pseudo replicates.'\n }\n peaks_pr1: {\n description: 'List of NARROWPEAK files (not blacklist filtered) for pseudo-replicate 1 of each biological replicate.',\n group: 'input_genomic_data',\n help: 'Define if you want to start pipeline from PEAK files. Define if chip.true_rep_only flag is off.'\n }\n peaks_pr2: {\n description: 'List of NARROWPEAK files (not blacklist filtered) for pseudo-replicate 2 of each biological replicate.',\n group: 'input_genomic_data',\n help: 'Define if you want to start pipeline from PEAK files. Define if chip.true_rep_only flag is off.'\n }\n peak_pooled: {\n description: 'NARROWPEAK file for pooled true replicate.',\n group: 'input_genomic_data',\n help: 'Define if you want to start pipeline from PEAK files. Define if you have multiple biological replicates. Pooled true replicate means analysis on pooled biological replicates.'\n }\n peak_ppr1: {\n description: 'NARROWPEAK file for pooled pseudo replicate 1.',\n group: 'input_genomic_data',\n help: 'Define if you want to start pipeline from PEAK files. Define if you have multiple biological replicates and chip.true_rep_only flag is off. PPR1 means analysis on pooled 1st pseudo replicates. Each biological replicate is shuf/split into two pseudos. This is a pooling of each replicate\\'s 1st pseudos.'\n }\n peak_ppr2: {\n description: 'NARROWPEAK file for pooled pseudo replicate 2.',\n group: 'input_genomic_data',\n help: 'Define if you want to start pipeline from PEAK files. Define if you have multiple biological replicates and chip.true_rep_only flag is off. PPR1 means analysis on pooled 2nd pseudo replicates. Each biological replicate is shuf/split into two pseudos. This is a pooling of each replicate\\'s 2nd pseudos.'\n }\n\n ctl_paired_end: {\n description: 'Sequencing endedness for all controls.',\n group: 'input_genomic_data_control',\n help: 'Setting this on means that all control replicates are paired ended. For mixed controls, use chip.ctl_paired_ends array instead.'\n }\n ctl_paired_ends: {\n description: 'Sequencing endedness array for mixed SE/PE controls.',\n group: 'input_genomic_data_control',\n help: 'Whether each control replicate is paired ended or not.'\n }\n ctl_fastqs_rep1_R1: {\n description: 'Read1 FASTQs to be merged for a control replicate 1.',\n group: 'input_genomic_data_control',\n help: 'Define if you want to start pipeline from FASTQs files. Pipeline can start from any type of controls (e.g. FASTQs, BAMs, ...). Choose one type and fill paramters for that type and leave other undefined. Make sure that they are consistent with read2 FASTQs (chip.ctl_fastqs_rep1_R2).',\n example: [\n 'https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R1.subsampled.80.fastq.gz'\n ]\n }\n ctl_fastqs_rep1_R2: {\n description: 'Read2 FASTQs to be merged for a control replicate 1.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.ctl_fastqs_rep1_R1). These FASTQs are usually technical replicates to be merged.',\n example: [\n 'https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R2.subsampled.80.fastq.gz'\n ]\n }\n ctl_fastqs_rep2_R1: {\n description: 'Read1 FASTQs to be merged for a control replicate 2.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.ctl_fastqs_rep2_R2). These FASTQs are usually technical replicates to be merged.',\n example: [\n 'https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R1.subsampled.80.fastq.gz'\n ]\n }\n ctl_fastqs_rep2_R2: {\n description: 'Read2 FASTQs to be merged for a control replicate 2.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.ctl_fastqs_rep2_R1). These FASTQs are usually technical replicates to be merged.',\n example: [\n 'https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R2.subsampled.80.fastq.gz'\n ]\n }\n ctl_fastqs_rep3_R1: {\n description: 'Read1 FASTQs to be merged for a control replicate 3.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.ctl_fastqs_rep3_R2). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep3_R2: {\n description: 'Read2 FASTQs to be merged for a control replicate 3.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.ctl_fastqs_rep3_R1). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep4_R1: {\n description: 'Read1 FASTQs to be merged for a control replicate 4.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.ctl_fastqs_rep4_R2). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep4_R2: {\n description: 'Read2 FASTQs to be merged for a control replicate 4.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.ctl_fastqs_rep4_R1). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep5_R1: {\n description: 'Read1 FASTQs to be merged for a control replicate 5.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.ctl_fastqs_rep5_R2). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep5_R2: {\n description: 'Read2 FASTQs to be merged for a control replicate 5.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.ctl_fastqs_rep5_R1). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep6_R1: {\n description: 'Read1 FASTQs to be merged for a control replicate 6.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.ctl_fastqs_rep6_R2). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep6_R2: {\n description: 'Read2 FASTQs to be merged for a control replicate 6.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.ctl_fastqs_rep6_R1). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep7_R1: {\n description: 'Read1 FASTQs to be merged for a control replicate 7.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.ctl_fastqs_rep7_R2). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep7_R2: {\n description: 'Read2 FASTQs to be merged for a control replicate 7.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.ctl_fastqs_rep7_R1). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep8_R1: {\n description: 'Read1 FASTQs to be merged for a control replicate 8.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.ctl_fastqs_rep8_R2). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep8_R2: {\n description: 'Read2 FASTQs to be merged for a control replicate 8.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.ctl_fastqs_rep8_R1). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep9_R1: {\n description: 'Read1 FASTQs to be merged for a control replicate 9.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.ctl_fastqs_rep9_R2). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep9_R2: {\n description: 'Read2 FASTQs to be merged for a control replicate 9.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.ctl_fastqs_rep9_R1). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep10_R1: {\n description: 'Read1 FASTQs to be merged for a control replicate 10.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read2 FASTQs (chip.ctl_fastqs_rep10_R2). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_fastqs_rep10_R2: {\n description: 'Read2 FASTQs to be merged for a control replicate 10.',\n group: 'input_genomic_data_control',\n help: 'Make sure that they are consistent with read1 FASTQs (chip.ctl_fastqs_rep10_R1). These FASTQs are usually technical replicates to be merged.'\n }\n ctl_bams: {\n description: 'List of unfiltered/raw BAM files for each control replicate.',\n group: 'input_genomic_data_control',\n help: 'Define if you want to start pipeline from BAM files. Unfiltered/raw BAM file generated from aligner (e.g. bowtie2). Each entry for each control replicate. e.g. [ctl1.bam, ctl2.bam, ctl3.bam, ...].'\n }\n ctl_nodup_bams: {\n description: 'List of filtered/deduped BAM files for each control replicate',\n group: 'input_genomic_data_control',\n help: 'Define if you want to start pipeline from filtered BAM files. Filtered/deduped BAM file. Each entry for each control replicate. e.g. [ctl1.nodup.bam, ctl2.nodup.bam, ctl3.nodup.bam, ...].'\n }\n ctl_tas: {\n description: 'List of TAG-ALIGN files for each biological replicate.',\n group: 'input_genomic_data_control',\n help: 'Define if you want to start pipeline from TAG-ALIGN files. TAG-ALIGN is in a 6-col BED format. It is a simplified version of BAM. Each entry for each control replicate. e.g. [ctl1.tagAlign.gz, ctl2.tagAlign.gz, ...].'\n }\n\n pipeline_type: {\n description: 'Pipeline type. tf for TF ChIP-Seq, histone for Histone ChIP-Seq or control for mapping controls only.',\n group: 'pipeline_parameter',\n help: 'Default peak caller is different for each type. spp For TF ChIP-Seq and macs2 for histone ChIP-Seq. Regardless of pipeline type, spp always requires controls but macs2 doesn\\'t. For control mode, chip.align_only is automatically turned on and cross-correlation analysis is disabled. Do not define ctl_* for control mode. Define fastqs_repX_RY instead.',\n choices: ['tf', 'histone', 'control'],\n example: 'tf'\n }\n align_only: {\n description: 'Align only mode.',\n group: 'pipeline_parameter',\n help: 'Reads will be aligned but there will be no peak-calling on them. It is turned on automatically if chip.pipeline_type is control.'\n }\n true_rep_only: {\n description: 'Disables all analyses related to pseudo-replicates.',\n group: 'pipeline_parameter',\n help: 'Pipeline generates 2 pseudo-replicate from one biological replicate. This flag turns off all analyses related to pseudos (with prefix/suffix pr, ppr).'\n }\n enable_count_signal_track: {\n description: 'Enables generation of count signal tracks.',\n group: 'pipeline_parameter'\n }\n enable_jsd: {\n description: 'Enables Jensen-Shannon Distance (JSD) plot generation.',\n group: 'pipeline_parameter'\n }\n enable_gc_bias: {\n description: 'Enables GC bias calculation.',\n group: 'pipeline_parameter'\n }\n\n aligner: {\n description: 'Aligner. bowtie2 or bwa',\n group: 'alignment',\n help: 'It is bowtie2 by default.',\n choices: ['bowtie2', 'bwa'],\n example: 'bowtie2'\n }\n use_bwa_mem_for_pe: {\n description: 'For paired end dataset with read length >= 70bp, use bwa mem instead of bwa aln.',\n group: 'alignment',\n help: 'Use it only for paired end reads >= 70bp.'\n }\n crop_length: {\n description: 'Crop FASTQs\\' reads longer than this length.',\n group: 'alignment',\n help: 'Also drop all reads shorter than chip.crop_length - chip.crop_length_tol.'\n }\n crop_length_tol: {\n description: 'Tolerance for cropping reads in FASTQs.',\n group: 'alignment',\n help: 'Drop all reads shorter than chip.crop_length - chip.crop_length_tol. Activated only when chip.crop_length is defined.'\n }\n xcor_trim_bp: {\n description: 'Trim experiment read1 FASTQ (for both SE and PE) for cross-correlation analysis.',\n group: 'alignment',\n help: 'This does not affect alignment of experimental/control replicates. Pipeline additionaly aligns R1 FASTQ only for cross-correlation analysis only. This parameter is used for it.'\n }\n use_filt_pe_ta_for_xcor: {\n description: 'Use filtered PE BAM for cross-correlation analysis.',\n group: 'alignment',\n help: 'If not defined, pipeline uses SE BAM generated from trimmed read1 FASTQ for cross-correlation analysis.'\n }\n dup_marker: {\n description: 'Marker for duplicate reads. picard or sambamba.',\n group: 'alignment',\n help: 'picard for Picard MarkDuplicates or sambamba for sambamba markdup.',\n choices: ['picard', 'sambamba'],\n example: 'picard'\n }\n no_dup_removal: {\n description: 'Disable removal of duplicate reads during filtering BAM.',\n group: 'alignment',\n help: 'Duplicate reads are filtererd out during filtering BAMs to gerenate NODUP_BAM. This flag will keep all duplicate reads in NODUP_BAM. This flag does not affect naming of NODUP_BAM. NODUP_BAM will still have .nodup. suffix in its filename.'\n }\n mapq_thresh: {\n description: 'Threshold for low MAPQ reads removal.',\n group: 'alignment',\n help: 'Low MAPQ reads are filtered out while filtering BAM.'\n }\n filter_chrs: {\n description: 'List of chromosomes to be filtered out while filtering BAM.',\n group: 'alignment',\n help: 'It is empty by default, hence no filtering out of specfic chromosomes. It is case-sensitive. Use exact word for chromosome names.'\n }\n subsample_reads: {\n description: 'Subsample reads. Shuffle and subsample reads.',\n group: 'alignment',\n help: 'This affects all downstream analyses after filtering experiment BAM. (e.g. all TAG-ALIGN files, peak-calling). Reads will be shuffled only if actual number of reads in BAM exceeds this number. 0 means disabled.'\n }\n ctl_subsample_reads: {\n description: 'Subsample control reads. Shuffle and subsample control reads.',\n group: 'alignment',\n help: 'This affects all downstream analyses after filtering control BAM. (e.g. all TAG-ALIGN files, peak-calling). Reads will be shuffled only if actual number of reads in BAM exceeds this number. 0 means disabled.'\n }\n xcor_subsample_reads: {\n description: 'Subsample reads for cross-corrlelation analysis only.',\n group: 'alignment',\n help: 'This does not affect downstream analyses after filtering BAM. It is for cross-correlation analysis only. 0 means disabled.'\n }\n xcor_exclusion_range_min: {\n description: 'Exclusion minimum for cross-correlation analysis.',\n group: 'alignment',\n help: 'For run_spp.R -s. Make sure that it is consistent with default strand shift -s=-500:5:1500 in run_spp.R.'\n }\n xcor_exclusion_range_max: {\n description: 'Exclusion maximum for cross-coorrelation analysis.',\n group: 'alignment',\n help: 'For run_spp.R -s. If not defined default value of `max(read length + 10, 50)` for TF and `max(read_len + 10, 100)` for histone are used'\n }\n\n ctl_depth_limit: {\n description: 'Hard limit for chosen control\\'s depth.',\n group: 'peak_calling',\n help: 'If control chosen by chip.always_use_pooled_ctl and chip.ctl_depth_ratio is deeper than this hard limit, then such control is subsampled.'\n }\n exp_ctl_depth_ratio_limit: {\n description: 'Second limit for chosen control\\'s depth.',\n group: 'peak_calling',\n help: 'If control chosen by chip.always_use_pooled_ctl and chip.ctl_depth_ratio is deeper than experiment replicate\\'s read depth multiplied by this factor then such control is subsampled down to maximum of multiplied value and hard limit chip.ctl_depth_limit.'\n }\n fraglen: {\n description: 'Fragment length for each biological replicate.',\n group: 'peak_calling',\n help: 'Fragment length is estimated by cross-correlation analysis, which is valid only when pipeline started from FASTQs. If defined, fragment length estimated by cross-correlation analysis is ignored.'\n }\n peak_caller: {\n description: 'Peak caller.',\n group: 'peak_calling',\n help: 'It is spp and macs2 by default for TF ChIP-seq and histone ChIP-seq, respectively. e.g. you can use macs2 for TF ChIP-Seq even though spp is by default for TF ChIP-Seq (chip.pipeline_type == tf).',\n choices: ['spp', 'macs2'],\n example: 'spp'\n }\n always_use_pooled_ctl: {\n description: 'Always choose a pooled control for each experiment replicate.',\n group: 'peak_calling',\n help: 'If turned on, ignores chip.ctl_depth_ratio.'\n }\n ctl_depth_ratio: {\n description: 'Maximum depth ratio between control replicates.',\n group: 'peak_calling',\n help: 'If ratio of depth between any two controls is higher than this, then always use a pooled control for all experiment replicates.'\n }\n\n cap_num_peak: {\n description: 'Upper limit on the number of peaks.',\n group: 'peak_calling',\n help: 'It is 30000000 and 50000000 by default for spp and macs2, respectively.'\n }\n pval_thresh: {\n description: 'p-value Threshold for MACS2 peak caller.',\n group: 'peak_calling',\n help: 'macs2 callpeak -p'\n }\n fdr_thresh: {\n description: 'FDR threshold for spp peak caller (phantompeakqualtools).',\n group: 'peak_calling',\n help: 'run_spp.R -fdr='\n }\n idr_thresh: {\n description: 'IDR threshold.',\n group: 'peak_calling'\n }\n\n align_cpu: {\n description: 'Number of cores for task align.',\n group: 'resource_parameter',\n help: 'Task align merges/crops/maps FASTQs.'\n }\n align_mem_mb: {\n description: 'Memory (MB) required for task align.',\n group: 'resource_parameter',\n help: 'This will be used for determining instance type for GCP/AWS or job\\'s maximum required memory for HPC.'\n }\n align_time_hr: {\n description: 'Walltime (h) required for task align.',\n group: 'resource_parameter',\n help: 'This is for HPCs only. e.g. SLURM, SGE, ...'\n }\n align_disks: {\n description: 'Persistent disk size to store intermediate files and outputs of task align.',\n group: 'resource_parameter',\n help: 'This is for GCP/AWS only.'\n }\n filter_cpu: {\n description: 'Number of cores for task filter.',\n group: 'resource_parameter',\n help: 'Task filter filters raw/unfilterd BAM to get filtered/deduped BAM.'\n }\n filter_mem_mb: {\n description: 'Memory (MB) required for task filter.',\n group: 'resource_parameter',\n help: 'This will be used for determining instance type for GCP/AWS or job\\'s maximum required memory for HPC.'\n }\n filter_time_hr: {\n description: 'Walltime (h) required for task filter.',\n group: 'resource_parameter',\n help: 'This is for HPCs only. e.g. SLURM, SGE, ...'\n }\n filter_disks: {\n description: 'Persistent disk size to store intermediate files and outputs of task filter.',\n group: 'resource_parameter',\n help: 'This is for GCP/AWS only.'\n }\n bam2ta_cpu: {\n description: 'Number of cores for task bam2ta.',\n group: 'resource_parameter',\n help: 'Task bam2ta converts filtered/deduped BAM in to TAG-ALIGN (6-col BED) format.'\n }\n bam2ta_mem_mb: {\n description: 'Memory (MB) required for task bam2ta.',\n group: 'resource_parameter',\n help: 'This will be used for determining instance type for GCP/AWS or job\\'s maximum required memory for HPC.'\n }\n bam2ta_time_hr: {\n description: 'Walltime (h) required for task bam2ta.',\n group: 'resource_parameter',\n help: 'This is for HPCs only. e.g. SLURM, SGE, ...'\n }\n bam2ta_disks: {\n description: 'Persistent disk size to store intermediate files and outputs of task bam2ta.',\n group: 'resource_parameter',\n help: 'This is for GCP/AWS only.'\n }\n spr_mem_mb: {\n description: 'Memory (MB) required for task spr.',\n group: 'resource_parameter',\n help: 'Task spr generates two pseudo-replicates from each biological replicate. This task uses Unix shuf/sort, which can take significant amount of memory.'\n }\n jsd_cpu: {\n description: 'Number of cores for task jsd.',\n group: 'resource_parameter',\n help: 'Task jsd plots Jensen-Shannon distance and metrics related to it.'\n }\n jsd_mem_mb: {\n description: 'Memory (MB) required for task jsd.',\n group: 'resource_parameter',\n help: 'This will be used for determining instance type for GCP/AWS or job\\'s maximum required memory for HPC.'\n }\n jsd_time_hr: {\n description: 'Walltime (h) required for task jsd.',\n group: 'resource_parameter',\n help: 'This is for HPCs only. e.g. SLURM, SGE, ...'\n }\n jsd_disks: {\n description: 'Persistent disk size to store intermediate files and outputs of task jsd.',\n group: 'resource_parameter',\n help: 'This is for GCP/AWS only.'\n }\n xcor_cpu: {\n description: 'Number of cores for task xcor.',\n group: 'resource_parameter',\n help: 'Task xcor does cross-correlation analysis (including a plot) on subsampled TAG-ALIGNs.'\n }\n xcor_mem_mb: {\n description: 'Memory (MB) required for task xcor.',\n group: 'resource_parameter',\n help: 'This will be used for determining instance type for GCP/AWS or job\\'s maximum required memory for HPC.'\n }\n xcor_time_hr: {\n description: 'Walltime (h) required for task xcor.',\n group: 'resource_parameter',\n help: 'This is for HPCs only. e.g. SLURM, SGE, ...'\n }\n xcor_disks: {\n description: 'Persistent disk size to store intermediate files and outputs of task xcor.',\n group: 'resource_parameter',\n help: 'This is for GCP/AWS only.'\n }\n call_peak_cpu: {\n description: 'Number of cores for task call_peak.',\n group: 'resource_parameter',\n help: 'Task call_peak call peaks on TAG-ALIGNs by using MACS2 peak caller.'\n }\n call_peak_mem_mb: {\n description: 'Memory (MB) required for task call_peak.',\n group: 'resource_parameter',\n help: 'This will be used for determining instance type for GCP/AWS or job\\'s maximum required memory for HPC.'\n }\n call_peak_time_hr: {\n description: 'Walltime (h) required for task call_peak.',\n group: 'resource_parameter',\n help: 'This is for HPCs only. e.g. SLURM, SGE, ...'\n }\n call_peak_disks: {\n description: 'Persistent disk size to store intermediate files and outputs of task call_peak.',\n group: 'resource_parameter',\n help: 'This is for GCP/AWS only.'\n }\n macs2_signal_track_mem_mb: {\n description: 'Memory (MB) required for task macs2_signal_track.',\n group: 'resource_parameter',\n help: 'Task macs2_signal_track_mem_mb uses MACS2 to get signal tracks for p-val (suffixed with .pval.) and fold enrichment (suffixed with .fc.). This will be used for determining instance type for GCP/AWS or job\\'s maximum required memory for HPC.'\n }\n macs2_signal_track_time_hr: {\n description: 'Walltime (h) required for task macs2_signal_track.',\n group: 'resource_parameter',\n help: 'This is for HPCs only. e.g. SLURM, SGE, ...'\n }\n macs2_signal_track_disks: {\n description: 'Persistent disk size to store intermediate files and outputs of task macs2_signal_track.',\n group: 'resource_parameter',\n help: 'This is for GCP/AWS only.'\n }\n align_trimmomatic_java_heap: {\n description: 'Maximum Java heap (java -Xmx) in task align.',\n group: 'resource_parameter',\n help: 'Maximum memory for Trimmomatic. If not defined, 90% of align_mem_mb will be used.'\n }\n filter_picard_java_heap: {\n description: 'Maximum Java heap (java -Xmx) in task filter.',\n group: 'resource_parameter',\n help: 'Maximum memory for Picard tools MarkDuplicates. If not defined, 90% of filter_mem_mb will be used.'\n }\n gc_bias_picard_java_heap: {\n description: 'Maximum Java heap (java -Xmx) in task gc_bias.',\n group: 'resource_parameter',\n help: 'Maximum memory for Picard tools CollectGcBiasMetrics. If not defined, 90% of 10000 MB will be used.'\n }\n }\n\n # read genome data and paths\n if ( defined(genome_tsv) ) {\n call read_genome_tsv { input: genome_tsv = genome_tsv }\n }\n File ref_fa_ = select_first([ref_fa, read_genome_tsv.ref_fa])\n File? bwa_idx_tar_ = if defined(bwa_idx_tar) then bwa_idx_tar\n else read_genome_tsv.bwa_idx_tar\n File bowtie2_idx_tar_ = select_first([bowtie2_idx_tar, read_genome_tsv.bowtie2_idx_tar])\n File chrsz_ = select_first([chrsz, read_genome_tsv.chrsz])\n String gensz_ = select_first([gensz, read_genome_tsv.gensz])\n File? blacklist1_ = if defined(blacklist) then blacklist\n else read_genome_tsv.blacklist\n File? blacklist2_ = if defined(blacklist2) then blacklist2\n else read_genome_tsv.blacklist2\n # merge multiple blacklists\n # two blacklists can have different number of columns (3 vs 6)\n # so we limit merged blacklist's columns to 3\n Array[File] blacklists = select_all([blacklist1_, blacklist2_])\n if ( length(blacklists) > 1 ) {\n call pool_ta as pool_blacklist { input:\n tas = blacklists,\n col = 3,\n }\n }\n File? blacklist_ = if length(blacklists) > 1 then pool_blacklist.ta_pooled\n else if length(blacklists) > 0 then blacklists[0]\n else blacklist2_\n String mito_chr_name_ = select_first([mito_chr_name, read_genome_tsv.mito_chr_name])\n String regex_bfilt_peak_chr_name_ = select_first([regex_bfilt_peak_chr_name, read_genome_tsv.regex_bfilt_peak_chr_name])\n String genome_name_ = select_first([genome_name, read_genome_tsv.genome_name, basename(chrsz_)])\n\n ### temp vars (do not define these)\n String aligner_ = aligner\n String peak_caller_ = if pipeline_type=='tf' then select_first([peak_caller, 'spp'])\n else select_first([peak_caller, 'macs2'])\n String peak_type_ = if peak_caller_=='spp' then 'regionPeak'\n else 'narrowPeak'\n Boolean enable_idr = pipeline_type=='tf' # enable_idr for TF chipseq only\n String idr_rank_ = if peak_caller_=='spp' then 'signal.value'\n else if peak_caller_=='macs2' then 'p.value'\n else 'p.value'\n Int cap_num_peak_spp = 300000\n Int cap_num_peak_macs2 = 500000 \n Int cap_num_peak_ = if peak_caller_ == 'spp' then select_first([cap_num_peak, cap_num_peak_spp])\n else select_first([cap_num_peak, cap_num_peak_macs2])\n Int mapq_thresh_ = mapq_thresh\n Boolean enable_xcor_ = if pipeline_type=='control' then false else true\n Boolean enable_count_signal_track_ = if pipeline_type=='control' then false else enable_count_signal_track\n Boolean enable_jsd_ = if pipeline_type=='control' then false else enable_jsd\n Boolean enable_gc_bias_ = if pipeline_type=='control' then false else enable_gc_bias\n Boolean align_only_ = if pipeline_type=='control' then true else align_only\n\n # temporary 2-dim fastqs array [rep_id][merge_id]\n Array[Array[File]] fastqs_R1 = \n if length(fastqs_rep10_R1)>0 then\n [fastqs_rep1_R1, fastqs_rep2_R1, fastqs_rep3_R1, fastqs_rep4_R1, fastqs_rep5_R1,\n fastqs_rep6_R1, fastqs_rep7_R1, fastqs_rep8_R1, fastqs_rep9_R1, fastqs_rep10_R1]\n else if length(fastqs_rep9_R1)>0 then\n [fastqs_rep1_R1, fastqs_rep2_R1, fastqs_rep3_R1, fastqs_rep4_R1, fastqs_rep5_R1,\n fastqs_rep6_R1, fastqs_rep7_R1, fastqs_rep8_R1, fastqs_rep9_R1]\n else if length(fastqs_rep8_R1)>0 then\n [fastqs_rep1_R1, fastqs_rep2_R1, fastqs_rep3_R1, fastqs_rep4_R1, fastqs_rep5_R1,\n fastqs_rep6_R1, fastqs_rep7_R1, fastqs_rep8_R1]\n else if length(fastqs_rep7_R1)>0 then\n [fastqs_rep1_R1, fastqs_rep2_R1, fastqs_rep3_R1, fastqs_rep4_R1, fastqs_rep5_R1,\n fastqs_rep6_R1, fastqs_rep7_R1]\n else if length(fastqs_rep6_R1)>0 then\n [fastqs_rep1_R1, fastqs_rep2_R1, fastqs_rep3_R1, fastqs_rep4_R1, fastqs_rep5_R1,\n fastqs_rep6_R1]\n else if length(fastqs_rep5_R1)>0 then\n [fastqs_rep1_R1, fastqs_rep2_R1, fastqs_rep3_R1, fastqs_rep4_R1, fastqs_rep5_R1]\n else if length(fastqs_rep4_R1)>0 then\n [fastqs_rep1_R1, fastqs_rep2_R1, fastqs_rep3_R1, fastqs_rep4_R1]\n else if length(fastqs_rep3_R1)>0 then\n [fastqs_rep1_R1, fastqs_rep2_R1, fastqs_rep3_R1]\n else if length(fastqs_rep2_R1)>0 then\n [fastqs_rep1_R1, fastqs_rep2_R1]\n else if length(fastqs_rep1_R1)>0 then\n [fastqs_rep1_R1]\n else []\n # no need to do that for R2 (R1 array will be used to determine presense of fastq for each rep)\n Array[Array[File]] fastqs_R2 = \n [fastqs_rep1_R2, fastqs_rep2_R2, fastqs_rep3_R2, fastqs_rep4_R2, fastqs_rep5_R2,\n fastqs_rep6_R2, fastqs_rep7_R2, fastqs_rep8_R2, fastqs_rep9_R2, fastqs_rep10_R2]\n\n # temporary 2-dim ctl fastqs array [rep_id][merge_id]\n Array[Array[File]] ctl_fastqs_R1 = \n if length(ctl_fastqs_rep10_R1)>0 then\n [ctl_fastqs_rep1_R1, ctl_fastqs_rep2_R1, ctl_fastqs_rep3_R1, ctl_fastqs_rep4_R1, ctl_fastqs_rep5_R1,\n ctl_fastqs_rep6_R1, ctl_fastqs_rep7_R1, ctl_fastqs_rep8_R1, ctl_fastqs_rep9_R1, ctl_fastqs_rep10_R1]\n else if length(ctl_fastqs_rep9_R1)>0 then\n [ctl_fastqs_rep1_R1, ctl_fastqs_rep2_R1, ctl_fastqs_rep3_R1, ctl_fastqs_rep4_R1, ctl_fastqs_rep5_R1,\n ctl_fastqs_rep6_R1, ctl_fastqs_rep7_R1, ctl_fastqs_rep8_R1, ctl_fastqs_rep9_R1]\n else if length(ctl_fastqs_rep8_R1)>0 then\n [ctl_fastqs_rep1_R1, ctl_fastqs_rep2_R1, ctl_fastqs_rep3_R1, ctl_fastqs_rep4_R1, ctl_fastqs_rep5_R1,\n ctl_fastqs_rep6_R1, ctl_fastqs_rep7_R1, ctl_fastqs_rep8_R1]\n else if length(ctl_fastqs_rep7_R1)>0 then\n [ctl_fastqs_rep1_R1, ctl_fastqs_rep2_R1, ctl_fastqs_rep3_R1, ctl_fastqs_rep4_R1, ctl_fastqs_rep5_R1,\n ctl_fastqs_rep6_R1, ctl_fastqs_rep7_R1]\n else if length(ctl_fastqs_rep6_R1)>0 then\n [ctl_fastqs_rep1_R1, ctl_fastqs_rep2_R1, ctl_fastqs_rep3_R1, ctl_fastqs_rep4_R1, ctl_fastqs_rep5_R1,\n ctl_fastqs_rep6_R1]\n else if length(ctl_fastqs_rep5_R1)>0 then\n [ctl_fastqs_rep1_R1, ctl_fastqs_rep2_R1, ctl_fastqs_rep3_R1, ctl_fastqs_rep4_R1, ctl_fastqs_rep5_R1]\n else if length(ctl_fastqs_rep4_R1)>0 then\n [ctl_fastqs_rep1_R1, ctl_fastqs_rep2_R1, ctl_fastqs_rep3_R1, ctl_fastqs_rep4_R1]\n else if length(ctl_fastqs_rep3_R1)>0 then\n [ctl_fastqs_rep1_R1, ctl_fastqs_rep2_R1, ctl_fastqs_rep3_R1]\n else if length(ctl_fastqs_rep2_R1)>0 then\n [ctl_fastqs_rep1_R1, ctl_fastqs_rep2_R1]\n else if length(ctl_fastqs_rep1_R1)>0 then\n [ctl_fastqs_rep1_R1]\n else []\n # no need to do that for R2 (R1 array will be used to determine presense of fastq for each rep)\n Array[Array[File]] ctl_fastqs_R2 = \n [ctl_fastqs_rep1_R2, ctl_fastqs_rep2_R2, ctl_fastqs_rep3_R2, ctl_fastqs_rep4_R2, ctl_fastqs_rep5_R2,\n ctl_fastqs_rep6_R2, ctl_fastqs_rep7_R2, ctl_fastqs_rep8_R2, ctl_fastqs_rep9_R2, ctl_fastqs_rep10_R2]\n\n # temporary variables to get number of replicates\n # WDLic implementation of max(A,B,C,...)\n Int num_rep_fastq = length(fastqs_R1)\n Int num_rep_bam = if length(bams)<num_rep_fastq then num_rep_fastq\n else length(bams)\n Int num_rep_nodup_bam = if length(nodup_bams)<num_rep_bam then num_rep_bam\n else length(nodup_bams)\n Int num_rep_ta = if length(tas)<num_rep_nodup_bam then num_rep_nodup_bam\n else length(tas)\n Int num_rep_peak = if length(peaks)<num_rep_ta then num_rep_ta\n else length(peaks)\n Int num_rep = num_rep_peak\n\n # temporary variables to get number of controls\n Int num_ctl_fastq = length(ctl_fastqs_R1)\n Int num_ctl_bam = if length(ctl_bams)<num_ctl_fastq then num_ctl_fastq\n else length(ctl_bams)\n Int num_ctl_nodup_bam = if length(ctl_nodup_bams)<num_ctl_bam then num_ctl_bam\n else length(ctl_nodup_bams)\n Int num_ctl_ta = if length(ctl_tas)<num_ctl_nodup_bam then num_ctl_nodup_bam\n else length(ctl_tas)\n Int num_ctl = num_ctl_ta\n\n # sanity check for inputs\n if ( num_rep == 0 && num_ctl == 0 ) {\n call raise_exception as error_input_data { input:\n msg = 'No FASTQ/BAM/TAG-ALIGN/PEAK defined in your input JSON. Check if your FASTQs are defined as \"chip.fastqs_repX_RY\". DO NOT MISS suffix _R1 even for single ended FASTQ.'\n }\n }\n if ( !align_only_ && peak_caller_ == 'spp' && num_ctl == 0 ) {\n call raise_exception as error_control_required { input:\n msg = 'SPP requires control inputs. Define control input files (\"chip.ctl_*\") in an input JSON file.'\n }\n }\n if ( (num_rep_fastq > 0 || num_ctl_fastq > 0) && aligner_ != 'bwa' && aligner_ != 'bowtie2' ) {\n call raise_exception as error_wrong_aligner { input:\n msg = 'Choose chip.aligner between bwa and bowtie2.'\n }\n }\n if ( aligner_ != 'bwa' && use_bwa_mem_for_pe ) {\n call raise_exception as error_use_bwa_mem_for_non_bwa { input:\n msg = 'To use chip.use_bwa_mem_for_pe, choose bwa for chip.aligner.'\n }\n } \n if ( ( ctl_depth_limit > 0 || exp_ctl_depth_ratio_limit > 0 ) && num_ctl > 1 && length(ctl_paired_ends) > 1 ) {\n call raise_exception as error_subsample_pooled_control_with_mixed_endedness { input:\n msg = 'Cannot use automatic control subsampling (\"chip.ctl_depth_limit\">0 and \"chip.exp_ctl_depth_limit\">0) for ' +\n 'multiple controls with mixed endedness (e.g. SE ctl-rep1 and PE ctl-rep2). ' +\n 'Automatic control subsampling is enabled by default. ' +\n 'Disable automatic control subsampling by explicitly defining the above two parameters as 0 in your input JSON file. ' +\n 'You can still use manual control subsamping (\"chip.ctl_subsample_reads\">0) since it is done ' +\n 'for individual control\\'s TAG-ALIGN output according to each control\\'s endedness. '\n }\n }\n if ( pipeline_type == 'control' && num_ctl > 0 ) {\n call raise_exception as error_ctl_input_defined_in_control_mode { input:\n msg = 'In control mode (chip.pipeline_type: control), do not define ctl_* input variables. Define fastqs_repX_RY instead.'\n }\n }\n if ( pipeline_type == 'control' && num_rep_fastq == 0 ) {\n call raise_exception as error_ctl_fastq_input_required_for_control_mode { input:\n msg = 'Control mode (chip.pipeline_type: control) is for FASTQs only. Define FASTQs in fastqs_repX_RY. Pipeline will recognize them as control FASTQs.'\n }\n }\n\n # align each replicate\n scatter(i in range(num_rep)) {\n # to override endedness definition for individual replicate\n # paired_end will override paired_ends[i]\n Boolean paired_end_ = if !defined(paired_end) && i<length(paired_ends) then paired_ends[i]\n else select_first([paired_end])\n\n Boolean has_input_of_align = i<length(fastqs_R1) && length(fastqs_R1[i])>0\n Boolean has_output_of_align = i<length(bams) && defined(bams[i])\n if ( has_input_of_align && !has_output_of_align ) {\n call align { input :\n fastqs_R1 = fastqs_R1[i],\n fastqs_R2 = fastqs_R2[i],\n crop_length = crop_length,\n crop_length_tol = crop_length_tol,\n\n aligner = aligner_,\n mito_chr_name = mito_chr_name_,\n idx_tar = if aligner=='bwa' then bwa_idx_tar_\n else bowtie2_idx_tar_,\n paired_end = paired_end_,\n use_bwa_mem_for_pe = use_bwa_mem_for_pe,\n\n trimmomatic_java_heap = align_trimmomatic_java_heap,\n cpu = align_cpu,\n mem_mb = align_mem_mb,\n time_hr = align_time_hr,\n disks = align_disks,\n }\n }\n File? bam_ = if has_output_of_align then bams[i] else align.bam\n\n Boolean has_input_of_filter = has_output_of_align || defined(align.bam)\n Boolean has_output_of_filter = i<length(nodup_bams) && defined(nodup_bams[i])\n # skip if we already have output of this step\n if ( has_input_of_filter && !has_output_of_filter ) {\n call filter { input :\n bam = bam_,\n paired_end = paired_end_,\n dup_marker = dup_marker,\n mapq_thresh = mapq_thresh_,\n filter_chrs = filter_chrs,\n chrsz = chrsz_,\n no_dup_removal = no_dup_removal,\n mito_chr_name = mito_chr_name_,\n\n cpu = filter_cpu,\n mem_mb = filter_mem_mb,\n picard_java_heap = filter_picard_java_heap,\n time_hr = filter_time_hr,\n disks = filter_disks,\n }\n }\n File? nodup_bam_ = if has_output_of_filter then nodup_bams[i] else filter.nodup_bam\n\n Boolean has_input_of_bam2ta = has_output_of_filter || defined(filter.nodup_bam)\n Boolean has_output_of_bam2ta = i<length(tas) && defined(tas[i])\n if ( has_input_of_bam2ta && !has_output_of_bam2ta ) {\n call bam2ta { input :\n bam = nodup_bam_,\n subsample = subsample_reads,\n paired_end = paired_end_,\n mito_chr_name = mito_chr_name_,\n\n cpu = bam2ta_cpu,\n mem_mb = bam2ta_mem_mb,\n time_hr = bam2ta_time_hr,\n disks = bam2ta_disks,\n }\n }\n File? ta_ = if has_output_of_bam2ta then tas[i] else bam2ta.ta\n\n Boolean has_input_of_spr = has_output_of_bam2ta || defined(bam2ta.ta)\n if ( has_input_of_spr && !align_only_ && !true_rep_only ) {\n call spr { input :\n ta = ta_,\n paired_end = paired_end_,\n mem_mb = spr_mem_mb,\n }\n }\n\n Boolean has_input_of_count_signal_track = has_output_of_bam2ta || defined(bam2ta.ta)\n if ( has_input_of_count_signal_track && enable_count_signal_track_ ) {\n # generate count signal track\n call count_signal_track { input :\n ta = ta_,\n chrsz = chrsz_,\n }\n }\n\n if ( enable_gc_bias_ && defined(nodup_bam_) && defined(ref_fa_) ) {\n call gc_bias { input :\n nodup_bam = nodup_bam_,\n ref_fa = ref_fa_,\n picard_java_heap = gc_bias_picard_java_heap,\n }\n }\n\n # special trimming/mapping for xcor (when starting from FASTQs)\n if ( has_input_of_align ) {\n call align as align_R1 { input :\n fastqs_R1 = fastqs_R1[i],\n fastqs_R2 = [],\n trim_bp = xcor_trim_bp,\n crop_length = 0,\n crop_length_tol = 0,\n\n aligner = aligner_,\n mito_chr_name = mito_chr_name_,\n idx_tar = if aligner=='bwa' then bwa_idx_tar_\n else bowtie2_idx_tar_,\n paired_end = false,\n use_bwa_mem_for_pe = use_bwa_mem_for_pe,\n\n cpu = align_cpu,\n mem_mb = align_mem_mb,\n time_hr = align_time_hr,\n disks = align_disks,\n }\n # no bam deduping for xcor\n call filter as filter_R1 { input :\n bam = align_R1.bam,\n paired_end = false,\n dup_marker = dup_marker,\n mapq_thresh = mapq_thresh_,\n filter_chrs = filter_chrs,\n chrsz = chrsz_,\n no_dup_removal = true,\n mito_chr_name = mito_chr_name_,\n\n cpu = filter_cpu,\n mem_mb = filter_mem_mb,\n picard_java_heap = filter_picard_java_heap,\n time_hr = filter_time_hr,\n disks = filter_disks,\n }\n call bam2ta as bam2ta_no_dedup_R1 { input :\n bam = filter_R1.nodup_bam, # it's named as nodup bam but it's not deduped but just filtered\n paired_end = false,\n subsample = 0,\n mito_chr_name = mito_chr_name_,\n\n cpu = bam2ta_cpu,\n mem_mb = bam2ta_mem_mb,\n time_hr = bam2ta_time_hr,\n disks = bam2ta_disks,\n }\n }\n\n # special trimming/mapping for xcor (when starting from BAMs)\n Boolean has_input_of_bam2ta_no_dedup = (has_output_of_align || defined(align.bam))\n && !defined(bam2ta_no_dedup_R1.ta)\n if ( has_input_of_bam2ta_no_dedup ) {\n call filter as filter_no_dedup { input :\n bam = bam_,\n paired_end = paired_end_,\n dup_marker = dup_marker,\n mapq_thresh = mapq_thresh_,\n filter_chrs = filter_chrs,\n chrsz = chrsz_,\n no_dup_removal = true,\n mito_chr_name = mito_chr_name_,\n\n cpu = filter_cpu,\n mem_mb = filter_mem_mb,\n picard_java_heap = filter_picard_java_heap,\n time_hr = filter_time_hr,\n disks = filter_disks,\n }\n call bam2ta as bam2ta_no_dedup { input :\n bam = filter_no_dedup.nodup_bam, # output name is nodup but it's not deduped\n paired_end = paired_end_,\n subsample = 0,\n mito_chr_name = mito_chr_name_,\n\n cpu = bam2ta_cpu,\n mem_mb = bam2ta_mem_mb,\n time_hr = bam2ta_time_hr,\n disks = bam2ta_disks,\n }\n }\n\n # use trimmed/unfilitered R1 tagAlign for paired end dataset\n # if not starting from fastqs, keep using old method\n # (mapping with both ends for tag-aligns to be used for xcor)\n # subsample tagalign (non-mito) and cross-correlation analysis\n File? ta_xcor = if defined(bam2ta_no_dedup_R1.ta) then bam2ta_no_dedup_R1.ta\n else if defined(bam2ta_no_dedup.ta) then bam2ta_no_dedup.ta\n else ta_\n Boolean paired_end_xcor = if defined(bam2ta_no_dedup_R1.ta) then false\n else paired_end_\n\n Boolean has_input_of_xcor = defined(ta_xcor)\n if ( has_input_of_xcor && enable_xcor_ ) {\n call xcor { input :\n ta = ta_xcor,\n paired_end = paired_end_xcor,\n subsample = xcor_subsample_reads,\n mito_chr_name = mito_chr_name_,\n chip_seq_type = pipeline_type,\n exclusion_range_min = xcor_exclusion_range_min,\n exclusion_range_max = xcor_exclusion_range_max,\n cpu = xcor_cpu,\n mem_mb = xcor_mem_mb,\n time_hr = xcor_time_hr,\n disks = xcor_disks,\n }\n }\n\n # before peak calling, get fragment length from xcor analysis or given input\n # if fraglen [] is defined in the input JSON, fraglen from xcor will be ignored\n Int? fraglen_ = if i<length(fraglen) && defined(fraglen[i]) then fraglen[i]\n else xcor.fraglen\n }\n\n # align each control\n scatter(i in range(num_ctl)) {\n # to override endedness definition for individual control\n # ctl_paired_end will override ctl_paired_ends[i]\n Boolean ctl_paired_end_ = if !defined(ctl_paired_end) && i<length(ctl_paired_ends) then ctl_paired_ends[i]\n else select_first([ctl_paired_end, paired_end])\n\n Boolean has_input_of_align_ctl = i<length(ctl_fastqs_R1) && length(ctl_fastqs_R1[i])>0\n Boolean has_output_of_align_ctl = i<length(ctl_bams) && defined(ctl_bams[i])\n if ( has_input_of_align_ctl && !has_output_of_align_ctl ) {\n call align as align_ctl { input :\n fastqs_R1 = ctl_fastqs_R1[i],\n fastqs_R2 = ctl_fastqs_R2[i],\n crop_length = crop_length,\n crop_length_tol = crop_length_tol,\n\n aligner = aligner_,\n mito_chr_name = mito_chr_name_,\n idx_tar = if aligner=='bwa' then bwa_idx_tar_\n else bowtie2_idx_tar_,\n paired_end = ctl_paired_end_,\n use_bwa_mem_for_pe = use_bwa_mem_for_pe,\n\n trimmomatic_java_heap = align_trimmomatic_java_heap,\n cpu = align_cpu,\n mem_mb = align_mem_mb,\n time_hr = align_time_hr,\n disks = align_disks,\n }\n }\n File? ctl_bam_ = if has_output_of_align_ctl then ctl_bams[i] else align_ctl.bam\n\n Boolean has_input_of_filter_ctl = has_output_of_align_ctl || defined(align_ctl.bam)\n Boolean has_output_of_filter_ctl = i<length(ctl_nodup_bams) && defined(ctl_nodup_bams[i])\n # skip if we already have output of this step\n if ( has_input_of_filter_ctl && !has_output_of_filter_ctl ) {\n call filter as filter_ctl { input :\n bam = ctl_bam_,\n paired_end = ctl_paired_end_,\n dup_marker = dup_marker,\n mapq_thresh = mapq_thresh_,\n filter_chrs = filter_chrs,\n chrsz = chrsz_,\n no_dup_removal = no_dup_removal,\n mito_chr_name = mito_chr_name_,\n\n cpu = filter_cpu,\n mem_mb = filter_mem_mb,\n picard_java_heap = filter_picard_java_heap,\n time_hr = filter_time_hr,\n disks = filter_disks,\n }\n }\n File? ctl_nodup_bam_ = if has_output_of_filter_ctl then ctl_nodup_bams[i] else filter_ctl.nodup_bam\n\n Boolean has_input_of_bam2ta_ctl = has_output_of_filter_ctl || defined(filter_ctl.nodup_bam)\n Boolean has_output_of_bam2ta_ctl = i<length(ctl_tas) && defined(ctl_tas[i])\n if ( has_input_of_bam2ta_ctl && !has_output_of_bam2ta_ctl ) {\n call bam2ta as bam2ta_ctl { input :\n bam = ctl_nodup_bam_,\n subsample = subsample_reads,\n paired_end = ctl_paired_end_,\n mito_chr_name = mito_chr_name_,\n\n cpu = bam2ta_cpu,\n mem_mb = bam2ta_mem_mb,\n time_hr = bam2ta_time_hr,\n disks = bam2ta_disks,\n }\n }\n File? ctl_ta_ = if has_output_of_bam2ta_ctl then ctl_tas[i] else bam2ta_ctl.ta\n }\n\n # if there are TAs for ALL replicates then pool them\n Boolean has_all_inputs_of_pool_ta = length(select_all(ta_))==num_rep\n if ( has_all_inputs_of_pool_ta && num_rep>1 ) {\n # pool tagaligns from true replicates\n call pool_ta { input :\n tas = ta_,\n prefix = 'rep',\n }\n }\n\n # if there are pr1 TAs for ALL replicates then pool them\n Boolean has_all_inputs_of_pool_ta_pr1 = length(select_all(spr.ta_pr1))==num_rep\n if ( has_all_inputs_of_pool_ta_pr1 && num_rep>1 && !align_only_ && !true_rep_only ) {\n # pool tagaligns from pseudo replicate 1\n call pool_ta as pool_ta_pr1 { input :\n tas = spr.ta_pr1,\n prefix = 'rep-pr1',\n }\n }\n\n # if there are pr2 TAs for ALL replicates then pool them\n Boolean has_all_inputs_of_pool_ta_pr2 = length(select_all(spr.ta_pr2))==num_rep\n if ( has_all_inputs_of_pool_ta_pr1 && num_rep>1 && !align_only_ && !true_rep_only ) {\n # pool tagaligns from pseudo replicate 2\n call pool_ta as pool_ta_pr2 { input :\n tas = spr.ta_pr2,\n prefix = 'rep-pr2',\n }\n }\n\n # if there are CTL TAs for ALL replicates then pool them\n Boolean has_all_inputs_of_pool_ta_ctl = length(select_all(ctl_ta_))==num_ctl\n if ( has_all_inputs_of_pool_ta_ctl && num_ctl>1 ) {\n # pool tagaligns from true replicates\n call pool_ta as pool_ta_ctl { input :\n tas = ctl_ta_,\n prefix = 'ctl',\n }\n }\n\n Boolean has_input_of_count_signal_track_pooled = defined(pool_ta.ta_pooled)\n if ( has_input_of_count_signal_track_pooled && enable_count_signal_track_ && num_rep>1 ) {\n call count_signal_track as count_signal_track_pooled { input :\n ta = pool_ta.ta_pooled,\n chrsz = chrsz_,\n }\n }\n\n Boolean has_input_of_jsd = defined(blacklist_) && length(select_all(nodup_bam_))==num_rep\n if ( has_input_of_jsd && num_rep > 0 && enable_jsd_ ) {\n # fingerprint and JS-distance plot\n call jsd { input :\n nodup_bams = nodup_bam_,\n ctl_bams = ctl_nodup_bam_, # use first control only\n blacklist = blacklist_,\n mapq_thresh = mapq_thresh_,\n\n cpu = jsd_cpu,\n mem_mb = jsd_mem_mb,\n time_hr = jsd_time_hr,\n disks = jsd_disks,\n }\n }\n\n Boolean has_all_input_of_choose_ctl = length(select_all(ta_))==num_rep\n && length(select_all(ctl_ta_))==num_ctl && num_ctl > 0\n if ( has_all_input_of_choose_ctl && !align_only_ ) {\n # choose appropriate control for each exp IP replicate\n # outputs:\n # choose_ctl.idx : control replicate index for each exp replicate \n # -1 means pooled ctl replicate\n call choose_ctl { input:\n tas = ta_,\n ctl_tas = ctl_ta_,\n ta_pooled = pool_ta.ta_pooled,\n ctl_ta_pooled = pool_ta_ctl.ta_pooled,\n always_use_pooled_ctl = always_use_pooled_ctl,\n ctl_depth_ratio = ctl_depth_ratio,\n ctl_depth_limit = ctl_depth_limit,\n exp_ctl_depth_ratio_limit = exp_ctl_depth_ratio_limit,\n }\n }\n\n scatter(i in range(num_rep)) {\n # make control ta array [[1,2,3,4]] -> [[1],[2],[3],[4]]\n # chosen_ctl_ta_id\n # >=0: control TA index (this means that control TA with this index exists)\n # -1: use pooled control\n # -2: there is no control\n Int chosen_ctl_ta_id = if has_all_input_of_choose_ctl && !align_only_ then\n select_first([choose_ctl.chosen_ctl_ta_ids])[i] else -2\n Array[File] chosen_ctl_tas = if chosen_ctl_ta_id == -2 then []\n else if chosen_ctl_ta_id == -1 then [ select_first([pool_ta_ctl.ta_pooled]) ]\n else [ select_first([ctl_ta_[ chosen_ctl_ta_id ]]) ]\n\n Int chosen_ctl_ta_subsample = if has_all_input_of_choose_ctl && !align_only_ then\n select_first([choose_ctl.chosen_ctl_ta_subsample])[i] else 0\n Boolean chosen_ctl_paired_end = if chosen_ctl_ta_id == -2 then false\n else if chosen_ctl_ta_id == -1 then ctl_paired_end_[0]\n else ctl_paired_end_[chosen_ctl_ta_id]\n }\n Int chosen_ctl_ta_pooled_subsample = if has_all_input_of_choose_ctl && !align_only_ then\n select_first([choose_ctl.chosen_ctl_ta_subsample_pooled]) else 0\n\n # workaround for dx error (Unsupported combination: womType: Int womValue: ([225], Array[Int]))\n Array[Int] fraglen_tmp = select_all(fraglen_)\n\n # we have all tas and ctl_tas (optional for histone chipseq) ready, let's call peaks\n scatter(i in range(num_rep)) {\n Boolean has_input_of_call_peak = defined(ta_[i])\n Boolean has_output_of_call_peak = i<length(peaks) && defined(peaks[i])\n if ( has_input_of_call_peak && !has_output_of_call_peak && !align_only_ ) {\n call call_peak { input :\n peak_caller = peak_caller_,\n peak_type = peak_type_,\n tas = flatten([[ta_[i]], chosen_ctl_tas[i]]),\n gensz = gensz_,\n chrsz = chrsz_,\n cap_num_peak = cap_num_peak_,\n ctl_subsample = chosen_ctl_ta_subsample[i],\n ctl_paired_end = chosen_ctl_paired_end[i],\n pval_thresh = pval_thresh,\n fdr_thresh = fdr_thresh,\n fraglen = fraglen_tmp[i],\n blacklist = blacklist_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n\n cpu = call_peak_cpu,\n mem_mb = call_peak_mem_mb,\n disks = call_peak_disks,\n time_hr = call_peak_time_hr,\n }\n }\n File? peak_ = if has_output_of_call_peak then peaks[i]\n else call_peak.peak\n\n # signal track\n if ( has_input_of_call_peak && !align_only_ ) {\n call macs2_signal_track { input :\n tas = flatten([[ta_[i]], chosen_ctl_tas[i]]),\n gensz = gensz_,\n chrsz = chrsz_,\n pval_thresh = pval_thresh,\n ctl_subsample = chosen_ctl_ta_subsample[i],\n ctl_paired_end = chosen_ctl_paired_end[i],\n fraglen = fraglen_tmp[i],\n\n mem_mb = macs2_signal_track_mem_mb,\n disks = macs2_signal_track_disks,\n time_hr = macs2_signal_track_time_hr,\n }\n }\n\n # call peaks on 1st pseudo replicated tagalign\n Boolean has_input_of_call_peak_pr1 = defined(spr.ta_pr1[i])\n Boolean has_output_of_call_peak_pr1 = i<length(peaks_pr1) && defined(peaks_pr1[i])\n if ( has_input_of_call_peak_pr1 && !has_output_of_call_peak_pr1 && !true_rep_only ) {\n call call_peak as call_peak_pr1 { input :\n peak_caller = peak_caller_,\n peak_type = peak_type_,\n tas = flatten([[spr.ta_pr1[i]], chosen_ctl_tas[i]]),\n gensz = gensz_,\n chrsz = chrsz_,\n cap_num_peak = cap_num_peak_,\n ctl_subsample = chosen_ctl_ta_subsample[i],\n ctl_paired_end = chosen_ctl_paired_end[i],\n pval_thresh = pval_thresh,\n fdr_thresh = fdr_thresh,\n fraglen = fraglen_tmp[i],\n blacklist = blacklist_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n \n cpu = call_peak_cpu,\n mem_mb = call_peak_mem_mb,\n disks = call_peak_disks,\n time_hr = call_peak_time_hr,\n }\n }\n File? peak_pr1_ = if has_output_of_call_peak_pr1 then peaks_pr1[i]\n else call_peak_pr1.peak\n\n # call peaks on 2nd pseudo replicated tagalign\n Boolean has_input_of_call_peak_pr2 = defined(spr.ta_pr2[i])\n Boolean has_output_of_call_peak_pr2 = i<length(peaks_pr2) && defined(peaks_pr2[i])\n if ( has_input_of_call_peak_pr2 && !has_output_of_call_peak_pr2 && !true_rep_only ) {\n call call_peak as call_peak_pr2 { input :\n peak_caller = peak_caller_,\n peak_type = peak_type_,\n tas = flatten([[spr.ta_pr2[i]], chosen_ctl_tas[i]]),\n gensz = gensz_,\n chrsz = chrsz_,\n cap_num_peak = cap_num_peak_,\n ctl_subsample = chosen_ctl_ta_subsample[i],\n ctl_paired_end = chosen_ctl_paired_end[i],\n pval_thresh = pval_thresh,\n fdr_thresh = fdr_thresh,\n fraglen = fraglen_tmp[i],\n blacklist = blacklist_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n\n cpu = call_peak_cpu,\n mem_mb = call_peak_mem_mb,\n disks = call_peak_disks,\n time_hr = call_peak_time_hr,\n }\n }\n File? peak_pr2_ = if has_output_of_call_peak_pr2 then peaks_pr2[i]\n else call_peak_pr2.peak\n }\n\n # if ( !align_only_ && num_rep > 1 ) {\n # rounded mean of fragment length, which will be used for \n # 1) calling peaks for pooled true/pseudo replicates\n # 2) calculating FRiP\n call rounded_mean as fraglen_mean { input :\n ints = fraglen_tmp,\n }\n # }\n\n # actually not an array\n Array[File?] chosen_ctl_ta_pooled = if !has_all_input_of_choose_ctl || align_only_ then []\n else if num_ctl < 2 then [ctl_ta_[0]] # choose first (only) control\n else select_all([pool_ta_ctl.ta_pooled]) # choose pooled control\n Boolean chosen_ctl_ta_pooled_paired_end = if !has_all_input_of_choose_ctl || align_only_ then false\n else ctl_paired_end_[0]\n\n Boolean has_input_of_call_peak_pooled = defined(pool_ta.ta_pooled)\n Boolean has_output_of_call_peak_pooled = defined(peak_pooled)\n if ( has_input_of_call_peak_pooled && !has_output_of_call_peak_pooled && !align_only_ && num_rep>1 ) {\n # call peaks on pooled replicate\n # always call peaks for pooled replicate to get signal tracks\n call call_peak as call_peak_pooled { input :\n peak_caller = peak_caller_,\n peak_type = peak_type_,\n tas = flatten([select_all([pool_ta.ta_pooled]), chosen_ctl_ta_pooled]),\n gensz = gensz_,\n chrsz = chrsz_,\n cap_num_peak = cap_num_peak_,\n ctl_subsample = chosen_ctl_ta_pooled_subsample,\n ctl_paired_end = chosen_ctl_ta_pooled_paired_end,\n pval_thresh = pval_thresh,\n fdr_thresh = fdr_thresh,\n fraglen = fraglen_mean.rounded_mean,\n blacklist = blacklist_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n\n cpu = call_peak_cpu,\n mem_mb = call_peak_mem_mb,\n disks = call_peak_disks,\n time_hr = call_peak_time_hr,\n }\n }\n File? peak_pooled_ = if has_output_of_call_peak_pooled then peak_pooled\n else call_peak_pooled.peak \n\n # macs2 signal track for pooled rep\n if ( has_input_of_call_peak_pooled && !align_only_ && num_rep>1 ) {\n call macs2_signal_track as macs2_signal_track_pooled { input :\n tas = flatten([select_all([pool_ta.ta_pooled]), chosen_ctl_ta_pooled]),\n gensz = gensz_,\n chrsz = chrsz_,\n pval_thresh = pval_thresh,\n ctl_subsample = chosen_ctl_ta_pooled_subsample,\n ctl_paired_end = chosen_ctl_ta_pooled_paired_end,\n fraglen = fraglen_mean.rounded_mean,\n\n mem_mb = macs2_signal_track_mem_mb,\n disks = macs2_signal_track_disks,\n time_hr = macs2_signal_track_time_hr,\n }\n }\n\n Boolean has_input_of_call_peak_ppr1 = defined(pool_ta_pr1.ta_pooled)\n Boolean has_output_of_call_peak_ppr1 = defined(peak_ppr1)\n if ( has_input_of_call_peak_ppr1 && !has_output_of_call_peak_ppr1 && !align_only_ && !true_rep_only && num_rep>1 ) {\n # call peaks on 1st pooled pseudo replicates\n call call_peak as call_peak_ppr1 { input :\n peak_caller = peak_caller_,\n peak_type = peak_type_,\n tas = flatten([select_all([pool_ta_pr1.ta_pooled]), chosen_ctl_ta_pooled]),\n gensz = gensz_,\n chrsz = chrsz_,\n cap_num_peak = cap_num_peak_,\n pval_thresh = pval_thresh,\n ctl_subsample = chosen_ctl_ta_pooled_subsample,\n ctl_paired_end = chosen_ctl_ta_pooled_paired_end,\n fdr_thresh = fdr_thresh,\n fraglen = fraglen_mean.rounded_mean,\n blacklist = blacklist_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n\n cpu = call_peak_cpu,\n mem_mb = call_peak_mem_mb,\n disks = call_peak_disks,\n time_hr = call_peak_time_hr,\n }\n }\n File? peak_ppr1_ = if has_output_of_call_peak_ppr1 then peak_ppr1\n else call_peak_ppr1.peak\n\n Boolean has_input_of_call_peak_ppr2 = defined(pool_ta_pr2.ta_pooled)\n Boolean has_output_of_call_peak_ppr2 = defined(peak_ppr2)\n if ( has_input_of_call_peak_ppr2 && !has_output_of_call_peak_ppr2 && !align_only_ && !true_rep_only && num_rep>1 ) {\n # call peaks on 2nd pooled pseudo replicates\n call call_peak as call_peak_ppr2 { input :\n peak_caller = peak_caller_,\n peak_type = peak_type_,\n tas = flatten([select_all([pool_ta_pr2.ta_pooled]), chosen_ctl_ta_pooled]),\n gensz = gensz_,\n chrsz = chrsz_,\n cap_num_peak = cap_num_peak_,\n pval_thresh = pval_thresh,\n ctl_subsample = chosen_ctl_ta_pooled_subsample,\n ctl_paired_end = chosen_ctl_ta_pooled_paired_end,\n fdr_thresh = fdr_thresh,\n fraglen = fraglen_mean.rounded_mean,\n blacklist = blacklist_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n\n cpu = call_peak_cpu,\n mem_mb = call_peak_mem_mb,\n disks = call_peak_disks,\n time_hr = call_peak_time_hr,\n }\n }\n File? peak_ppr2_ = if has_output_of_call_peak_ppr2 then peak_ppr2\n else call_peak_ppr2.peak\n\n # do IDR/overlap on all pairs of two replicates (i,j)\n # where i and j are zero-based indices and 0 <= i < j < num_rep\n scatter( pair in cross(range(num_rep),range(num_rep)) ) {\n # pair.left = 0-based index of 1st replicate\n # pair.right = 0-based index of 2nd replicate\n File? peak1_ = peak_[pair.left]\n File? peak2_ = peak_[pair.right]\n if ( !align_only_ && pair.left<pair.right ) {\n # Naive overlap on every pair of true replicates\n call overlap { input :\n prefix = 'rep'+(pair.left+1)+'_vs_rep'+(pair.right+1),\n peak1 = peak1_,\n peak2 = peak2_,\n peak_pooled = peak_pooled_,\n fraglen = fraglen_mean.rounded_mean,\n peak_type = peak_type_,\n blacklist = blacklist_,\n chrsz = chrsz_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n ta = pool_ta.ta_pooled,\n }\n }\n if ( enable_idr && !align_only_ && pair.left<pair.right ) {\n # IDR on every pair of true replicates\n call idr { input :\n prefix = 'rep'+(pair.left+1)+'_vs_rep'+(pair.right+1),\n peak1 = peak1_,\n peak2 = peak2_,\n peak_pooled = peak_pooled_,\n fraglen = fraglen_mean.rounded_mean,\n idr_thresh = idr_thresh,\n peak_type = peak_type_,\n rank = idr_rank_,\n blacklist = blacklist_,\n chrsz = chrsz_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n ta = pool_ta.ta_pooled,\n }\n }\n }\n\n # overlap on pseudo-replicates (pr1, pr2) for each true replicate\n if ( !align_only_ && !true_rep_only ) {\n scatter( i in range(num_rep) ) {\n call overlap as overlap_pr { input :\n prefix = 'rep'+(i+1)+'-pr1_vs_rep'+(i+1)+'-pr2',\n peak1 = peak_pr1_[i],\n peak2 = peak_pr2_[i],\n peak_pooled = peak_[i],\n fraglen = fraglen_[i],\n peak_type = peak_type_,\n blacklist = blacklist_,\n chrsz = chrsz_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n ta = ta_[i],\n }\n }\n }\n\n if ( !align_only_ && !true_rep_only && enable_idr ) {\n scatter( i in range(num_rep) ) {\n # IDR on pseduo replicates\n call idr as idr_pr { input :\n prefix = 'rep'+(i+1)+'-pr1_vs_rep'+(i+1)+'-pr2',\n peak1 = peak_pr1_[i],\n peak2 = peak_pr2_[i],\n peak_pooled = peak_[i],\n fraglen = fraglen_[i],\n idr_thresh = idr_thresh,\n peak_type = peak_type_,\n rank = idr_rank_,\n blacklist = blacklist_,\n chrsz = chrsz_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n ta = ta_[i],\n }\n }\n }\n\n if ( !align_only_ && !true_rep_only && num_rep > 1 ) {\n # Naive overlap on pooled pseudo replicates\n call overlap as overlap_ppr { input :\n prefix = 'pooled-pr1_vs_pooled-pr2',\n peak1 = peak_ppr1_,\n peak2 = peak_ppr2_,\n peak_pooled = peak_pooled_,\n peak_type = peak_type_,\n fraglen = fraglen_mean.rounded_mean,\n blacklist = blacklist_,\n chrsz = chrsz_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n ta = pool_ta.ta_pooled,\n }\n }\n\n if ( !align_only_ && !true_rep_only && num_rep > 1 && enable_idr ) {\n # IDR on pooled pseduo replicates\n call idr as idr_ppr { input :\n prefix = 'pooled-pr1_vs_pooled-pr2',\n peak1 = peak_ppr1_,\n peak2 = peak_ppr2_,\n peak_pooled = peak_pooled_,\n idr_thresh = idr_thresh,\n peak_type = peak_type_,\n fraglen = fraglen_mean.rounded_mean,\n rank = idr_rank_,\n blacklist = blacklist_,\n chrsz = chrsz_,\n regex_bfilt_peak_chr_name = regex_bfilt_peak_chr_name_,\n ta = pool_ta.ta_pooled,\n }\n }\n\n # reproducibility QC for overlap/IDR peaks\n if ( !align_only_ && !true_rep_only && num_rep > 0 ) {\n # reproducibility QC for overlapping peaks\n call reproducibility as reproducibility_overlap { input :\n prefix = 'overlap',\n peaks = select_all(overlap.bfilt_overlap_peak),\n peaks_pr = overlap_pr.bfilt_overlap_peak,\n peak_ppr = overlap_ppr.bfilt_overlap_peak,\n peak_type = peak_type_,\n chrsz = chrsz_,\n }\n }\n\n if ( !align_only_ && !true_rep_only && num_rep > 0 && enable_idr ) {\n # reproducibility QC for IDR peaks\n call reproducibility as reproducibility_idr { input :\n prefix = 'idr',\n peaks = select_all(idr.bfilt_idr_peak),\n peaks_pr = idr_pr.bfilt_idr_peak,\n peak_ppr = idr_ppr.bfilt_idr_peak,\n peak_type = peak_type_,\n chrsz = chrsz_,\n }\n }\n\n # Generate final QC report and JSON\n call qc_report { input :\n pipeline_ver = pipeline_ver,\n title = title,\n description = description,\n genome = genome_name_,\n paired_ends = paired_end_,\n ctl_paired_ends = ctl_paired_end_,\n pipeline_type = pipeline_type,\n aligner = aligner_,\n peak_caller = peak_caller_,\n cap_num_peak = cap_num_peak_,\n idr_thresh = idr_thresh,\n pval_thresh = pval_thresh,\n xcor_trim_bp = xcor_trim_bp,\n xcor_subsample_reads = xcor_subsample_reads,\n\n samstat_qcs = select_all(align.samstat_qc),\n nodup_samstat_qcs = select_all(filter.samstat_qc),\n dup_qcs = select_all(filter.dup_qc),\n lib_complexity_qcs = select_all(filter.lib_complexity_qc),\n xcor_plots = select_all(xcor.plot_png),\n xcor_scores = select_all(xcor.score),\n\n ctl_samstat_qcs = select_all(align_ctl.samstat_qc),\n ctl_nodup_samstat_qcs = select_all(filter_ctl.samstat_qc),\n ctl_dup_qcs = select_all(filter_ctl.dup_qc),\n ctl_lib_complexity_qcs = select_all(filter_ctl.lib_complexity_qc),\n\n jsd_plot = jsd.plot,\n jsd_qcs = jsd.jsd_qcs,\n\n frip_qcs = select_all(call_peak.frip_qc),\n frip_qcs_pr1 = select_all(call_peak_pr1.frip_qc),\n frip_qcs_pr2 = select_all(call_peak_pr2.frip_qc),\n frip_qc_pooled = call_peak_pooled.frip_qc,\n frip_qc_ppr1 = call_peak_ppr1.frip_qc,\n frip_qc_ppr2 = call_peak_ppr2.frip_qc,\n\n idr_plots = select_all(idr.idr_plot),\n idr_plots_pr = idr_pr.idr_plot,\n idr_plot_ppr = idr_ppr.idr_plot,\n frip_idr_qcs = select_all(idr.frip_qc),\n frip_idr_qcs_pr = idr_pr.frip_qc,\n frip_idr_qc_ppr = idr_ppr.frip_qc,\n frip_overlap_qcs = select_all(overlap.frip_qc),\n frip_overlap_qcs_pr = overlap_pr.frip_qc,\n frip_overlap_qc_ppr = overlap_ppr.frip_qc,\n idr_reproducibility_qc = reproducibility_idr.reproducibility_qc,\n overlap_reproducibility_qc = reproducibility_overlap.reproducibility_qc,\n\n gc_plots = select_all(gc_bias.gc_plot),\n\n peak_region_size_qcs = select_all(call_peak.peak_region_size_qc),\n peak_region_size_plots = select_all(call_peak.peak_region_size_plot),\n num_peak_qcs = select_all(call_peak.num_peak_qc),\n\n idr_opt_peak_region_size_qc = reproducibility_idr.peak_region_size_qc,\n idr_opt_peak_region_size_plot = reproducibility_overlap.peak_region_size_plot,\n idr_opt_num_peak_qc = reproducibility_idr.num_peak_qc,\n\n overlap_opt_peak_region_size_qc = reproducibility_overlap.peak_region_size_qc,\n overlap_opt_peak_region_size_plot = reproducibility_overlap.peak_region_size_plot,\n overlap_opt_num_peak_qc = reproducibility_overlap.num_peak_qc,\n }\n\n output {\n File report = qc_report.report\n File qc_json = qc_report.qc_json\n Boolean qc_json_ref_match = qc_report.qc_json_ref_match\n }\n}\n\ntask align {\n input {\n Array[File] fastqs_R1 # [merge_id]\n Array[File] fastqs_R2\n Int? trim_bp # this is for R1 only\n Int crop_length\n Int crop_length_tol\n String aligner\n String mito_chr_name\n Int? multimapping\n File? idx_tar # reference index tar\n Boolean paired_end\n Boolean use_bwa_mem_for_pe\n\n String? trimmomatic_java_heap\n Int cpu\n Int mem_mb\n Int time_hr\n String disks\n }\n\n Float trimmomatic_java_heap_factor = 0.9\n Array[Array[File]] tmp_fastqs = if paired_end then transpose([fastqs_R1, fastqs_R2])\n else transpose([fastqs_R1])\n command {\n set -e\n\n # check if pipeline dependencies can be found\n if [[ -z \"$(which encode_task_merge_fastq.py 2> /dev/null || true)\" ]]\n then\n echo -e \"\\n* Error: pipeline dependencies not found.\" 1>&2\n echo 'Conda users: Did you activate Conda environment (conda activate encode-chip-seq-pipeline)?' 1>&2\n echo ' Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2\n echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2\n echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2\n echo -e \"\\n\" 1>&2\n exit 3\n fi\n python3 $(which encode_task_merge_fastq.py) \\\n ${write_tsv(tmp_fastqs)} \\\n ${if paired_end then '--paired-end' else ''} \\\n ${'--nth ' + cpu}\n\n if [ -z '${trim_bp}' ]; then\n SUFFIX=\n else\n SUFFIX=_trimmed\n python3 $(which encode_task_trim_fastq.py) \\\n R1/*.fastq.gz \\\n --trim-bp ${trim_bp} \\\n --out-dir R1$SUFFIX\n if [ '${paired_end}' == 'true' ]; then\n python3 $(which encode_task_trim_fastq.py) \\\n R2/*.fastq.gz \\\n --trim-bp ${trim_bp} \\\n --out-dir R2$SUFFIX\n fi\n fi\n if [ '${crop_length}' == '0' ]; then\n SUFFIX=$SUFFIX\n else\n NEW_SUFFIX=\"$SUFFIX\"_cropped\n python3 $(which encode_task_trimmomatic.py) \\\n --fastq1 R1$SUFFIX/*.fastq.gz \\\n ${if paired_end then '--fastq2 R2$SUFFIX/*.fastq.gz' else ''} \\\n ${if paired_end then '--paired-end' else ''} \\\n --crop-length ${crop_length} \\\n --crop-length-tol \"${crop_length_tol}\" \\\n --out-dir-R1 R1$NEW_SUFFIX \\\n ${if paired_end then '--out-dir-R2 R2$NEW_SUFFIX' else ''} \\\n ${'--trimmomatic-java-heap ' + if defined(trimmomatic_java_heap) then trimmomatic_java_heap else (round(mem_mb * trimmomatic_java_heap_factor) + 'M')} \\\n ${'--nth ' + cpu}\n SUFFIX=$NEW_SUFFIX\n fi\n\n if [ '${aligner}' == 'bwa' ]; then\n python3 $(which encode_task_bwa.py) \\\n ${idx_tar} \\\n R1$SUFFIX/*.fastq.gz \\\n ${if paired_end then 'R2$SUFFIX/*.fastq.gz' else ''} \\\n ${if paired_end then '--paired-end' else ''} \\\n ${if use_bwa_mem_for_pe then '--use-bwa-mem-for-pe' else ''} \\\n ${'--nth ' + cpu}\n\n elif [ '${aligner}' == 'bowtie2' ]; then\n python3 $(which encode_task_bowtie2.py) \\\n ${idx_tar} \\\n R1$SUFFIX/*.fastq.gz \\\n ${if paired_end then 'R2$SUFFIX/*.fastq.gz' else ''} \\\n ${'--multimapping ' + multimapping} \\\n ${if paired_end then '--paired-end' else ''} \\\n ${'--nth ' + cpu}\n fi \n\n python3 $(which encode_task_post_align.py) \\\n R1$SUFFIX/*.fastq.gz $(ls *.bam) \\\n ${'--mito-chr-name ' + mito_chr_name} \\\n ${'--nth ' + cpu}\n rm -rf R1 R2 R1$SUFFIX R2$SUFFIX\n }\n output {\n File bam = glob('*.bam')[0]\n File bai = glob('*.bai')[0]\n File samstat_qc = glob('*.samstats.qc')[0]\n File read_len_log = glob('*.read_length.txt')[0]\n }\n runtime {\n cpu : cpu\n memory : '${mem_mb} MB'\n time : time_hr\n disks : disks\n preemptible: 0\n }\n}\n\ntask filter {\n input {\n File? bam\n Boolean paired_end\n String dup_marker # picard.jar MarkDuplicates (picard) or \n # sambamba markdup (sambamba)\n Int mapq_thresh # threshold for low MAPQ reads removal\n Array[String] filter_chrs # chrs to be removed from final (nodup/filt) BAM\n File chrsz # 2-col chromosome sizes file\n Boolean no_dup_removal # no dupe reads removal when filtering BAM\n String mito_chr_name\n\n Int cpu\n Int mem_mb\n String? picard_java_heap\n Int time_hr\n String disks\n }\n Float picard_java_heap_factor = 0.9\n\n command {\n set -e\n python3 $(which encode_task_filter.py) \\\n ${bam} \\\n ${if paired_end then '--paired-end' else ''} \\\n --multimapping 0 \\\n ${'--dup-marker ' + dup_marker} \\\n ${'--mapq-thresh ' + mapq_thresh} \\\n --filter-chrs ${sep=' ' filter_chrs} \\\n ${'--chrsz ' + chrsz} \\\n ${if no_dup_removal then '--no-dup-removal' else ''} \\\n ${'--mito-chr-name ' + mito_chr_name} \\\n ${'--nth ' + cpu} \\\n ${'--picard-java-heap ' + if defined(picard_java_heap) then picard_java_heap else (round(mem_mb * picard_java_heap_factor) + 'M')}\n }\n output {\n File nodup_bam = glob('*.bam')[0]\n File nodup_bai = glob('*.bai')[0]\n File samstat_qc = glob('*.samstats.qc')[0]\n File dup_qc = glob('*.dup.qc')[0]\n File lib_complexity_qc = glob('*.lib_complexity.qc')[0]\n }\n runtime {\n cpu : cpu\n memory : '${mem_mb} MB'\n time : time_hr\n disks : disks\n }\n}\n\ntask bam2ta {\n input {\n File? bam\n Boolean paired_end\n String mito_chr_name # mito chromosome name\n Int subsample # number of reads to subsample TAGALIGN\n # this affects all downstream analysis\n Int cpu\n Int mem_mb\n Int time_hr\n String disks\n }\n\n command {\n set -e\n python3 $(which encode_task_bam2ta.py) \\\n ${bam} \\\n --disable-tn5-shift \\\n ${if paired_end then '--paired-end' else ''} \\\n ${'--mito-chr-name ' + mito_chr_name} \\\n ${'--subsample ' + subsample} \\\n ${'--nth ' + cpu}\n }\n output {\n File ta = glob('*.tagAlign.gz')[0]\n }\n runtime {\n cpu : cpu\n memory : '${mem_mb} MB'\n time : time_hr\n disks : disks\n }\n}\n\ntask spr {\n input {\n File? ta\n Boolean paired_end\n\n Int mem_mb\n }\n\n command {\n set -e\n python3 $(which encode_task_spr.py) \\\n ${ta} \\\n ${if paired_end then '--paired-end' else ''}\n }\n output {\n File ta_pr1 = glob('*.pr1.tagAlign.gz')[0]\n File ta_pr2 = glob('*.pr2.tagAlign.gz')[0]\n }\n runtime {\n cpu : 1\n memory : '${mem_mb} MB'\n time : 1\n disks : 'local-disk 50 HDD'\n }\n}\n\ntask pool_ta {\n input {\n Array[File?] tas\n Int? col # number of columns in pooled TA\n String? prefix # basename prefix\n }\n\n command {\n set -e\n python3 $(which encode_task_pool_ta.py) \\\n ${sep=' ' select_all(tas)} \\\n ${'--prefix ' + prefix} \\\n ${'--col ' + col}\n }\n output {\n File ta_pooled = glob('*.tagAlign.gz')[0]\n }\n runtime {\n cpu : 1\n memory : '4000 MB'\n time : 1\n disks : 'local-disk 50 HDD'\n }\n}\n\ntask xcor {\n input {\n File? ta\n Boolean paired_end\n String mito_chr_name\n Int subsample # number of reads to subsample TAGALIGN\n # this will be used for xcor only\n # will not affect any downstream analysis\n String? chip_seq_type\n Int? exclusion_range_min\n Int? exclusion_range_max\n\n Int cpu\n Int mem_mb \n Int time_hr\n String disks\n }\n\n command {\n set -e\n python3 $(which encode_task_xcor.py) \\\n ${ta} \\\n ${if paired_end then '--paired-end' else ''} \\\n ${'--mito-chr-name ' + mito_chr_name} \\\n ${'--subsample ' + subsample} \\\n ${'--chip-seq-type ' + chip_seq_type} \\\n ${'--exclusion-range-min ' + exclusion_range_min} \\\n ${'--exclusion-range-max ' + exclusion_range_max} \\\n ${'--subsample ' + subsample} \\\n ${'--nth ' + cpu}\n }\n output {\n File plot_pdf = glob('*.cc.plot.pdf')[0]\n File plot_png = glob('*.cc.plot.png')[0]\n File score = glob('*.cc.qc')[0]\n File fraglen_log = glob('*.cc.fraglen.txt')[0]\n Int fraglen = read_int(fraglen_log)\n }\n runtime {\n cpu : cpu\n memory : '${mem_mb} MB'\n time : time_hr\n disks : disks\n }\n}\n\ntask jsd {\n input {\n Array[File?] nodup_bams\n Array[File?] ctl_bams\n File? blacklist\n Int mapq_thresh\n\n Int cpu\n Int mem_mb\n Int time_hr\n String disks\n }\n\n command {\n set -e\n python3 $(which encode_task_jsd.py) \\\n ${sep=' ' select_all(nodup_bams)} \\\n ${if length(ctl_bams)>0 then '--ctl-bam '+ select_first(ctl_bams) else ''} \\\n ${'--mapq-thresh '+ mapq_thresh} \\\n ${'--blacklist '+ blacklist} \\\n ${'--nth ' + cpu}\n }\n output {\n File plot = glob('*.png')[0]\n Array[File] jsd_qcs = glob('*.jsd.qc')\n }\n runtime {\n cpu : cpu\n memory : '${mem_mb} MB'\n time : time_hr\n disks : disks\n }\n}\n\ntask choose_ctl {\n input {\n Array[File?] tas\n Array[File?] ctl_tas\n File? ta_pooled\n File? ctl_ta_pooled\n Boolean always_use_pooled_ctl # always use pooled control for all exp rep.\n Float ctl_depth_ratio # if ratio between controls is higher than this\n # then always use pooled control for all exp rep.\n Int ctl_depth_limit\n Float exp_ctl_depth_ratio_limit\n }\n\n command {\n set -e\n python3 $(which encode_task_choose_ctl.py) \\\n --tas ${sep=' ' select_all(tas)} \\\n --ctl-tas ${sep=' ' select_all(ctl_tas)} \\\n ${'--ta-pooled ' + ta_pooled} \\\n ${'--ctl-ta-pooled ' + ctl_ta_pooled} \\\n ${if always_use_pooled_ctl then '--always-use-pooled-ctl' else ''} \\\n ${'--ctl-depth-ratio ' + ctl_depth_ratio} \\\n ${'--ctl-depth-limit ' + ctl_depth_limit} \\\n ${'--exp-ctl-depth-ratio-limit ' + exp_ctl_depth_ratio_limit}\n }\n output {\n File chosen_ctl_id_tsv = glob('chosen_ctl.tsv')[0]\n File chosen_ctl_subsample_tsv = glob('chosen_ctl_subsample.tsv')[0]\n File chosen_ctl_subsample_pooled_txt = glob('chosen_ctl_subsample_pooled.txt')[0]\n Array[Int] chosen_ctl_ta_ids = read_lines(chosen_ctl_id_tsv)\n Array[Int] chosen_ctl_ta_subsample = read_lines(chosen_ctl_subsample_tsv)\n Int chosen_ctl_ta_subsample_pooled = read_int(chosen_ctl_subsample_pooled_txt)\n }\n runtime {\n cpu : 1\n memory : '8000 MB'\n time : 1\n disks : 'local-disk 50 HDD'\n } \n}\n\ntask count_signal_track {\n input {\n File? ta # tag-align\n File chrsz # 2-col chromosome sizes file\n }\n\n command {\n set -e\n python3 $(which encode_task_count_signal_track.py) \\\n ${ta} \\\n ${'--chrsz ' + chrsz}\n }\n output {\n File pos_bw = glob('*.positive.bigwig')[0]\n File neg_bw = glob('*.negative.bigwig')[0]\n }\n runtime {\n cpu : 1\n memory : '8000 MB'\n time : 4\n disks : 'local-disk 50 HDD'\n }\n}\n\ntask call_peak {\n input {\n String peak_caller\n String peak_type\n Array[File?] tas # [ta, control_ta]. control_ta is optional\n Int fraglen # fragment length from xcor\n String gensz # Genome size (sum of entries in 2nd column of \n # chr. sizes file, or hs for human, ms for mouse)\n File chrsz # 2-col chromosome sizes file\n Int cap_num_peak # cap number of raw peaks called from MACS2\n Int ctl_subsample # subsample control if >0. 0: no subsampling\n Boolean ctl_paired_end # control is paired end\n Float pval_thresh # p.value threshold for MACS2\n Float? fdr_thresh # FDR threshold for SPP\n\n File? blacklist # blacklist BED to filter raw peaks\n String? regex_bfilt_peak_chr_name\n\n Int cpu \n Int mem_mb\n Int time_hr\n String disks\n }\n\n command {\n set -e\n\n if [ '${peak_caller}' == 'macs2' ]; then\n python3 $(which encode_task_macs2_chip.py) \\\n ${sep=' ' select_all(tas)} \\\n ${'--gensz '+ gensz} \\\n ${'--chrsz ' + chrsz} \\\n ${'--fraglen ' + fraglen} \\\n ${'--cap-num-peak ' + cap_num_peak} \\\n ${'--ctl-subsample ' + ctl_subsample} \\\n ${if ctl_paired_end then '--ctl-paired-end' else ''} \\\n ${'--pval-thresh '+ pval_thresh}\n\n elif [ '${peak_caller}' == 'spp' ]; then\n python3 $(which encode_task_spp.py) \\\n ${sep=' ' select_all(tas)} \\\n ${'--chrsz ' + chrsz} \\\n ${'--fraglen ' + fraglen} \\\n ${'--cap-num-peak ' + cap_num_peak} \\\n ${'--ctl-subsample ' + ctl_subsample} \\\n ${if ctl_paired_end then '--ctl-paired-end' else ''} \\\n ${'--fdr-thresh '+ fdr_thresh} \\\n ${'--nth ' + cpu}\n fi\n\n python3 $(which encode_task_post_call_peak_chip.py) \\\n $(ls *Peak.gz) \\\n ${'--ta ' + tas[0]} \\\n ${'--regex-bfilt-peak-chr-name \\'' + regex_bfilt_peak_chr_name + '\\''} \\\n ${'--chrsz ' + chrsz} \\\n ${'--fraglen ' + fraglen} \\\n ${'--peak-type ' + peak_type} \\\n ${'--blacklist ' + blacklist} \n }\n output {\n File peak = glob('*[!.][!b][!f][!i][!l][!t].'+peak_type+'.gz')[0]\n # generated by post_call_peak py\n File bfilt_peak = glob('*.bfilt.'+peak_type+'.gz')[0]\n File bfilt_peak_bb = glob('*.bfilt.'+peak_type+'.bb')[0]\n File bfilt_peak_hammock = glob('*.bfilt.'+peak_type+'.hammock.gz*')[0]\n File bfilt_peak_hammock_tbi = glob('*.bfilt.'+peak_type+'.hammock.gz*')[1]\n File frip_qc = glob('*.frip.qc')[0]\n File peak_region_size_qc = glob('*.peak_region_size.qc')[0]\n File peak_region_size_plot = glob('*.peak_region_size.png')[0]\n File num_peak_qc = glob('*.num_peak.qc')[0]\n }\n runtime {\n cpu : if peak_caller == 'macs2' then 1 else cpu\n memory : '${mem_mb} MB'\n time : time_hr\n disks : disks\n preemptible: 0 \n }\n}\n\ntask macs2_signal_track {\n input {\n Array[File?] tas # [ta, control_ta]. control_ta is optional\n Int fraglen # fragment length from xcor\n String gensz # Genome size (sum of entries in 2nd column of \n # chr. sizes file, or hs for human, ms for mouse)\n File chrsz # 2-col chromosome sizes file\n Int ctl_subsample # subsample control if >0. 0: no subsampling\n Boolean ctl_paired_end # control is paired end \n Float pval_thresh # p.value threshold\n\n Int mem_mb\n Int time_hr\n String disks\n }\n\n command {\n set -e\n python3 $(which encode_task_macs2_signal_track_chip.py) \\\n ${sep=' ' select_all(tas)} \\\n ${'--gensz '+ gensz} \\\n ${'--chrsz ' + chrsz} \\\n ${'--fraglen ' + fraglen} \\\n ${'--pval-thresh '+ pval_thresh} \\\n ${'--ctl-subsample ' + ctl_subsample} \\\n ${if ctl_paired_end then '--ctl-paired-end' else ''}\n }\n output {\n File pval_bw = glob('*.pval.signal.bigwig')[0]\n File fc_bw = glob('*.fc.signal.bigwig')[0]\n }\n runtime {\n cpu : 1\n memory : '${mem_mb} MB'\n time : time_hr\n disks : disks\n preemptible: 0\n }\n}\n\ntask idr {\n input {\n String prefix # prefix for IDR output file\n File? peak1\n File? peak2\n File? peak_pooled\n Float idr_thresh\n File? blacklist # blacklist BED to filter raw peaks\n String regex_bfilt_peak_chr_name\n # parameters to compute FRiP\n File? ta # to calculate FRiP\n Int? fraglen # fragment length from xcor\n File chrsz # 2-col chromosome sizes file\n String peak_type\n String rank\n }\n\n command {\n set -e\n ${if defined(ta) then '' else 'touch null.frip.qc'}\n touch null \n python3 $(which encode_task_idr.py) \\\n ${peak1} ${peak2} ${peak_pooled} \\\n ${'--prefix ' + prefix} \\\n ${'--idr-thresh ' + idr_thresh} \\\n ${'--peak-type ' + peak_type} \\\n --idr-rank ${rank} \\\n ${'--fraglen ' + fraglen} \\\n ${'--chrsz ' + chrsz} \\\n ${'--blacklist '+ blacklist} \\\n ${'--regex-bfilt-peak-chr-name \\'' + regex_bfilt_peak_chr_name + '\\''} \\\n ${'--ta ' + ta}\n }\n output {\n File idr_peak = glob('*[!.][!b][!f][!i][!l][!t].'+peak_type+'.gz')[0]\n File bfilt_idr_peak = glob('*.bfilt.'+peak_type+'.gz')[0]\n File bfilt_idr_peak_bb = glob('*.bfilt.'+peak_type+'.bb')[0]\n File bfilt_idr_peak_hammock = glob('*.bfilt.'+peak_type+'.hammock.gz*')[0]\n File bfilt_idr_peak_hammock_tbi = glob('*.bfilt.'+peak_type+'.hammock.gz*')[1]\n File idr_plot = glob('*.txt.png')[0]\n File idr_unthresholded_peak = glob('*.txt.gz')[0]\n File idr_log = glob('*.idr*.log')[0]\n File frip_qc = if defined(ta) then glob('*.frip.qc')[0] else glob('null')[0]\n }\n runtime {\n cpu : 1\n memory : '4000 MB'\n time : 1\n disks : 'local-disk 50 HDD'\n } \n}\n\ntask overlap {\n input {\n String prefix # prefix for IDR output file\n File? peak1\n File? peak2\n File? peak_pooled\n File? blacklist # blacklist BED to filter raw peaks\n String regex_bfilt_peak_chr_name\n # parameters to compute FRiP\n File? ta # to calculate FRiP\n Int? fraglen # fragment length from xcor (for FRIP)\n File chrsz # 2-col chromosome sizes file\n String peak_type\n }\n\n command {\n set -e\n ${if defined(ta) then '' else 'touch null.frip.qc'}\n touch null \n python3 $(which encode_task_overlap.py) \\\n ${peak1} ${peak2} ${peak_pooled} \\\n ${'--prefix ' + prefix} \\\n ${'--peak-type ' + peak_type} \\\n ${'--fraglen ' + fraglen} \\\n ${'--chrsz ' + chrsz} \\\n ${'--blacklist '+ blacklist} \\\n --nonamecheck \\\n ${'--regex-bfilt-peak-chr-name \\'' + regex_bfilt_peak_chr_name + '\\''} \\\n ${'--ta ' + ta}\n }\n output {\n File overlap_peak = glob('*[!.][!b][!f][!i][!l][!t].'+peak_type+'.gz')[0]\n File bfilt_overlap_peak = glob('*.bfilt.'+peak_type+'.gz')[0]\n File bfilt_overlap_peak_bb = glob('*.bfilt.'+peak_type+'.bb')[0]\n File bfilt_overlap_peak_hammock = glob('*.bfilt.'+peak_type+'.hammock.gz*')[0]\n File bfilt_overlap_peak_hammock_tbi = glob('*.bfilt.'+peak_type+'.hammock.gz*')[1]\n File frip_qc = if defined(ta) then glob('*.frip.qc')[0] else glob('null')[0]\n }\n runtime {\n cpu : 1\n memory : '4000 MB'\n time : 1\n disks : 'local-disk 50 HDD'\n } \n}\n\ntask reproducibility {\n input {\n String prefix\n Array[File] peaks # peak files from pair of true replicates\n # in a sorted order. for example of 4 replicates,\n # 1,2 1,3 1,4 2,3 2,4 3,4.\n # x,y means peak file from rep-x vs rep-y\n Array[File]? peaks_pr # peak files from pseudo replicates\n File? peak_ppr # Peak file from pooled pseudo replicate.\n String peak_type\n File chrsz # 2-col chromosome sizes file\n }\n\n command {\n set -e\n python3 $(which encode_task_reproducibility.py) \\\n ${sep=' ' peaks} \\\n --peaks-pr ${sep=' ' peaks_pr} \\\n ${'--peak-ppr '+ peak_ppr} \\\n --prefix ${prefix} \\\n ${'--peak-type ' + peak_type} \\\n ${'--chrsz ' + chrsz}\n }\n output {\n File optimal_peak = glob('*optimal_peak.*.gz')[0]\n File optimal_peak_bb = glob('*optimal_peak.*.bb')[0]\n File optimal_peak_hammock = glob('*optimal_peak.*.hammock.gz*')[0]\n File optimal_peak_hammock_tbi = glob('*optimal_peak.*.hammock.gz*')[1]\n File conservative_peak = glob('*conservative_peak.*.gz')[0]\n File conservative_peak_bb = glob('*conservative_peak.*.bb')[0]\n File conservative_peak_hammock = glob('*conservative_peak.*.hammock.gz*')[0]\n File conservative_peak_hammock_tbi = glob('*conservative_peak.*.hammock.gz*')[1]\n File reproducibility_qc = glob('*reproducibility.qc')[0]\n # QC metrics for optimal peak\n File peak_region_size_qc = glob('*.peak_region_size.qc')[0]\n File peak_region_size_plot = glob('*.peak_region_size.png')[0]\n File num_peak_qc = glob('*.num_peak.qc')[0]\n }\n runtime {\n cpu : 1\n memory : '4000 MB'\n time : 1\n disks : 'local-disk 50 HDD'\n } \n}\n\ntask gc_bias {\n input {\n File? nodup_bam\n File ref_fa\n\n String? picard_java_heap\n }\n Int mem_mb = 10000\n Float picard_java_heap_factor = 0.9\n\n command {\n set -e\n python3 $(which encode_task_gc_bias.py) \\\n ${'--nodup-bam ' + nodup_bam} \\\n ${'--ref-fa ' + ref_fa} \\\n ${'--picard-java-heap ' + if defined(picard_java_heap) then picard_java_heap else (round(mem_mb * picard_java_heap_factor) + 'M')}\n }\n output {\n File gc_plot = glob('*.gc_plot.png')[0]\n File gc_log = glob('*.gc.txt')[0]\n }\n runtime {\n cpu : 1\n memory : '${mem_mb} MB'\n time : 6\n disks : 'local-disk 100 HDD'\n }\n}\n\ntask qc_report {\n input {\n # optional metadata\n String pipeline_ver\n String title # name of sample\n String description # description for sample\n String? genome\n #String? encode_accession_id # ENCODE accession ID of sample\n # workflow params\n Array[Boolean] paired_ends\n Array[Boolean] ctl_paired_ends\n String pipeline_type\n String aligner\n String peak_caller\n Int cap_num_peak\n Float idr_thresh\n Float pval_thresh\n Int xcor_trim_bp\n Int xcor_subsample_reads\n # QCs\n Array[File] samstat_qcs\n Array[File] nodup_samstat_qcs\n Array[File] dup_qcs\n Array[File] lib_complexity_qcs\n Array[File] ctl_samstat_qcs\n Array[File] ctl_nodup_samstat_qcs\n Array[File] ctl_dup_qcs\n Array[File] ctl_lib_complexity_qcs\n Array[File] xcor_plots\n Array[File] xcor_scores\n File? jsd_plot\n Array[File]? jsd_qcs\n Array[File] idr_plots\n Array[File]? idr_plots_pr\n File? idr_plot_ppr\n Array[File] frip_qcs\n Array[File] frip_qcs_pr1\n Array[File] frip_qcs_pr2\n File? frip_qc_pooled\n File? frip_qc_ppr1 \n File? frip_qc_ppr2 \n Array[File] frip_idr_qcs\n Array[File]? frip_idr_qcs_pr\n File? frip_idr_qc_ppr \n Array[File] frip_overlap_qcs\n Array[File]? frip_overlap_qcs_pr\n File? frip_overlap_qc_ppr\n File? idr_reproducibility_qc\n File? overlap_reproducibility_qc\n\n Array[File] gc_plots\n\n Array[File] peak_region_size_qcs\n Array[File] peak_region_size_plots\n Array[File] num_peak_qcs\n\n File? idr_opt_peak_region_size_qc\n File? idr_opt_peak_region_size_plot\n File? idr_opt_num_peak_qc\n\n File? overlap_opt_peak_region_size_qc\n File? overlap_opt_peak_region_size_plot\n File? overlap_opt_num_peak_qc\n\n File? qc_json_ref\n }\n\n command {\n set -e\n python3 $(which encode_task_qc_report.py) \\\n ${'--pipeline-ver ' + pipeline_ver} \\\n ${\"--title '\" + sub(title,\"'\",\"_\") + \"'\"} \\\n ${\"--desc '\" + sub(description,\"'\",\"_\") + \"'\"} \\\n ${'--genome ' + genome} \\\n ${'--multimapping ' + 0} \\\n --paired-ends ${sep=' ' paired_ends} \\\n --ctl-paired-ends ${sep=' ' ctl_paired_ends} \\\n --pipeline-type ${pipeline_type} \\\n --aligner ${aligner} \\\n --peak-caller ${peak_caller} \\\n ${'--cap-num-peak ' + cap_num_peak} \\\n --idr-thresh ${idr_thresh} \\\n --pval-thresh ${pval_thresh} \\\n --xcor-trim-bp ${xcor_trim_bp} \\\n --xcor-subsample-reads ${xcor_subsample_reads} \\\n --samstat-qcs ${sep='_:_' samstat_qcs} \\\n --nodup-samstat-qcs ${sep='_:_' nodup_samstat_qcs} \\\n --dup-qcs ${sep='_:_' dup_qcs} \\\n --lib-complexity-qcs ${sep='_:_' lib_complexity_qcs} \\\n --xcor-plots ${sep='_:_' xcor_plots} \\\n --xcor-scores ${sep='_:_' xcor_scores} \\\n --idr-plots ${sep='_:_' idr_plots} \\\n --idr-plots-pr ${sep='_:_' idr_plots_pr} \\\n --ctl-samstat-qcs ${sep='_:_' ctl_samstat_qcs} \\\n --ctl-nodup-samstat-qcs ${sep='_:_' ctl_nodup_samstat_qcs} \\\n --ctl-dup-qcs ${sep='_:_' ctl_dup_qcs} \\\n --ctl-lib-complexity-qcs ${sep='_:_' ctl_lib_complexity_qcs} \\\n ${'--jsd-plot ' + jsd_plot} \\\n --jsd-qcs ${sep='_:_' jsd_qcs} \\\n ${'--idr-plot-ppr ' + idr_plot_ppr} \\\n --frip-qcs ${sep='_:_' frip_qcs} \\\n --frip-qcs-pr1 ${sep='_:_' frip_qcs_pr1} \\\n --frip-qcs-pr2 ${sep='_:_' frip_qcs_pr2} \\\n ${'--frip-qc-pooled ' + frip_qc_pooled} \\\n ${'--frip-qc-ppr1 ' + frip_qc_ppr1} \\\n ${'--frip-qc-ppr2 ' + frip_qc_ppr2} \\\n --frip-idr-qcs ${sep='_:_' frip_idr_qcs} \\\n --frip-idr-qcs-pr ${sep='_:_' frip_idr_qcs_pr} \\\n ${'--frip-idr-qc-ppr ' + frip_idr_qc_ppr} \\\n --frip-overlap-qcs ${sep='_:_' frip_overlap_qcs} \\\n --frip-overlap-qcs-pr ${sep='_:_' frip_overlap_qcs_pr} \\\n ${'--frip-overlap-qc-ppr ' + frip_overlap_qc_ppr} \\\n ${'--idr-reproducibility-qc ' + idr_reproducibility_qc} \\\n ${'--overlap-reproducibility-qc ' + overlap_reproducibility_qc} \\\n --gc-plots ${sep='_:_' gc_plots} \\\n --peak-region-size-qcs ${sep='_:_' peak_region_size_qcs} \\\n --peak-region-size-plots ${sep='_:_' peak_region_size_plots} \\\n --num-peak-qcs ${sep='_:_' num_peak_qcs} \\\n ${'--idr-opt-peak-region-size-qc ' + idr_opt_peak_region_size_qc} \\\n ${'--idr-opt-peak-region-size-plot ' + idr_opt_peak_region_size_plot} \\\n ${'--idr-opt-num-peak-qc ' + idr_opt_num_peak_qc} \\\n ${'--overlap-opt-peak-region-size-qc ' + overlap_opt_peak_region_size_qc} \\\n ${'--overlap-opt-peak-region-size-plot ' + overlap_opt_peak_region_size_plot} \\\n ${'--overlap-opt-num-peak-qc ' + overlap_opt_num_peak_qc} \\\n --out-qc-html qc.html \\\n --out-qc-json qc.json \\\n ${'--qc-json-ref ' + qc_json_ref}\n }\n output {\n File report = glob('*qc.html')[0]\n File qc_json = glob('*qc.json')[0]\n Boolean qc_json_ref_match = read_string('qc_json_ref_match.txt')=='True'\n }\n runtime {\n cpu : 1\n memory : '4000 MB'\n time : 1\n disks : 'local-disk 50 HDD'\n } \n}\n\n### workflow system tasks\ntask read_genome_tsv {\n input {\n File? genome_tsv\n String? null_s\n }\n command <<<\n echo \"$(basename ~{genome_tsv})\" > genome_name\n # create empty files for all entries\n touch ref_fa bowtie2_idx_tar bwa_idx_tar chrsz gensz blacklist blacklist2\n touch mito_chr_name\n touch regex_bfilt_peak_chr_name\n\n python <<CODE\n import os\n with open('~{genome_tsv}','r') as fp:\n for line in fp:\n arr = line.strip('\\n').split('\\t')\n if arr:\n key, val = arr\n with open(key,'w') as fp2:\n fp2.write(val)\n CODE\n >>>\n output {\n String? genome_name = read_string('genome_name')\n String? ref_fa = if size('ref_fa')==0 then null_s else read_string('ref_fa')\n String? bwa_idx_tar = if size('bwa_idx_tar')==0 then null_s else read_string('bwa_idx_tar')\n String? bowtie2_idx_tar = if size('bowtie2_idx_tar')==0 then null_s else read_string('bowtie2_idx_tar')\n String? chrsz = if size('chrsz')==0 then null_s else read_string('chrsz')\n String? gensz = if size('gensz')==0 then null_s else read_string('gensz')\n String? blacklist = if size('blacklist')==0 then null_s else read_string('blacklist')\n String? blacklist2 = if size('blacklist2')==0 then null_s else read_string('blacklist2')\n String? mito_chr_name = if size('mito_chr_name')==0 then null_s else read_string('mito_chr_name')\n String? regex_bfilt_peak_chr_name = if size('regex_bfilt_peak_chr_name')==0 then 'chr[\\\\dXY]+'\n else read_string('regex_bfilt_peak_chr_name')\n }\n runtime {\n maxRetries : 0\n cpu : 1\n memory : '4000 MB'\n time : 1\n disks : 'local-disk 50 HDD' \n }\n}\n\ntask rounded_mean {\n input {\n Array[Int] ints\n }\n command <<<\n python <<CODE\n arr = [~{sep=',' ints}]\n with open('tmp.txt','w') as fp:\n if len(arr):\n sum_ = sum(arr)\n mean_ = sum(arr)/float(len(arr))\n fp.write('{}'.format(int(round(mean_))))\n else:\n fp.write('0')\n CODE\n >>>\n output {\n Int rounded_mean = read_int('tmp.txt')\n }\n runtime {\n cpu : 1\n memory : '4000 MB'\n time : 1\n disks : 'local-disk 50 HDD'\n } \n}\n\ntask raise_exception {\n input {\n String msg\n Array[String]? vals\n }\n command {\n echo -e \"\\n* Error: ${msg}\\n\" >&2\n echo -e \"* Vals: ${sep=',' vals}\\n\" >&2\n exit 2\n }\n output {\n String error_msg = '${msg}'\n }\n runtime {\n maxRetries : 0\n }\n}\n",
"root": "",
"options": "{\n \"backend\": \"Local\",\n \"default_runtime_attributes\": {\n \"maxRetries\": 1\n }\n}",
"inputs": "{\n \"chip.pipeline_type\": \"tf\",\n \"chip.genome_tsv\": \"/PHShome/bk724/H3K27ac_sample/23af10fb66962b8784fbace80dd24dee/hg19_caper.local.tsv\",\n \"chip.fastqs_rep1_R1\": [\n \"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_H3K27ac_R1.fastq.gz\"\n ],\n \"chip.fastqs_rep1_R2\": [\n \"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_H3K27ac_R2.fastq.gz\"\n ],\n \"chip.ctl_fastqs_rep1_R1\": [\n \"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_input_R1.fastq.gz\"\n ],\n \"chip.ctl_fastqs_rep1_R2\": [\n \"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_input_R2.fastq.gz\"\n ],\n \"chip.paired_end\": true,\n \"chip.title\": \"H3K27ac\",\n \"chip.description\": \"CLLepigenome_for_dbGAP by the Ott lab\"\n}",
"workflowUrl": "/PHShome/bk724/chip-seq-pipeline2/chip.wdl",
"labels": "{\n \"caper-backend\": \"Local\",\n \"caper-str-label\": \"H3K27ac_sample\",\n \"caper-user\": \"bk724\"\n}"
},
"calls": {
"chip.filter_R1": [
{
"executionStatus": "Done",
"stdout": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0/execution/stdout",
"backendStatus": "Done",
"commandLine": "set -e\npython3 $(which encode_task_filter.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0/inputs/964644000/JB0009_H3K27ac_R1.trim_50bp.bam \\\n \\\n --multimapping 0 \\\n --dup-marker picard \\\n --mapq-thresh 30 \\\n --filter-chrs \\\n --chrsz /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0/inputs/415522508/hg19.chrom.sizes \\\n --no-dup-removal \\\n --mito-chr-name chrM \\\n --nth 2 \\\n --picard-java-heap 18000M",
"shardIndex": 0,
"outputs": {
"dup_qc": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0/execution/glob-1b4794e80f740fd50137e3a948d7ba79/JB0009_H3K27ac_R1.trim_50bp.dup.qc",
"samstat_qc": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0/execution/glob-7b38d9959cf6f3deb83ac2bd156d8317/JB0009_H3K27ac_R1.trim_50bp.filt.samstats.qc",
"nodup_bai": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0/execution/glob-6efbc60cb1e0959bab4e467327a9416c/JB0009_H3K27ac_R1.trim_50bp.filt.bam.bai",
"nodup_bam": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0/execution/glob-3bcbe4e7489c90f75e0523ac6f3a9385/JB0009_H3K27ac_R1.trim_50bp.filt.bam",
"lib_complexity_qc": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0/execution/glob-57d0acc485690c2359d57a5a21e0aedb/JB0009_H3K27ac_R1.trim_50bp.lib_complexity.qc"
},
"runtimeAttributes": {
"maxRetries": "1",
"failOnStderr": "false",
"continueOnReturnCode": "0"
},
"callCaching": {
"allowResultReuse": true,
"hit": false,
"result": "Cache Miss",
"hashes": {
"output count": "E4DA3B7FBBCE2345D7772B0674A318D5",
"runtime attribute": {
"docker": "N/A",
"failOnStderr": "68934A3E9455FA72420237EB05902327",
"continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA"
},
"output expression": {
"File samstat_qc": "EF7F94200A66D0F363422B1FF5348D90",
"File dup_qc": "D34C4EB2504C6E5EE8A8B6FF197A9EC7",
"File nodup_bai": "DC8EF4DE24731F8D0B23B82E2CA15141",
"File lib_complexity_qc": "84CE21EEA5C3132AC521760733899074",
"File nodup_bam": "4D95F2D23046D5295A149CA785FD216E"
},
"input count": "C20AD4D76FE97759AA27A0C99BFF6710",
"backend name": "509820290D57F333403F490DDE7316F4",
"command template": "AFCF4863CE65A953C90B0A43D3EB1D21",
"input": {
"Int mem_mb": "D9798CDF31C02D86B8B81CC119D94836",
"String disks": "085EEFBDBB0F02BAD180CD29A54CCBA2",
"String mito_chr_name": "4A8385B4D27A8EEDB503793999FE2D1B",
"Boolean paired_end": "68934A3E9455FA72420237EB05902327",
"Int time_hr": "1FF1DE774005F8DA13F42943881C655F",
"File chrsz": "82e36e01fa26480931039ac178ad8e83",
"Int cpu": "C81E728D9D4C2F636F067F89CC14862C",
"Float picard_java_heap_factor": "A894124CC6D5C5C71AFE060D5DDE0762",
"String dup_marker": "620573C97469647DF116392F8B351FEB",
"Boolean no_dup_removal": "B326B5062B2F0E69046810717534CB09",
"File bam": "21b088b89ec8ab5132e09662f0e2b123",
"Int mapq_thresh": "34173CB38F07F89DDBEBC2AC9128303F"
}
},
"effectiveCallCachingMode": "ReadAndWriteCache"
},
"inputs": {
"no_dup_removal": true,
"mem_mb": 20000,
"time_hr": 24,
"mito_chr_name": "chrM",
"chrsz": "/PHShome/bk724/H3K27ac_sample/cf672570da74971be478218e3c209190/hg19.chrom.sizes",
"picard_java_heap": null,
"disks": "local-disk 400 HDD",
"picard_java_heap_factor": 0.9,
"mapq_thresh": 30,
"cpu": 2,
"paired_end": false,
"dup_marker": "picard",
"bam": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0/execution/glob-3bcbe4e7489c90f75e0523ac6f3a9385/JB0009_H3K27ac_R1.trim_50bp.bam",
"filter_chrs": []
},
"returnCode": 0,
"jobId": "7470",
"backend": "Local",
"end": "2020-07-15T15:50:43.320Z",
"stderr": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0/execution/stderr",
"callRoot": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0",
"attempt": 1,
"executionEvents": [
{
"startTime": "2020-07-15T15:12:03.393Z",
"endTime": "2020-07-15T15:50:42.717Z",
"description": "RunningJob"
},
{
"endTime": "2020-07-15T15:12:03.374Z",
"description": "RequestingExecutionToken",
"startTime": "2020-07-15T15:12:03.267Z"
},
{
"startTime": "2020-07-15T15:12:03.374Z",
"description": "PreparingJob",
"endTime": "2020-07-15T15:12:03.381Z"
},
{
"startTime": "2020-07-15T15:50:42.717Z",
"endTime": "2020-07-15T15:50:43.319Z",
"description": "UpdatingCallCache"
},
{
"endTime": "2020-07-15T15:50:43.320Z",
"description": "UpdatingJobStore",
"startTime": "2020-07-15T15:50:43.319Z"
},
{
"startTime": "2020-07-15T15:12:03.381Z",
"description": "CallCacheReading",
"endTime": "2020-07-15T15:12:03.393Z"
},
{
"startTime": "2020-07-15T15:12:03.374Z",
"description": "WaitingForValueStore",
"endTime": "2020-07-15T15:12:03.374Z"
},
{
"startTime": "2020-07-15T15:12:03.267Z",
"description": "Pending",
"endTime": "2020-07-15T15:12:03.267Z"
}
],
"start": "2020-07-15T15:12:03.266Z"
}
],
"chip.align_R1": [
{
"executionStatus": "Done",
"stdout": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0/execution/stdout",
"backendStatus": "Done",
"commandLine": "set -e\n\n# check if pipeline dependencies can be found\nif [[ -z \"$(which encode_task_merge_fastq.py 2> /dev/null || true)\" ]]\nthen\n echo -e \"\\n* Error: pipeline dependencies not found.\" 1>&2\n echo 'Conda users: Did you activate Conda environment (conda activate encode-chip-seq-pipeline)?' 1>&2\n echo ' Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2\n echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2\n echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2\n echo -e \"\\n\" 1>&2\n exit 3\nfi\npython3 $(which encode_task_merge_fastq.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0/execution/write_tsv_c4c4719758551b412a2995c93d909a1d.tmp \\\n \\\n --nth 4\n\nif [ -z '50' ]; then\n SUFFIX=\nelse\n SUFFIX=_trimmed\n python3 $(which encode_task_trim_fastq.py) \\\n R1/*.fastq.gz \\\n --trim-bp 50 \\\n --out-dir R1$SUFFIX\n if [ 'false' == 'true' ]; then\n python3 $(which encode_task_trim_fastq.py) \\\n R2/*.fastq.gz \\\n --trim-bp 50 \\\n --out-dir R2$SUFFIX\n fi\nfi\nif [ '0' == '0' ]; then\n SUFFIX=$SUFFIX\nelse\n NEW_SUFFIX=\"$SUFFIX\"_cropped\n python3 $(which encode_task_trimmomatic.py) \\\n --fastq1 R1$SUFFIX/*.fastq.gz \\\n \\\n \\\n --crop-length 0 \\\n --crop-length-tol \"0\" \\\n --out-dir-R1 R1$NEW_SUFFIX \\\n \\\n --trimmomatic-java-heap 18000M \\\n --nth 4\n SUFFIX=$NEW_SUFFIX\nfi\n\nif [ 'bowtie2' == 'bwa' ]; then\n python3 $(which encode_task_bwa.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0/inputs/-498531974/male.hg19.fa.tar \\\n R1$SUFFIX/*.fastq.gz \\\n \\\n \\\n \\\n --nth 4\n\nelif [ 'bowtie2' == 'bowtie2' ]; then\n python3 $(which encode_task_bowtie2.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0/inputs/-498531974/male.hg19.fa.tar \\\n R1$SUFFIX/*.fastq.gz \\\n \\\n \\\n \\\n --nth 4\nfi \n\npython3 $(which encode_task_post_align.py) \\\n R1$SUFFIX/*.fastq.gz $(ls *.bam) \\\n --mito-chr-name chrM \\\n --nth 4\nrm -rf R1 R2 R1$SUFFIX R2$SUFFIX",
"shardIndex": 0,
"outputs": {
"read_len_log": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0/execution/glob-773fb92850749a2b4a829cf3c8c4de27/JB0009_H3K27ac_R1.trim_50bp.read_length.txt",
"samstat_qc": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0/execution/glob-7b38d9959cf6f3deb83ac2bd156d8317/JB0009_H3K27ac_R1.trim_50bp.samstats.qc",
"bai": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0/execution/glob-6efbc60cb1e0959bab4e467327a9416c/JB0009_H3K27ac_R1.trim_50bp.bam.bai",
"bam": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0/execution/glob-3bcbe4e7489c90f75e0523ac6f3a9385/JB0009_H3K27ac_R1.trim_50bp.bam"
},
"runtimeAttributes": {
"maxRetries": "1",
"failOnStderr": "false",
"continueOnReturnCode": "0"
},
"callCaching": {
"allowResultReuse": true,
"hit": false,
"result": "Cache Miss",
"hashes": {
"output count": "A87FF679A2F3E71D9181A67B7542122C",
"runtime attribute": {
"docker": "N/A",
"failOnStderr": "68934A3E9455FA72420237EB05902327",
"continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA"
},
"output expression": {
"File bam": "4D95F2D23046D5295A149CA785FD216E",
"File read_len_log": "CC7C87A75C5BFD29B68744DB9D1A3E61",
"File samstat_qc": "EF7F94200A66D0F363422B1FF5348D90",
"File bai": "DC8EF4DE24731F8D0B23B82E2CA15141"
},
"input count": "9BF31C7FF062936A96D3C8BD1F8F2FF3",
"backend name": "509820290D57F333403F490DDE7316F4",
"command template": "1A155C84DB5CCA39EB57AF59A522131C",
"input": {
"Int mem_mb": "D9798CDF31C02D86B8B81CC119D94836",
"File tmp_fastqs": [
[
"5d3a391d9c648e6bfbb62ac42e99a5fa"
]
],
"File fastqs_R1": [
"5d3a391d9c648e6bfbb62ac42e99a5fa"
],
"String disks": "085EEFBDBB0F02BAD180CD29A54CCBA2",
"String mito_chr_name": "4A8385B4D27A8EEDB503793999FE2D1B",
"Int crop_length_tol": "CFCD208495D565EF66E7DFF9F98764DA",
"Boolean paired_end": "68934A3E9455FA72420237EB05902327",
"Int trim_bp": "C0C7C76D30BD3DCAEFC96F40275BDC0A",
"Int time_hr": "642E92EFB79421734881B53E1E1B18B6",
"Int crop_length": "CFCD208495D565EF66E7DFF9F98764DA",
"Int cpu": "A87FF679A2F3E71D9181A67B7542122C",
"File idx_tar": "08b74cce1d9551f812d9f8a7ab4feddf",
"Boolean use_bwa_mem_for_pe": "68934A3E9455FA72420237EB05902327",
"String aligner": "99E6A6AC2CD2C1E66B6C4CA6FE0D3837",
"Float trimmomatic_java_heap_factor": "A894124CC6D5C5C71AFE060D5DDE0762"
}
},
"effectiveCallCachingMode": "ReadAndWriteCache"
},
"inputs": {
"trimmomatic_java_heap_factor": 0.9,
"trimmomatic_java_heap": null,
"use_bwa_mem_for_pe": false,
"fastqs_R2": [],
"multimapping": null,
"mem_mb": 20000,
"time_hr": 48,
"crop_length": 0,
"mito_chr_name": "chrM",
"tmp_fastqs": [
[
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_H3K27ac_R1.fastq.gz"
]
],
"disks": "local-disk 400 HDD",
"fastqs_R1": [
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_H3K27ac_R1.fastq.gz"
],
"trim_bp": 50,
"aligner": "bowtie2",
"cpu": 4,
"paired_end": false,
"crop_length_tol": 0,
"idx_tar": "/PHShome/bk724/H3K27ac_sample/e28e11c41785d0b78376bed93bb0795e/male.hg19.fa.tar"
},
"returnCode": 0,
"jobId": "26249",
"backend": "Local",
"end": "2020-07-15T15:12:01.323Z",
"stderr": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0/execution/stderr",
"callRoot": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_R1/shard-0",
"attempt": 1,
"executionEvents": [
{
"description": "UpdatingJobStore",
"endTime": "2020-07-15T15:12:01.322Z",
"startTime": "2020-07-15T15:12:01.322Z"
},
{
"startTime": "2020-07-15T13:13:19.373Z",
"description": "WaitingForValueStore",
"endTime": "2020-07-15T13:13:19.374Z"
},
{
"startTime": "2020-07-15T13:13:14.427Z",
"endTime": "2020-07-15T13:13:14.427Z",
"description": "Pending"
},
{
"startTime": "2020-07-15T13:13:14.427Z",
"description": "RequestingExecutionToken",
"endTime": "2020-07-15T13:13:19.373Z"
},
{
"startTime": "2020-07-15T15:11:59.297Z",
"endTime": "2020-07-15T15:12:01.322Z",
"description": "UpdatingCallCache"
},
{
"startTime": "2020-07-15T13:13:19.382Z",
"description": "RunningJob",
"endTime": "2020-07-15T15:11:59.297Z"
},
{
"startTime": "2020-07-15T13:13:19.374Z",
"description": "PreparingJob",
"endTime": "2020-07-15T13:13:19.378Z"
},
{
"startTime": "2020-07-15T13:13:19.378Z",
"description": "CallCacheReading",
"endTime": "2020-07-15T13:13:19.382Z"
}
],
"start": "2020-07-15T13:13:14.426Z"
}
],
"chip.align": [
{
"executionStatus": "Done",
"stdout": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align/shard-0/execution/stdout",
"backendStatus": "Done",
"commandLine": "set -e\n\n# check if pipeline dependencies can be found\nif [[ -z \"$(which encode_task_merge_fastq.py 2> /dev/null || true)\" ]]\nthen\n echo -e \"\\n* Error: pipeline dependencies not found.\" 1>&2\n echo 'Conda users: Did you activate Conda environment (conda activate encode-chip-seq-pipeline)?' 1>&2\n echo ' Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2\n echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2\n echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2\n echo -e \"\\n\" 1>&2\n exit 3\nfi\npython3 $(which encode_task_merge_fastq.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align/shard-0/execution/write_tsv_d1d1a8a7b1ba477a5aeb45552cfc88ed.tmp \\\n --paired-end \\\n --nth 4\n\nif [ -z '' ]; then\n SUFFIX=\nelse\n SUFFIX=_trimmed\n python3 $(which encode_task_trim_fastq.py) \\\n R1/*.fastq.gz \\\n --trim-bp \\\n --out-dir R1$SUFFIX\n if [ 'true' == 'true' ]; then\n python3 $(which encode_task_trim_fastq.py) \\\n R2/*.fastq.gz \\\n --trim-bp \\\n --out-dir R2$SUFFIX\n fi\nfi\nif [ '0' == '0' ]; then\n SUFFIX=$SUFFIX\nelse\n NEW_SUFFIX=\"$SUFFIX\"_cropped\n python3 $(which encode_task_trimmomatic.py) \\\n --fastq1 R1$SUFFIX/*.fastq.gz \\\n --fastq2 R2$SUFFIX/*.fastq.gz \\\n --paired-end \\\n --crop-length 0 \\\n --crop-length-tol \"2\" \\\n --out-dir-R1 R1$NEW_SUFFIX \\\n --out-dir-R2 R2$NEW_SUFFIX \\\n --trimmomatic-java-heap 18000M \\\n --nth 4\n SUFFIX=$NEW_SUFFIX\nfi\n\nif [ 'bowtie2' == 'bwa' ]; then\n python3 $(which encode_task_bwa.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align/shard-0/inputs/-498531974/male.hg19.fa.tar \\\n R1$SUFFIX/*.fastq.gz \\\n R2$SUFFIX/*.fastq.gz \\\n --paired-end \\\n \\\n --nth 4\n\nelif [ 'bowtie2' == 'bowtie2' ]; then\n python3 $(which encode_task_bowtie2.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align/shard-0/inputs/-498531974/male.hg19.fa.tar \\\n R1$SUFFIX/*.fastq.gz \\\n R2$SUFFIX/*.fastq.gz \\\n \\\n --paired-end \\\n --nth 4\nfi \n\npython3 $(which encode_task_post_align.py) \\\n R1$SUFFIX/*.fastq.gz $(ls *.bam) \\\n --mito-chr-name chrM \\\n --nth 4\nrm -rf R1 R2 R1$SUFFIX R2$SUFFIX",
"shardIndex": 0,
"outputs": {
"read_len_log": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align/shard-0/execution/glob-773fb92850749a2b4a829cf3c8c4de27/JB0009_H3K27ac_R1.read_length.txt",
"samstat_qc": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align/shard-0/execution/glob-7b38d9959cf6f3deb83ac2bd156d8317/JB0009_H3K27ac_R1.samstats.qc",
"bai": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align/shard-0/execution/glob-6efbc60cb1e0959bab4e467327a9416c/JB0009_H3K27ac_R1.bam.bai",
"bam": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align/shard-0/execution/glob-3bcbe4e7489c90f75e0523ac6f3a9385/JB0009_H3K27ac_R1.bam"
},
"runtimeAttributes": {
"maxRetries": "1",
"failOnStderr": "false",
"continueOnReturnCode": "0"
},
"callCaching": {
"allowResultReuse": true,
"hit": false,
"result": "Cache Miss",
"hashes": {
"output count": "A87FF679A2F3E71D9181A67B7542122C",
"runtime attribute": {
"docker": "N/A",
"continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA",
"failOnStderr": "68934A3E9455FA72420237EB05902327"
},
"output expression": {
"File bam": "4D95F2D23046D5295A149CA785FD216E",
"File read_len_log": "CC7C87A75C5BFD29B68744DB9D1A3E61",
"File samstat_qc": "EF7F94200A66D0F363422B1FF5348D90",
"File bai": "DC8EF4DE24731F8D0B23B82E2CA15141"
},
"input count": "C74D97B01EAE257E44AA9D5BADE97BAF",
"backend name": "509820290D57F333403F490DDE7316F4",
"command template": "1A155C84DB5CCA39EB57AF59A522131C",
"input": {
"Int mem_mb": "D9798CDF31C02D86B8B81CC119D94836",
"File tmp_fastqs": [
[
"5d3a391d9c648e6bfbb62ac42e99a5fa",
"3919ee1b3421eabd4b9f0c66c6734124"
]
],
"File fastqs_R1": [
"5d3a391d9c648e6bfbb62ac42e99a5fa"
],
"String disks": "085EEFBDBB0F02BAD180CD29A54CCBA2",
"String mito_chr_name": "4A8385B4D27A8EEDB503793999FE2D1B",
"Int crop_length_tol": "C81E728D9D4C2F636F067F89CC14862C",
"Boolean paired_end": "B326B5062B2F0E69046810717534CB09",
"Int time_hr": "642E92EFB79421734881B53E1E1B18B6",
"Int crop_length": "CFCD208495D565EF66E7DFF9F98764DA",
"File fastqs_R2": [
"3919ee1b3421eabd4b9f0c66c6734124"
],
"Int cpu": "A87FF679A2F3E71D9181A67B7542122C",
"File idx_tar": "08b74cce1d9551f812d9f8a7ab4feddf",
"Boolean use_bwa_mem_for_pe": "68934A3E9455FA72420237EB05902327",
"String aligner": "99E6A6AC2CD2C1E66B6C4CA6FE0D3837",
"Float trimmomatic_java_heap_factor": "A894124CC6D5C5C71AFE060D5DDE0762"
}
},
"effectiveCallCachingMode": "ReadAndWriteCache"
},
"inputs": {
"trimmomatic_java_heap_factor": 0.9,
"trimmomatic_java_heap": null,
"use_bwa_mem_for_pe": false,
"fastqs_R2": [
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_H3K27ac_R2.fastq.gz"
],
"multimapping": null,
"mem_mb": 20000,
"time_hr": 48,
"crop_length": 0,
"mito_chr_name": "chrM",
"tmp_fastqs": [
[
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_H3K27ac_R1.fastq.gz",
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_H3K27ac_R2.fastq.gz"
]
],
"disks": "local-disk 400 HDD",
"fastqs_R1": [
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_H3K27ac_R1.fastq.gz"
],
"trim_bp": null,
"aligner": "bowtie2",
"cpu": 4,
"paired_end": true,
"crop_length_tol": 2,
"idx_tar": "/PHShome/bk724/H3K27ac_sample/e28e11c41785d0b78376bed93bb0795e/male.hg19.fa.tar"
},
"returnCode": 0,
"jobId": "26160",
"backend": "Local",
"end": "2020-07-15T20:21:07.311Z",
"stderr": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align/shard-0/execution/stderr",
"callRoot": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align/shard-0",
"attempt": 1,
"executionEvents": [
{
"startTime": "2020-07-15T13:13:14.426Z",
"description": "Pending",
"endTime": "2020-07-15T13:13:14.426Z"
},
{
"description": "UpdatingJobStore",
"startTime": "2020-07-15T20:21:07.311Z",
"endTime": "2020-07-15T20:21:07.311Z"
},
{
"description": "UpdatingCallCache",
"startTime": "2020-07-15T20:21:05.519Z",
"endTime": "2020-07-15T20:21:07.311Z"
},
{
"description": "RequestingExecutionToken",
"startTime": "2020-07-15T13:13:14.426Z",
"endTime": "2020-07-15T13:13:15.373Z"
},
{
"startTime": "2020-07-15T13:13:15.373Z",
"description": "PreparingJob",
"endTime": "2020-07-15T13:13:15.387Z"
},
{
"endTime": "2020-07-15T13:13:15.373Z",
"startTime": "2020-07-15T13:13:15.373Z",
"description": "WaitingForValueStore"
},
{
"endTime": "2020-07-15T13:13:15.395Z",
"startTime": "2020-07-15T13:13:15.387Z",
"description": "CallCacheReading"
},
{
"description": "RunningJob",
"endTime": "2020-07-15T20:21:05.519Z",
"startTime": "2020-07-15T13:13:15.395Z"
}
],
"start": "2020-07-15T13:13:14.426Z"
}
],
"chip.filter_ctl": [
{
"retryableFailure": true,
"executionStatus": "RetryableFailure",
"stdout": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/execution/stdout",
"backendStatus": "Done",
"commandLine": "set -e\npython3 $(which encode_task_filter.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/inputs/1588305472/JB0009_input_R1.bam \\\n --paired-end \\\n --multimapping 0 \\\n --dup-marker picard \\\n --mapq-thresh 30 \\\n --filter-chrs \\\n --chrsz /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/inputs/415522508/hg19.chrom.sizes \\\n \\\n --mito-chr-name chrM \\\n --nth 2 \\\n --picard-java-heap 18000M",
"shardIndex": 0,
"runtimeAttributes": {
"maxRetries": "1",
"failOnStderr": "false",
"continueOnReturnCode": "0"
},
"callCaching": {
"allowResultReuse": false,
"hit": false,
"result": "Cache Miss",
"hashes": {
"output count": "E4DA3B7FBBCE2345D7772B0674A318D5",
"runtime attribute": {
"docker": "N/A",
"failOnStderr": "68934A3E9455FA72420237EB05902327",
"continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA"
},
"output expression": {
"File samstat_qc": "EF7F94200A66D0F363422B1FF5348D90",
"File dup_qc": "D34C4EB2504C6E5EE8A8B6FF197A9EC7",
"File nodup_bai": "DC8EF4DE24731F8D0B23B82E2CA15141",
"File lib_complexity_qc": "84CE21EEA5C3132AC521760733899074",
"File nodup_bam": "4D95F2D23046D5295A149CA785FD216E"
},
"input count": "C20AD4D76FE97759AA27A0C99BFF6710",
"backend name": "509820290D57F333403F490DDE7316F4",
"command template": "AFCF4863CE65A953C90B0A43D3EB1D21",
"input": {
"Int mem_mb": "D9798CDF31C02D86B8B81CC119D94836",
"String disks": "085EEFBDBB0F02BAD180CD29A54CCBA2",
"String mito_chr_name": "4A8385B4D27A8EEDB503793999FE2D1B",
"Boolean paired_end": "B326B5062B2F0E69046810717534CB09",
"Int time_hr": "1FF1DE774005F8DA13F42943881C655F",
"File chrsz": "82e36e01fa26480931039ac178ad8e83",
"Int cpu": "C81E728D9D4C2F636F067F89CC14862C",
"Float picard_java_heap_factor": "A894124CC6D5C5C71AFE060D5DDE0762",
"String dup_marker": "620573C97469647DF116392F8B351FEB",
"Boolean no_dup_removal": "68934A3E9455FA72420237EB05902327",
"File bam": "0a1ea2408bb732cca23d2a1f728e189e",
"Int mapq_thresh": "34173CB38F07F89DDBEBC2AC9128303F"
}
},
"effectiveCallCachingMode": "ReadAndWriteCache"
},
"inputs": {
"no_dup_removal": false,
"mem_mb": 20000,
"time_hr": 24,
"mito_chr_name": "chrM",
"chrsz": "/PHShome/bk724/H3K27ac_sample/cf672570da74971be478218e3c209190/hg19.chrom.sizes",
"picard_java_heap": null,
"disks": "local-disk 400 HDD",
"picard_java_heap_factor": 0.9,
"mapq_thresh": 30,
"cpu": 2,
"paired_end": true,
"dup_marker": "picard",
"bam": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/execution/glob-3bcbe4e7489c90f75e0523ac6f3a9385/JB0009_input_R1.bam",
"filter_chrs": []
},
"returnCode": 1,
"failures": [
{
"message": "Job chip.filter_ctl:0:1 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
}
],
"jobId": "43561",
"backend": "Local",
"end": "2020-07-15T19:55:14.389Z",
"stderr": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/execution/stderr",
"callRoot": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0",
"attempt": 1,
"executionEvents": [
{
"startTime": "2020-07-15T19:32:13.380Z",
"endTime": "2020-07-15T19:55:14.389Z",
"description": "RunningJob"
},
{
"startTime": "2020-07-15T19:32:13.377Z",
"endTime": "2020-07-15T19:32:13.380Z",
"description": "CallCacheReading"
},
{
"startTime": "2020-07-15T19:32:13.374Z",
"endTime": "2020-07-15T19:32:13.374Z",
"description": "WaitingForValueStore"
},
{
"endTime": "2020-07-15T19:32:12.597Z",
"startTime": "2020-07-15T19:32:12.597Z",
"description": "Pending"
},
{
"startTime": "2020-07-15T19:32:13.374Z",
"endTime": "2020-07-15T19:32:13.377Z",
"description": "PreparingJob"
},
{
"startTime": "2020-07-15T19:32:12.597Z",
"endTime": "2020-07-15T19:32:13.374Z",
"description": "RequestingExecutionToken"
},
{
"startTime": "2020-07-15T19:55:14.389Z",
"endTime": "2020-07-15T19:55:14.389Z",
"description": "UpdatingJobStore"
}
],
"start": "2020-07-15T19:32:12.597Z"
},
{
"retryableFailure": false,
"executionStatus": "Failed",
"stdout": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/attempt-2/execution/stdout",
"backendStatus": "Done",
"commandLine": "set -e\npython3 $(which encode_task_filter.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/attempt-2/inputs/1588305472/JB0009_input_R1.bam \\\n --paired-end \\\n --multimapping 0 \\\n --dup-marker picard \\\n --mapq-thresh 30 \\\n --filter-chrs \\\n --chrsz /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/attempt-2/inputs/415522508/hg19.chrom.sizes \\\n \\\n --mito-chr-name chrM \\\n --nth 2 \\\n --picard-java-heap 18000M",
"shardIndex": 0,
"runtimeAttributes": {
"maxRetries": "1",
"failOnStderr": "false",
"continueOnReturnCode": "0"
},
"callCaching": {
"allowResultReuse": false,
"effectiveCallCachingMode": "WriteCache",
"hashes": {
"output count": "E4DA3B7FBBCE2345D7772B0674A318D5",
"runtime attribute": {
"docker": "N/A",
"failOnStderr": "68934A3E9455FA72420237EB05902327",
"continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA"
},
"output expression": {
"File samstat_qc": "EF7F94200A66D0F363422B1FF5348D90",
"File dup_qc": "D34C4EB2504C6E5EE8A8B6FF197A9EC7",
"File nodup_bai": "DC8EF4DE24731F8D0B23B82E2CA15141",
"File lib_complexity_qc": "84CE21EEA5C3132AC521760733899074",
"File nodup_bam": "4D95F2D23046D5295A149CA785FD216E"
},
"input count": "C20AD4D76FE97759AA27A0C99BFF6710",
"backend name": "509820290D57F333403F490DDE7316F4",
"command template": "AFCF4863CE65A953C90B0A43D3EB1D21",
"input": {
"Int mem_mb": "D9798CDF31C02D86B8B81CC119D94836",
"String disks": "085EEFBDBB0F02BAD180CD29A54CCBA2",
"String mito_chr_name": "4A8385B4D27A8EEDB503793999FE2D1B",
"Boolean paired_end": "B326B5062B2F0E69046810717534CB09",
"Int time_hr": "1FF1DE774005F8DA13F42943881C655F",
"File chrsz": "82e36e01fa26480931039ac178ad8e83",
"Int cpu": "C81E728D9D4C2F636F067F89CC14862C",
"Float picard_java_heap_factor": "A894124CC6D5C5C71AFE060D5DDE0762",
"String dup_marker": "620573C97469647DF116392F8B351FEB",
"Boolean no_dup_removal": "68934A3E9455FA72420237EB05902327",
"File bam": "0a1ea2408bb732cca23d2a1f728e189e",
"Int mapq_thresh": "34173CB38F07F89DDBEBC2AC9128303F"
}
}
},
"inputs": {
"no_dup_removal": false,
"mem_mb": 20000,
"time_hr": 24,
"mito_chr_name": "chrM",
"chrsz": "/PHShome/bk724/H3K27ac_sample/cf672570da74971be478218e3c209190/hg19.chrom.sizes",
"picard_java_heap": null,
"disks": "local-disk 400 HDD",
"picard_java_heap_factor": 0.9,
"mapq_thresh": 30,
"cpu": 2,
"paired_end": true,
"dup_marker": "picard",
"bam": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/execution/glob-3bcbe4e7489c90f75e0523ac6f3a9385/JB0009_input_R1.bam",
"filter_chrs": []
},
"returnCode": 1,
"failures": [
{
"causedBy": [],
"message": "Job chip.filter_ctl:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."
}
],
"jobId": "62614",
"backend": "Local",
"end": "2020-07-15T20:16:39.481Z",
"stderr": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/attempt-2/execution/stderr",
"callRoot": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/attempt-2",
"attempt": 2,
"executionEvents": [
{
"endTime": "2020-07-15T20:16:39.481Z",
"description": "UpdatingJobStore",
"startTime": "2020-07-15T20:16:39.481Z"
},
{
"endTime": "2020-07-15T19:55:15.377Z",
"startTime": "2020-07-15T19:55:15.374Z",
"description": "PreparingJob"
},
{
"startTime": "2020-07-15T19:55:14.705Z",
"description": "Pending",
"endTime": "2020-07-15T19:55:14.705Z"
},
{
"startTime": "2020-07-15T19:55:15.373Z",
"description": "WaitingForValueStore",
"endTime": "2020-07-15T19:55:15.374Z"
},
{
"startTime": "2020-07-15T19:55:15.377Z",
"endTime": "2020-07-15T20:16:39.481Z",
"description": "RunningJob"
},
{
"endTime": "2020-07-15T19:55:15.373Z",
"description": "RequestingExecutionToken",
"startTime": "2020-07-15T19:55:14.705Z"
}
],
"start": "2020-07-15T19:55:14.694Z"
}
],
"chip.align_ctl": [
{
"executionStatus": "Done",
"stdout": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/execution/stdout",
"backendStatus": "Done",
"commandLine": "set -e\n\n# check if pipeline dependencies can be found\nif [[ -z \"$(which encode_task_merge_fastq.py 2> /dev/null || true)\" ]]\nthen\n echo -e \"\\n* Error: pipeline dependencies not found.\" 1>&2\n echo 'Conda users: Did you activate Conda environment (conda activate encode-chip-seq-pipeline)?' 1>&2\n echo ' Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2\n echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2\n echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2\n echo -e \"\\n\" 1>&2\n exit 3\nfi\npython3 $(which encode_task_merge_fastq.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/execution/write_tsv_68bc35f89a8a6d89b141f7a5a52448f6.tmp \\\n --paired-end \\\n --nth 4\n\nif [ -z '' ]; then\n SUFFIX=\nelse\n SUFFIX=_trimmed\n python3 $(which encode_task_trim_fastq.py) \\\n R1/*.fastq.gz \\\n --trim-bp \\\n --out-dir R1$SUFFIX\n if [ 'true' == 'true' ]; then\n python3 $(which encode_task_trim_fastq.py) \\\n R2/*.fastq.gz \\\n --trim-bp \\\n --out-dir R2$SUFFIX\n fi\nfi\nif [ '0' == '0' ]; then\n SUFFIX=$SUFFIX\nelse\n NEW_SUFFIX=\"$SUFFIX\"_cropped\n python3 $(which encode_task_trimmomatic.py) \\\n --fastq1 R1$SUFFIX/*.fastq.gz \\\n --fastq2 R2$SUFFIX/*.fastq.gz \\\n --paired-end \\\n --crop-length 0 \\\n --crop-length-tol \"2\" \\\n --out-dir-R1 R1$NEW_SUFFIX \\\n --out-dir-R2 R2$NEW_SUFFIX \\\n --trimmomatic-java-heap 18000M \\\n --nth 4\n SUFFIX=$NEW_SUFFIX\nfi\n\nif [ 'bowtie2' == 'bwa' ]; then\n python3 $(which encode_task_bwa.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/inputs/-498531974/male.hg19.fa.tar \\\n R1$SUFFIX/*.fastq.gz \\\n R2$SUFFIX/*.fastq.gz \\\n --paired-end \\\n \\\n --nth 4\n\nelif [ 'bowtie2' == 'bowtie2' ]; then\n python3 $(which encode_task_bowtie2.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/inputs/-498531974/male.hg19.fa.tar \\\n R1$SUFFIX/*.fastq.gz \\\n R2$SUFFIX/*.fastq.gz \\\n \\\n --paired-end \\\n --nth 4\nfi \n\npython3 $(which encode_task_post_align.py) \\\n R1$SUFFIX/*.fastq.gz $(ls *.bam) \\\n --mito-chr-name chrM \\\n --nth 4\nrm -rf R1 R2 R1$SUFFIX R2$SUFFIX",
"shardIndex": 0,
"outputs": {
"bai": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/execution/glob-6efbc60cb1e0959bab4e467327a9416c/JB0009_input_R1.bam.bai",
"bam": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/execution/glob-3bcbe4e7489c90f75e0523ac6f3a9385/JB0009_input_R1.bam",
"read_len_log": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/execution/glob-773fb92850749a2b4a829cf3c8c4de27/JB0009_input_R1.read_length.txt",
"samstat_qc": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/execution/glob-7b38d9959cf6f3deb83ac2bd156d8317/JB0009_input_R1.samstats.qc"
},
"runtimeAttributes": {
"failOnStderr": "false",
"continueOnReturnCode": "0",
"maxRetries": "1"
},
"callCaching": {
"allowResultReuse": true,
"hit": false,
"result": "Cache Miss",
"hashes": {
"output count": "A87FF679A2F3E71D9181A67B7542122C",
"runtime attribute": {
"docker": "N/A",
"failOnStderr": "68934A3E9455FA72420237EB05902327",
"continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA"
},
"output expression": {
"File bam": "4D95F2D23046D5295A149CA785FD216E",
"File read_len_log": "CC7C87A75C5BFD29B68744DB9D1A3E61",
"File samstat_qc": "EF7F94200A66D0F363422B1FF5348D90",
"File bai": "DC8EF4DE24731F8D0B23B82E2CA15141"
},
"input count": "C74D97B01EAE257E44AA9D5BADE97BAF",
"backend name": "509820290D57F333403F490DDE7316F4",
"command template": "1A155C84DB5CCA39EB57AF59A522131C",
"input": {
"Int mem_mb": "D9798CDF31C02D86B8B81CC119D94836",
"File tmp_fastqs": [
[
"fbb48360361d418c85e68a96e2453fbf",
"018c94177d4530ff1a0bfebd2e4c6ba8"
]
],
"File fastqs_R1": [
"fbb48360361d418c85e68a96e2453fbf"
],
"String disks": "085EEFBDBB0F02BAD180CD29A54CCBA2",
"String mito_chr_name": "4A8385B4D27A8EEDB503793999FE2D1B",
"Int crop_length_tol": "C81E728D9D4C2F636F067F89CC14862C",
"Boolean paired_end": "B326B5062B2F0E69046810717534CB09",
"Int time_hr": "642E92EFB79421734881B53E1E1B18B6",
"Int crop_length": "CFCD208495D565EF66E7DFF9F98764DA",
"File fastqs_R2": [
"018c94177d4530ff1a0bfebd2e4c6ba8"
],
"Int cpu": "A87FF679A2F3E71D9181A67B7542122C",
"File idx_tar": "08b74cce1d9551f812d9f8a7ab4feddf",
"Boolean use_bwa_mem_for_pe": "68934A3E9455FA72420237EB05902327",
"String aligner": "99E6A6AC2CD2C1E66B6C4CA6FE0D3837",
"Float trimmomatic_java_heap_factor": "A894124CC6D5C5C71AFE060D5DDE0762"
}
},
"effectiveCallCachingMode": "ReadAndWriteCache"
},
"inputs": {
"trimmomatic_java_heap_factor": 0.9,
"trimmomatic_java_heap": null,
"use_bwa_mem_for_pe": false,
"fastqs_R2": [
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_input_R2.fastq.gz"
],
"multimapping": null,
"mem_mb": 20000,
"time_hr": 48,
"crop_length": 0,
"mito_chr_name": "chrM",
"tmp_fastqs": [
[
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_input_R1.fastq.gz",
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_input_R2.fastq.gz"
]
],
"disks": "local-disk 400 HDD",
"fastqs_R1": [
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_input_R1.fastq.gz"
],
"trim_bp": null,
"aligner": "bowtie2",
"cpu": 4,
"paired_end": true,
"crop_length_tol": 2,
"idx_tar": "/PHShome/bk724/H3K27ac_sample/e28e11c41785d0b78376bed93bb0795e/male.hg19.fa.tar"
},
"returnCode": 0,
"jobId": "26181",
"backend": "Local",
"end": "2020-07-15T19:32:07.312Z",
"stderr": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0/execution/stderr",
"callRoot": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-align_ctl/shard-0",
"attempt": 1,
"executionEvents": [
{
"description": "Pending",
"endTime": "2020-07-15T13:13:14.426Z",
"startTime": "2020-07-15T13:13:14.426Z"
},
{
"startTime": "2020-07-15T13:13:17.374Z",
"endTime": "2020-07-15T13:13:17.379Z",
"description": "PreparingJob"
},
{
"startTime": "2020-07-15T13:13:17.379Z",
"endTime": "2020-07-15T13:13:17.383Z",
"description": "CallCacheReading"
},
{
"startTime": "2020-07-15T19:32:07.311Z",
"endTime": "2020-07-15T19:32:07.312Z",
"description": "UpdatingJobStore"
},
{
"description": "WaitingForValueStore",
"startTime": "2020-07-15T13:13:17.373Z",
"endTime": "2020-07-15T13:13:17.374Z"
},
{
"endTime": "2020-07-15T13:13:17.373Z",
"description": "RequestingExecutionToken",
"startTime": "2020-07-15T13:13:14.426Z"
},
{
"description": "UpdatingCallCache",
"startTime": "2020-07-15T19:32:04.344Z",
"endTime": "2020-07-15T19:32:07.311Z"
},
{
"description": "RunningJob",
"endTime": "2020-07-15T19:32:04.344Z",
"startTime": "2020-07-15T13:13:17.383Z"
}
],
"start": "2020-07-15T13:13:14.426Z"
}
],
"chip.bam2ta_no_dedup_R1": [
{
"executionStatus": "Done",
"stdout": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-bam2ta_no_dedup_R1/shard-0/execution/stdout",
"backendStatus": "Done",
"commandLine": "set -e\npython3 $(which encode_task_bam2ta.py) \\\n /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-bam2ta_no_dedup_R1/shard-0/inputs/181616037/JB0009_H3K27ac_R1.trim_50bp.filt.bam \\\n --disable-tn5-shift \\\n \\\n --mito-chr-name chrM \\\n --subsample 0 \\\n --nth 2",
"shardIndex": 0,
"outputs": {
"ta": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-bam2ta_no_dedup_R1/shard-0/execution/glob-199637d3015dccbe277f621a18be9eb4/JB0009_H3K27ac_R1.trim_50bp.filt.tagAlign.gz"
},
"runtimeAttributes": {
"maxRetries": "1",
"failOnStderr": "false",
"continueOnReturnCode": "0"
},
"callCaching": {
"allowResultReuse": true,
"hit": false,
"result": "Cache Miss",
"hashes": {
"output count": "C4CA4238A0B923820DCC509A6F75849B",
"runtime attribute": {
"docker": "N/A",
"failOnStderr": "68934A3E9455FA72420237EB05902327",
"continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA"
},
"output expression": {
"File ta": "9EB184A935F0EDE297EE65CA4AE0260B"
},
"input count": "C9F0F895FB98AB9159F51FD0297E236D",
"backend name": "509820290D57F333403F490DDE7316F4",
"command template": "ECF7F9D39746F1654A344F75E6BF30FE",
"input": {
"Int mem_mb": "B7A782741F667201B54880C925FAEC4B",
"String disks": "43113E0452AA60E72696070B209DE28A",
"String mito_chr_name": "4A8385B4D27A8EEDB503793999FE2D1B",
"Boolean paired_end": "68934A3E9455FA72420237EB05902327",
"Int time_hr": "1679091C5A880FAF6FB5E6087EB1B2DC",
"Int subsample": "CFCD208495D565EF66E7DFF9F98764DA",
"Int cpu": "C81E728D9D4C2F636F067F89CC14862C",
"File bam": "a63eb6e5374596ea2d1549eba574ec29"
}
},
"effectiveCallCachingMode": "ReadAndWriteCache"
},
"inputs": {
"subsample": 0,
"mem_mb": 10000,
"time_hr": 6,
"mito_chr_name": "chrM",
"disks": "local-disk 100 HDD",
"cpu": 2,
"paired_end": false,
"bam": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_R1/shard-0/execution/glob-3bcbe4e7489c90f75e0523ac6f3a9385/JB0009_H3K27ac_R1.trim_50bp.filt.bam"
},
"returnCode": 0,
"jobId": "43766",
"backend": "Local",
"end": "2020-07-15T15:52:49.310Z",
"stderr": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-bam2ta_no_dedup_R1/shard-0/execution/stderr",
"callRoot": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-bam2ta_no_dedup_R1/shard-0",
"attempt": 1,
"executionEvents": [
{
"startTime": "2020-07-15T15:52:49.310Z",
"description": "UpdatingJobStore",
"endTime": "2020-07-15T15:52:49.310Z"
},
{
"startTime": "2020-07-15T15:50:45.374Z",
"endTime": "2020-07-15T15:50:45.378Z",
"description": "PreparingJob"
},
{
"description": "RequestingExecutionToken",
"startTime": "2020-07-15T15:50:44.817Z",
"endTime": "2020-07-15T15:50:45.374Z"
},
{
"startTime": "2020-07-15T15:50:45.381Z",
"description": "RunningJob",
"endTime": "2020-07-15T15:52:49.296Z"
},
{
"startTime": "2020-07-15T15:50:44.817Z",
"endTime": "2020-07-15T15:50:44.817Z",
"description": "Pending"
},
{
"startTime": "2020-07-15T15:52:49.296Z",
"description": "UpdatingCallCache",
"endTime": "2020-07-15T15:52:49.310Z"
},
{
"description": "CallCacheReading",
"endTime": "2020-07-15T15:50:45.381Z",
"startTime": "2020-07-15T15:50:45.378Z"
},
{
"description": "WaitingForValueStore",
"startTime": "2020-07-15T15:50:45.374Z",
"endTime": "2020-07-15T15:50:45.374Z"
}
],
"start": "2020-07-15T15:50:44.816Z"
}
],
"chip.read_genome_tsv": [
{
"executionStatus": "Done",
"stdout": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-read_genome_tsv/execution/stdout",
"backendStatus": "Done",
"commandLine": "echo \"$(basename /PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-read_genome_tsv/inputs/-1099509898/hg19_caper.local.tsv)\" > genome_name\n# create empty files for all entries\ntouch ref_fa bowtie2_idx_tar bwa_idx_tar chrsz gensz blacklist blacklist2\ntouch mito_chr_name\ntouch regex_bfilt_peak_chr_name\n\npython <<CODE\nimport os\nwith open('/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-read_genome_tsv/inputs/-1099509898/hg19_caper.local.tsv','r') as fp:\n for line in fp:\n arr = line.strip('\\n').split('\\t')\n if arr:\n key, val = arr\n with open(key,'w') as fp2:\n fp2.write(val)\nCODE",
"shardIndex": -1,
"outputs": {
"blacklist": "/PHShome/bk724/H3K27ac_sample/9ae4adc56c2648faae9bdb0d86eaf455/wgEncodeDacMapabilityConsensusExcludable.bed.gz",
"ref_fa": "/PHShome/bk724/H3K27ac_sample/d7ca4991175aa42056d90443022d42e2/male.hg19.fa.gz",
"bwa_idx_tar": "/PHShome/bk724/H3K27ac_sample/4f04151bdb011938b6bc836d794f8a73/male.hg19.fa.tar",
"mito_chr_name": "chrM",
"chrsz": "/PHShome/bk724/H3K27ac_sample/cf672570da74971be478218e3c209190/hg19.chrom.sizes",
"bowtie2_idx_tar": "/PHShome/bk724/H3K27ac_sample/e28e11c41785d0b78376bed93bb0795e/male.hg19.fa.tar",
"genome_name": "hg19",
"gensz": "hs",
"regex_bfilt_peak_chr_name": "chr[\\dXY]+",
"blacklist2": null
},
"runtimeAttributes": {
"maxRetries": "0",
"failOnStderr": "false",
"continueOnReturnCode": "0"
},
"callCaching": {
"allowResultReuse": true,
"hit": false,
"result": "Cache Miss",
"hashes": {
"output count": "D3D9446802A44259755D38E6D163E820",
"runtime attribute": {
"docker": "N/A",
"failOnStderr": "68934A3E9455FA72420237EB05902327",
"continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA"
},
"output expression": {
"Optional(String) chrsz": "D4F18CDF90FCD98B1D444A137124EEFF",
"Optional(String) gensz": "4E9C991DACE79D32DC95E1786A251C28",
"Optional(String) bowtie2_idx_tar": "7002F9AE911D4075E4A0FB7FC2FBB1B0",
"Optional(String) genome_name": "7EC158EA4FE08BC5C050FCE05BB702F5",
"Optional(String) bwa_idx_tar": "E4CE5B3CBB134A3B9BC84F7C86E3F2A0",
"Optional(String) mito_chr_name": "95ACE380B920F62E8B4189B7BFA8DE5A",
"Optional(String) ref_fa": "FF4031A4B1F506164DB071652273A922",
"Optional(String) regex_bfilt_peak_chr_name": "A0964AEF638DE5FC78473C761B5B4B81",
"Optional(String) blacklist": "844E654445100F2887674F102FB5369F",
"Optional(String) blacklist2": "8B8D9781F2CFB8BF355C4AC17C0CCB14"
},
"input count": "C4CA4238A0B923820DCC509A6F75849B",
"backend name": "509820290D57F333403F490DDE7316F4",
"command template": "3F5339DBA6D35AF110BDA78AEDB4C3B2",
"input": {
"File genome_tsv": "6beeff2307fa372d7b6e3bc8d48d8de9"
}
},
"effectiveCallCachingMode": "ReadAndWriteCache"
},
"inputs": {
"null_s": null,
"genome_tsv": "/PHShome/bk724/H3K27ac_sample/23af10fb66962b8784fbace80dd24dee/hg19_caper.local.tsv"
},
"returnCode": 0,
"jobId": "25887",
"backend": "Local",
"end": "2020-07-15T13:13:10.357Z",
"stderr": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-read_genome_tsv/execution/stderr",
"callRoot": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-read_genome_tsv",
"attempt": 1,
"executionEvents": [
{
"endTime": "2020-07-15T13:12:57.533Z",
"description": "CallCacheReading",
"startTime": "2020-07-15T13:12:57.448Z"
},
{
"startTime": "2020-07-15T13:12:57.389Z",
"description": "PreparingJob",
"endTime": "2020-07-15T13:12:57.448Z"
},
{
"startTime": "2020-07-15T13:12:57.069Z",
"description": "Pending",
"endTime": "2020-07-15T13:12:57.081Z"
},
{
"startTime": "2020-07-15T13:13:09.848Z",
"endTime": "2020-07-15T13:13:10.357Z",
"description": "UpdatingCallCache"
},
{
"endTime": "2020-07-15T13:13:10.358Z",
"startTime": "2020-07-15T13:13:10.357Z",
"description": "UpdatingJobStore"
},
{
"description": "RunningJob",
"startTime": "2020-07-15T13:12:57.533Z",
"endTime": "2020-07-15T13:13:09.848Z"
},
{
"description": "WaitingForValueStore",
"endTime": "2020-07-15T13:12:57.389Z",
"startTime": "2020-07-15T13:12:57.384Z"
},
{
"startTime": "2020-07-15T13:12:57.081Z",
"description": "RequestingExecutionToken",
"endTime": "2020-07-15T13:12:57.384Z"
}
],
"start": "2020-07-15T13:12:57.057Z"
}
]
},
"outputs": {},
"workflowRoot": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0",
"actualWorkflowLanguage": "WDL",
"id": "5bd41c7c-170e-426a-8ee6-b6eb9d5015b0",
"inputs": {
"blacklist": null,
"fastqs_rep6_R1": [],
"chip.align_ctl.trim_bp": null,
"chip.error_control_required.vals": null,
"ctl_fastqs_rep5_R1": [],
"chip.align_R1.multimapping": null,
"chip.align.multimapping": null,
"peaks_pr2": [],
"align_cpu": 4,
"fastqs_rep4_R1": [],
"use_bwa_mem_for_pe": false,
"fdr_thresh": 0.01,
"xcor_mem_mb": 16000,
"align_time_hr": 48,
"fastqs_rep2_R1": [],
"ctl_fastqs_rep2_R2": [],
"jsd_disks": "local-disk 200 HDD",
"ref_fa": null,
"chip.read_genome_tsv.null_s": null,
"chip.error_use_bwa_mem_for_non_bwa.vals": null,
"cap_num_peak": null,
"macs2_signal_track_mem_mb": 16000,
"gc_bias_picard_java_heap": null,
"filter_disks": "local-disk 400 HDD",
"bwa_idx_tar": null,
"ctl_subsample_reads": 0,
"jsd_time_hr": 6,
"fastqs_rep5_R2": [],
"ctl_fastqs_rep9_R2": [],
"fastqs_rep7_R2": [],
"description": "CLLepigenome_for_dbGAP by the Ott lab",
"ctl_fastqs_rep1_R1": [
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_input_R1.fastq.gz"
],
"chip.error_subsample_pooled_control_with_mixed_endedness.vals": null,
"bam2ta_mem_mb": 10000,
"pipeline_type": "tf",
"enable_gc_bias": true,
"fraglen": [],
"bam2ta_time_hr": 6,
"xcor_exclusion_range_min": -500,
"no_dup_removal": false,
"chip.pool_ta.col": null,
"ctl_fastqs_rep7_R2": [],
"filter_cpu": 2,
"chip.error_ctl_fastq_input_required_for_control_mode.vals": null,
"ctl_paired_end": null,
"call_peak_cpu": 2,
"fastqs_rep9_R2": [],
"fastqs_rep8_R2": [],
"crop_length": 0,
"tas": [],
"mito_chr_name": null,
"use_filt_pe_ta_for_xcor": false,
"pval_thresh": 0.01,
"chrsz": null,
"peak_ppr2": null,
"xcor_disks": "local-disk 100 HDD",
"fastqs_rep6_R2": [],
"bowtie2_idx_tar": null,
"true_rep_only": false,
"bam2ta_cpu": 2,
"jsd_mem_mb": 12000,
"chip.pool_blacklist.prefix": null,
"chip.align_R1.trimmomatic_java_heap": null,
"ctl_fastqs_rep1_R2": [
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_input_R2.fastq.gz"
],
"idr_thresh": 0.05,
"fastqs_rep3_R2": [],
"peaks_pr1": [],
"chip.error_wrong_aligner.vals": null,
"xcor_subsample_reads": 15000000,
"ctl_fastqs_rep6_R2": [],
"chip.pool_ta_pr1.col": null,
"ctl_fastqs_rep9_R1": [],
"jsd_cpu": 2,
"fastqs_rep5_R1": [],
"ctl_fastqs_rep10_R1": [],
"mapq_thresh": 30,
"genome_tsv": "/PHShome/bk724/H3K27ac_sample/23af10fb66962b8784fbace80dd24dee/hg19_caper.local.tsv",
"fastqs_rep4_R2": [],
"xcor_exclusion_range_max": null,
"chip.align.trim_bp": null,
"subsample_reads": 0,
"peak_ppr1": null,
"genome_name": null,
"fastqs_rep8_R1": [],
"enable_jsd": true,
"ctl_fastqs_rep8_R2": [],
"fastqs_rep10_R2": [],
"ctl_bams": [],
"chip.qc_report.qc_json_ref": null,
"filter_time_hr": 24,
"aligner": "bowtie2",
"xcor_time_hr": 24,
"fastqs_rep9_R1": [],
"ctl_nodup_bams": [],
"gensz": null,
"macs2_signal_track_time_hr": 24,
"ctl_fastqs_rep7_R1": [],
"exp_ctl_depth_ratio_limit": 5.0,
"ctl_tas": [],
"call_peak_mem_mb": 16000,
"fastqs_rep2_R2": [],
"fastqs_rep3_R1": [],
"fastqs_rep1_R1": [
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_H3K27ac_R1.fastq.gz"
],
"enable_count_signal_track": false,
"ctl_fastqs_rep3_R1": [],
"call_peak_time_hr": 72,
"filter_mem_mb": 20000,
"regex_bfilt_peak_chr_name": null,
"filter_picard_java_heap": null,
"chip.align_ctl.multimapping": null,
"align_trimmomatic_java_heap": null,
"nodup_bams": [],
"blacklist2": null,
"paired_end": true,
"always_use_pooled_ctl": true,
"ctl_depth_limit": 200000000,
"dup_marker": "picard",
"ctl_fastqs_rep6_R1": [],
"crop_length_tol": 2,
"peaks": [],
"title": "H3K27ac",
"chip.pool_ta_pr2.col": null,
"bam2ta_disks": "local-disk 100 HDD",
"chip.error_ctl_input_defined_in_control_mode.vals": null,
"peak_caller": null,
"ctl_fastqs_rep4_R1": [],
"align_disks": "local-disk 400 HDD",
"fastqs_rep10_R1": [],
"ctl_fastqs_rep8_R1": [],
"ctl_fastqs_rep10_R2": [],
"fastqs_rep1_R2": [
"/PHShome/bk724/chip-seq-pipeline2/example_input_json/data/JB0009_H3K27ac_R2.fastq.gz"
],
"ctl_fastqs_rep3_R2": [],
"xcor_trim_bp": 50,
"ctl_fastqs_rep2_R1": [],
"macs2_signal_track_disks": "local-disk 400 HDD",
"filter_chrs": [],
"peak_pooled": null,
"ctl_paired_ends": [],
"call_peak_disks": "local-disk 200 HDD",
"align_mem_mb": 20000,
"xcor_cpu": 2,
"ctl_depth_ratio": 1.2,
"chip.error_input_data.vals": null,
"spr_mem_mb": 16000,
"align_only": false,
"paired_ends": [],
"fastqs_rep7_R1": [],
"chip.pool_ta_ctl.col": null,
"ctl_fastqs_rep4_R2": [],
"ctl_fastqs_rep5_R2": [],
"bams": []
},
"labels": {
"caper-str-label": "H3K27ac_sample",
"caper-backend": "Local",
"caper-user": "bk724",
"cromwell-workflow-id": "cromwell-5bd41c7c-170e-426a-8ee6-b6eb9d5015b0"
},
"submission": "2020-07-15T13:12:49.709Z",
"status": "Failed",
"failures": [
{
"causedBy": [
{
"message": "Job chip.filter_ctl:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
"causedBy": []
}
],
"message": "Workflow failed"
}
],
"end": "2020-07-15T20:21:08.306Z",
"start": "2020-07-15T13:12:49.821Z"
}Please run caper debug metadata.json (do it on your metadata JSON file).
Barbarak17
commented
4 years ago Thanks, this is the output:
2020-07-20 13:05:38,853|caper.cromwell_rest_api|ERROR| Help: cannot connect to server. Check if server is dead or still spinning up.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1262, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1308, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1257, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1028, in _send_output
self.send(msg)
File "/usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 968, in send
self.connect()
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x114497910>: Failed to establish a new connection: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/workflows/v1/query?additionalQueryResultFields=labels (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x114497910>: Failed to establish a new connection: [Errno 61] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/caper/cromwell_rest_api.py", line 256, in __request_get
headers={'accept': 'application/json'})
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/workflows/v1/query?additionalQueryResultFields=labels (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x114497910>: Failed to establish a new connection: [Errno 61] Connection refused'))Did you run it on the directory where your metadata.json file resides?
Barbarak17
commented
4 years ago Yes.
leepc12
commented
4 years ago If caper debug doesn't work, can you manually look at stderr of failed tasks (calls in your metadata.json). stdout should also be useful to debug. For example,
{
"retryableFailure": false,
"executionStatus": "Failed",
"stdout": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/attempt-2/execution/stdout",
.................................
"stderr": "/PHShome/bk724/chip/5bd41c7c-170e-426a-8ee6-b6eb9d5015b0/call-filter_ctl/shard-0/attempt-2/execution/stderr",Thanks, I was able to locate an error on one file. It appears that numpy needs to reinstalled. I will try this and run the pipeline again. I am closing this issue.
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Describe the bug Not sure if the issue here was a runtime error. In the metadata.json says that reason is "transpose requires all collections have the same size." Have you seen a similar error before?
OS/Platform
Caper configuration file Paste contents of
~/.caper/default.conf.Input JSON file
Error log
$ caper debug [WORKFLOW_ID_OR_METADATA_JSON_FILE]