dimkonstanto commented 6 years ago

Greetings,

I am running the pipeline using an already aligned dataset:

(1) bds /home/dimitris/Downloads/atac_dnase_pipelines/atac.bds -species hg19 -nth 20 -bam1 bam/genecore/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.bam -se -enable_idr -out_dir bds_output

and i get this report:

== git info Latest git commit : a0ff6c2fd4537f6e5af5bf2682ad7436fa4b6314 (Fri Sep 22 02:05:31 2017) Reading parameters from section (default) in file(/home/dimitris/Downloads/atac_dnase_pipelines/default.env)...

== configuration file info Hostname : prometheus.fleming.local Configuration file : Environment file : /home/dimitris/Downloads/atac_dnase_pipelines/default.env

== parallelization info No parallel jobs : false Maximum # threads : 20

== cluster/system info Walltime (general) : 5h50m Max. memory (general) : 7G Force to use a system : local Process priority (niceness) : 0 Retiral for failed tasks : 0 Submit tasks to a cluster queue : Unlimited cluster mem./walltime : false Java temporary directory : ${TMPDIR}

Info: Environments module not found on your system (e.g. /etc/profile.d/modules.sh). Ignoring shell env. parameters like '-mod'.

== shell environment info Conda env. : bds_atac Conda env. for python3 : bds_atac_py3 Conda bin. directory :

Shell cmd. for init. : if [[ -f $(which conda) && $(conda env list | grep bds_atac | wc -l) != "0" ]]; then source activate bds_atac; sleep 5; fi; export PATH=/home/dimitris/Downloads/atac_dnase_pipelines/.:/home/dimitris/Downloads/atac_dnase_pipelines/modules:/home/dimitris/Downloads/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

Shell cmd. for init.(py3) : if [[ -f $(which conda) && $(conda env list | grep bds_atac_py3 | wc -l) != "0" ]]; then source activate bds_atac_py3; sleep 5; fi; export PATH=/home/dimitris/Downloads/atac_dnase_pipelines/.:/home/dimitris/Downloads/atac_dnase_pipelines/modules:/home/dimitris/Downloads/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

Shell cmd. for fin. : TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."; sleep 0

Cluster task min. len. : 60

Cluster task delay : 0

== output directory/title info Output dir. : /media/raid/users/dimitris/data/ATAC-seq/bds_output Title (prefix) : bds_output Reading parameters from section (default) in file(/home/dimitris/Downloads/atac_dnase_pipelines/default.env)... Reading parameters from section (hg19) in file(/media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/bds_atac_species.conf)...

== species settings Species : hg19 Species file : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/bds_atac_species.conf

Species name (WashU browser) : hg19 Ref. genome seq. fasta : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/male.hg19.fa Chr. sizes file : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/hg19.chrom.sizes Black list bed : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/wgEncodeDacMapabilityConsensusExcludable.bed.gz Ref. genome seq. dir. :

== ENCODE accession settings ENCODE experiment accession : ENCODE award RFA : ENCODE assay category : ENCODE assay title : ENCODE award : ENCODE lab : ENCODE assembly genome : ENCODE alias prefix : KLAB_PIPELINE ENCODE alias suffix :

== report settings URL root for output directory : Genome coord. for browser tracks :

== align multimapping settings

alignments reported for multimapping : 0

== align bowtie2 settings Bowtie2 index : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/bowtie2_index/male.hg19.fa Replacement --score-min for bowtie2 : Walltime (bowtie2) : 47h Max. memory (bowtie2) : 12G Extra param. (bowtie2) :

== adapter trimmer settings Maximum allowed error rate for cutadapt : 0.10 Minimum trim. length for cutadapt -m : 5 Walltime (adapter trimming) : 23h Max. memory (adapter trimming) : 12G

== postalign bam settings MAPQ reads rm thresh. : 30 Rm. tag reads with str. : No dupe removal in filtering raw bam : false Walltime (bam filter) : 23h Max. memory (bam filter) : 12G Dup marker : picard Use sambamba markdup (instead of picard) : false

== postalign bed/tagalign settings Max. memory for UNIX shuf : 12G

== postalign cross-corr. analysis settings Max. memory for UNIX shuf : 12G User-defined cross-corr. peak strandshift : -1 Extra parameters for cross-corr. analysis :

== callpeak macs2 settings Genome size (hs,mm) : hs Walltime (macs2) : 23h Max. memory (macs2) : 15G Extra parameters for macs2 callpeak :

== callpeak naiver overlap settings Bedtools intersect -nonamecheck : false

== callpeak etc settings

of top peaks to pick up in peak files : 500000

== IDR settings Append IDR threshold to IDR out_dir : false

== ATAQC settings TSS enrichment bed : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/ataqc/hg19_gencode_tss_unique.bed.gz DNase bed for ataqc : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/ataqc/reg2map_honeybadger2_dnase_all_p10_ucsc.bed.gz Promoter bed for ataqc : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/ataqc/reg2map_honeybadger2_dnase_prom_p2.bed.gz Enhancer bed for ataqc : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/ataqc/reg2map_honeybadger2_dnase_enh_p2.bed.gz Reg2map for ataqc : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/ataqc/dnase_avgs_reg2map_p10_merged_named.pvals.gz Reg2map_bed for ataqc : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/ataqc/reg2map_honeybadger2_dnase_all_p10_ucsc.bed.gz Roadmap metadata for ataqc : /media/raid/users/dimitris/Annotations/genome_for_encode_atacseq_pipeline/hg19/ataqc/eid_to_mnemonic.txt Max. memory for ATAQC : 20G Walltime for ATAQC : 47h

== atac pipeline settings Type of pipeline : atac-seq Align only : false

reads to subsample replicates (0 if no subsampling) : 0

reads to subsample for cross-corr. analysis : 25000000

No pseudo replicates : false No ATAQC (advanced QC report) : false No Cross-corr. analysis : false Use CSEM for alignment : false Smoothing window for MACS2 : 150 DNase Seq : false IDR threshold : 0.1 Force to use ENCODE3 parameter set : false Force to use ENCODE parameter set : false Disable genome browser tracks : false p-val thresh. for overlapped peaks : 0.01 MACS2 p-val thresh. for peaks : 0.01 MACS2 p-val thresh. for BIGWIGs : 0.01 Enable IDR on called peaks : true Automatically find/trim adapters : false

== checking atac parameters ...

Checking parameters and data files for ATAQC.

Warning: ATAQC is available for fastq inputs only. Disabling ATAQC...

== checking adapters to be trimmed ...

== checking input files ...

Rep1 bam (SE) : /media/raid/users/dimitris/data/ATAC-seq/bam/genecore/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.bam Distributing 20 to ... {1=20} Task failed: Program & line : '/home/dimitris/Downloads/atac_dnase_pipelines/modules/postalign_xcor.bds', line 121 Task Name : 'xcor rep1' Task ID : 'atac.bds.20171002_105658_650_parallel_44/task.postalign_xcor.xcor_rep1.line_121.id_10' Task PID : '26883' Task hint : 'if [[ $(which run_spp_nodups.R 2> /dev/null | wc -l || echo) == "1" ]]; then RUN_SPP=$(which run_spp_nodups.R);; else RUN_SPP=$(which run_spp.R);; fi; Rscript ${RUN_SPP} -rf; -c=/media/raid/users/dimitris/data/ATAC-seq/bds_output/align/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_seq' Task resources : 'cpus: 20 mem: -1.0 B wall-timeout: 8640000' State : 'ERROR' Dependency state : 'ERROR' Retries available : '1' Input files : '[/media/raid/users/dimitris/data/ATAC-seq/bds_output/align/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.nodup.25M.tagAlign.gz]' Output files : '[/media/raid/users/dimitris/data/ATAC-seq/bds_output/qc/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.nodup.25M.cc.qc, /media/raid/users/dimitris/data/ATAC-seq/bds_output/qc/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.nodup.25M.cc.plot.pdf]' Script file : '/media/raid/users/dimitris/data/ATAC-seq/atac.bds.20171002_105658_650_parallel_44/task.postalign_xcor.xcor_rep1.line_121.id_10.sh' Exit status : '1' Program :

    # SYS command. line 123

     if [[ -f $(which conda) && $(conda env list | grep bds_atac | wc -l) != "0" ]]; then source activate bds_atac; sleep 5; fi;  export PATH=/home/dimitris/Downloads/atac_dnase_pipelines/.:/home/dimitris/Downloads/atac_dnase_pipelines/modules:/home/dimitris/Downloads/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

    # SYS command. line 126

     if [[ $(which run_spp_nodups.R 2> /dev/null | wc -l || echo) == "1" ]]; then RUN_SPP=$(which run_spp_nodups.R); \
                else RUN_SPP=$(which run_spp.R); \
                fi

    # SYS command. line 132

     Rscript ${RUN_SPP} -rf \
                -c=/media/raid/users/dimitris/data/ATAC-seq/bds_output/align/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.nodup.25M.tagAlign.gz -p=20 \
                -filtchr=chrM -savp=/media/raid/users/dimitris/data/ATAC-seq/bds_output/qc/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.nodup.25M.cc.plot.pdf -out=/media/raid/users/dimitris/data/ATAC-seq/bds_output/qc/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.nodup.25M.cc.qc -speak=0 

    # SYS command. line 135

     sed -r 's/,[^\t]+//g' /media/raid/users/dimitris/data/ATAC-seq/bds_output/qc/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.nodup.25M.cc.qc > /media/raid/users/dimitris/data/ATAC-seq/bds_output/qc/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.nodup.25M.cc.qc.tmp

    # SYS command. line 136

     mv /media/raid/users/dimitris/data/ATAC-seq/bds_output/qc/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.nodup.25M.cc.qc.tmp /media/raid/users/dimitris/data/ATAC-seq/bds_output/qc/rep1/C88MFACXX_ATAC_TAS_15s014547-1-2_Fousteri_lane3ATAC_sequence.nodup.25M.cc.qc

    # SYS command. line 138

     TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."; sleep 0

Fatal error: /home/dimitris/Downloads/atac_dnase_pipelines/atac.bds, line 671, pos 4. Task/s failed.

Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done. Fatal error: /home/dimitris/Downloads/atac_dnase_pipelines/atac.bds, line 417, pos 2. Task/s failed.

Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done.

(2) I also run the pipeline with xcor disabled (-no_xcor), but then i get another error: