i think it failed at run_ataqc.py i have the bigwig and narrowpeak files output already
i put the log here if somebody could take a look. thank you very much
the error is at the very end
INFO 2018-04-05 18:46:43 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:46:45 MarkDuplicates Read 39,000,000 records. Elapsed time: 00:02:07s. Time for last 1,000,000: 1s. Last read position: chrM:2,432
INFO 2018-04-05 18:46:45 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:46:48 MarkDuplicates Read 40,000,000 records. Elapsed time: 00:02:10s. Time for last 1,000,000: 2s. Last read position: chrM:3,492
INFO 2018-04-05 18:46:48 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:46:49 MarkDuplicates Read 41,000,000 records. Elapsed time: 00:02:12s. Time for last 1,000,000: 1s. Last read position: chrM:5,289
INFO 2018-04-05 18:46:49 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:46:52 MarkDuplicates Read 42,000,000 records. Elapsed time: 00:02:14s. Time for last 1,000,000: 2s. Last read position: chrM:7,201
INFO 2018-04-05 18:46:52 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:46:54 MarkDuplicates Read 43,000,000 records. Elapsed time: 00:02:16s. Time for last 1,000,000: 1s. Last read position: chrM:9,238
INFO 2018-04-05 18:46:54 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:46:56 MarkDuplicates Read 44,000,000 records. Elapsed time: 00:02:19s. Time for last 1,000,000: 2s. Last read position: chrM:10,506
INFO 2018-04-05 18:46:56 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
00:43:45.811 ExecutionerLocal 'Local[39]': Tasks [Local[39]] Pending: 0 Running: 1 Done: 5 Failed: 0
INFO 2018-04-05 18:47:31 MarkDuplicates Read 45,000,000 records. Elapsed time: 00:02:53s. Time for last 1,000,000: 34s. Last read position: chrM:11,537
INFO 2018-04-05 18:47:31 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:47:33 MarkDuplicates Read 46,000,000 records. Elapsed time: 00:02:55s. Time for last 1,000,000: 2s. Last read position: chrM:12,617
INFO 2018-04-05 18:47:33 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:47:35 MarkDuplicates Read 47,000,000 records. Elapsed time: 00:02:57s. Time for last 1,000,000: 1s. Last read position: chrM:13,647
INFO 2018-04-05 18:47:35 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:47:38 MarkDuplicates Read 48,000,000 records. Elapsed time: 00:03:00s. Time for last 1,000,000: 2s. Last read position: chrM:14,824
INFO 2018-04-05 18:47:38 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:47:39 MarkDuplicates Read 49,000,000 records. Elapsed time: 00:03:02s. Time for last 1,000,000: 1s. Last read position: chrM:15,748
INFO 2018-04-05 18:47:39 MarkDuplicates Tracking 0 as yet unmatched pairs. 0 records in RAM.
INFO 2018-04-05 18:47:43 MarkDuplicates Read 49808141 records. 0 pairs never matched.
INFO 2018-04-05 18:47:47 MarkDuplicates After buildSortedReadEndLists freeMemory: 5083086640; totalMemory: 5334106112; maxMemory: 11453595648
INFO 2018-04-05 18:47:47 MarkDuplicates Will retain up to 357924864 duplicate indices before spilling to disk.
INFO 2018-04-05 18:47:48 MarkDuplicates Traversing read pair information and detecting duplicates.
INFO 2018-04-05 18:47:48 MarkDuplicates Traversing fragment information and detecting duplicates.
00:44:46.826 ExecutionerLocal 'Local[39]': Tasks [Local[39]] Pending: 0 Running: 1 Done: 5 Failed: 0
INFO 2018-04-05 18:48:26 MarkDuplicates Sorting list of duplicate records.
INFO 2018-04-05 18:48:28 MarkDuplicates After generateDuplicateIndexes freeMemory: 5713939664; totalMemory: 8651800576; maxMemory: 11453595648
INFO 2018-04-05 18:48:28 MarkDuplicates Marking 19636737 records as duplicates.
INFO 2018-04-05 18:48:28 MarkDuplicates Found 0 optical duplicate clusters.
00:45:47.842 ExecutionerLocal 'Local[39]': Tasks [Local[39]] Pending: 0 Running: 1 Done: 5 Failed: 0
INFO 2018-04-05 18:49:56 MarkDuplicates Written 10,000,000 records. Elapsed time: 00:01:27s. Time for last 10,000,000: 87s. Last read position: chr4:102,436,910
00:46:48.857 ExecutionerLocal 'Local[39]': Tasks [Local[39]] Pending: 0 Running: 1 Done: 5 Failed: 0
INFO 2018-04-05 18:51:24 MarkDuplicates Written 20,000,000 records. Elapsed time: 00:02:55s. Time for last 10,000,000: 88s. Last read position: chr9:110,499,498
00:48:50.887 ExecutionerLocal 'Local[39]': Tasks [Local[39]] Pending: 0 Running: 1 Done: 5 Failed: 0
INFO 2018-04-05 18:52:52 MarkDuplicates Written 30,000,000 records. Elapsed time: 00:04:24s. Time for last 10,000,000: 88s. Last read position: chr16:55,670,809
00:49:51.903 ExecutionerLocal 'Local[39]': Tasks [Local[39]] Pending: 0 Running: 1 Done: 5 Failed: 0
INFO 2018-04-05 18:54:12 MarkDuplicates Written 40,000,000 records. Elapsed time: 00:05:43s. Time for last 10,000,000: 79s. Last read position: chrM:3,492
00:50:52.918 ExecutionerLocal 'Local[39]': Tasks [Local[39]] Pending: 0 Running: 1 Done: 5 Failed: 0
macs2 callpeak; -t /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.tagAlign.gz -f BED -n "/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.pf"
INFO @ Thu, 05 Apr 2018 19:21:58: 27000000
INFO @ Thu, 05 Apr 2018 19:21:59: 28000000
INFO @ Thu, 05 Apr 2018 19:22:00: 29000000
INFO @ Thu, 05 Apr 2018 19:22:02: 30000000
INFO @ Thu, 05 Apr 2018 19:22:02: #1 tag size is determined as 43 bps
INFO @ Thu, 05 Apr 2018 19:22:02: #1 tag size = 43
INFO @ Thu, 05 Apr 2018 19:22:02: #1 total tags in treatment: 30148394
INFO @ Thu, 05 Apr 2018 19:22:02: #1 finished!
INFO @ Thu, 05 Apr 2018 19:22:02: #2 Build Peak Model...
INFO @ Thu, 05 Apr 2018 19:22:02: #2 Skipped...
INFO @ Thu, 05 Apr 2018 19:22:02: #2 Sequencing ends will be shifted towards 5' by 75 bp(s)
INFO @ Thu, 05 Apr 2018 19:22:02: #2 Use 150 as fragment length
INFO @ Thu, 05 Apr 2018 19:22:02: #3 Call peaks...
INFO @ Thu, 05 Apr 2018 19:22:02: #3 Going to call summits inside each peak ...
INFO @ Thu, 05 Apr 2018 19:22:02: #3 Call peaks with given -log10pvalue cutoff: 2.00000 ...
INFO @ Thu, 05 Apr 2018 19:22:02: #3 Pre-compute pvalue-qvalue table...
INFO @ Thu, 05 Apr 2018 19:22:13: #3 In the peak calling step, the following will be performed simultaneously:
INFO @ Thu, 05 Apr 2018 19:22:13: #3 Write bedGraph files for treatment pileup (after scaling if necessary)... /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.pf_treat_pileup.bdg
INFO @ Thu, 05 Apr 2018 19:22:13: #3 Write bedGraph files for control lambda (after scaling if necessary)... /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.pf_control_lambda.bdg
INFO @ Thu, 05 Apr 2018 19:22:13: #3 --SPMR is requested, so pileup will be normalized by sequencing depth in million reads.
INFO @ Thu, 05 Apr 2018 19:22:13: #3 Call peaks for each chromosome...
INFO @ Thu, 05 Apr 2018 19:22:14: #3 In the peak calling step, the following will be performed simultaneously:
INFO @ Thu, 05 Apr 2018 19:22:14: #3 Write bedGraph files for treatment pileup (after scaling if necessary)... /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.pf_treat_pileup.bdg
INFO @ Thu, 05 Apr 2018 19:22:14: #3 Write bedGraph files for control lambda (after scaling if necessary)... /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.pf_control_lambda.bdg
INFO @ Thu, 05 Apr 2018 19:22:14: #3 --SPMR is requested, so pileup will be normalized by sequencing depth in million reads.
INFO @ Thu, 05 Apr 2018 19:22:14: #3 Call peaks for each chromosome...
01:19:35.371 ExecutionerLocal 'Local[39]': Tasks [Local[39]] Pending: 0 Running: 3 Done: 12 Failed: 0
PID
Task state
Task name
Dependencies
Task definition
130264
running (RUNNING)
macs2 n/s rep1-pr2
macs2 callpeak; -t /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.tagAlign.gz -f BED -n "/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.pf"
macs2 callpeak; -t /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.tagAlign.gz -f BED -n "/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.pf"
INFO @ Thu, 05 Apr 2018 19:23:01: #3 In the peak calling step, the following will be performed simultaneously:
INFO @ Thu, 05 Apr 2018 19:23:01: #3 Write bedGraph files for treatment pileup (after scaling if necessary)... /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.pf_treat_pileup.bdg
INFO @ Thu, 05 Apr 2018 19:23:01: #3 Write bedGraph files for control lambda (after scaling if necessary)... /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.pf_control_lambda.bdg
INFO @ Thu, 05 Apr 2018 19:23:01: #3 --SPMR is requested, so pileup will be normalized by sequencing depth in million reads.
INFO @ Thu, 05 Apr 2018 19:23:01: #3 Call peaks for each chromosome...
01:20:36.385 ExecutionerLocal 'Local[39]': Tasks [Local[39]] Pending: 0 Running: 3 Done: 12 Failed: 0
PID
Task state
Task name
Dependencies
Task definition
130264
running (RUNNING)
macs2 n/s rep1-pr2
macs2 callpeak; -t /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.tagAlign.gz -f BED -n "/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.pf"
set +o pipefail; zcat /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.pf.narrowPeak.gz
set +o pipefail; zcat /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.pf.narrowPeak.gz
Task has finished (2 seconds).
Task has finished (1 seconds).
Task has finished (2 seconds).
Task has finished (6 seconds).
== Done naive_overlap()
species_ataqc: hg19
/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
Traceback (most recent call last):
File "/mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/ataqc/run_ataqc.py", line 14, in
import metaseq
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/metaseq/init.py", line 7, in
import plotutils
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/metaseq/plotutils.py", line 6, in
from matplotlib import pyplot as plt
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/pyplot.py", line 36, in
from matplotlib.figure import Figure, figaspect
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/figure.py", line 40, in
from matplotlib.axes import Axes, SubplotBase, subplot_class_factory
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/axes/init.py", line 4, in
from ._subplots import *
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/axes/_subplots.py", line 10, in
from matplotlib.axes._axes import Axes
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/axes/axes.py", line 22, in
import matplotlib.dates as # <-registers a date unit converter
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/dates.py", line 126, in
from dateutil.rrule import (rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY,
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/dateutil/rrule.py", line 19, in
from six.moves import _thread, range
ImportError: cannot import name _thread
Task failed:
Program & line : '/mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/modules/ataqc.bds', line 109
Task Name : 'ataqc rep1'
Task ID : 'atac.bds.20180405_180322_656_parallel_88/task.ataqc.ataqc_rep1.line_109.id_41'
Task PID : '131434'
Task hint : 'if [ "${TMPDIR}" != "" ] && [ -d "${TMPDIR}" ]; then; fi; cd /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1; /mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/ataqc/run_ataqc.py; --workdir /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1; --outdir /mnt/'
Task resources : 'cpus: -1 mem: -1.0 B wall-timeout: 8640000'
State : 'ERROR'
Dependency state : 'ERROR'
Retries available : '1'
Input files : '[/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.fastq.gz, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.bam, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.align.log, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.nodup.pbc.qc, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.dup.qc, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.nodup.bam, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.tagAlign.gz, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/signal/macs2/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.pf.pval.signal.bigwig, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/overlap/AH716_S6_L008_R1_001.trim.nodup.tn5.pf.500K.pval0.01.naive_overlap.filt.narrowPeak.gz]'
Output files : '[/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim_qc.html, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim_qc.txt]'
Script file : '/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac.bds.20180405_180322_656_parallel_88/task.ataqc.ataqc_rep1.line_109.id_41.sh'
Exit status : '1'
Program :
# SYS command. line 111
if [[ -f $(which conda) && $(conda env list | grep bds_atac | wc -l) != "0" ]]; then source activate bds_atac; sleep 5; fi; export PATH=/mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/.:/mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/modules:/mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)
# SYS command. line 114
export _JAVA_OPTIONS="-Xms256M -Xmx20G -XX:ParallelGCThreads=1"
# SYS command. line 116
if [ "${TMPDIR}" != "" ] && [ -d "${TMPDIR}" ]; then \
export _JAVA_OPTIONS="${_JAVA_OPTIONS} -Djava.io.tmpdir=${TMPDIR}"; \
fi
# SYS command. line 119
cd /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1
# SYS command. line 122
if [ -f "$(which picard)" ]; then export PICARDROOT="$(dirname $(which picard))/../share/picard"*; fi
# SYS command. line 124
/mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/ataqc/run_ataqc.py \
--workdir /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1 \
--outdir /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1 \
--outprefix AH716_S6_L008_R1_001.trim \
--genome hg19 \
--chromsizes /mnt/NFS/homeGene/HarrisLab/sxx128/atac/bds_pipeline_genome_data/hg19/hg19/hg19.chrom.sizes \
--ref /mnt/NFS/homeGene/HarrisLab/sxx128/atac/bds_pipeline_genome_data/hg19/hg19/male.hg19.fa \
--tss /mnt/NFS/homeGene/HarrisLab/sxx128/atac/bds_pipeline_genome_data/hg19/hg19/ataqc/hg19_gencode_tss_unique.bed.gz \
--dnase /mnt/NFS/homeGene/HarrisLab/sxx128/atac/bds_pipeline_genome_data/hg19/hg19/ataqc/reg2map_honeybadger2_dnase_all_p10_ucsc.bed.gz \
--blacklist /mnt/NFS/homeGene/HarrisLab/sxx128/atac/bds_pipeline_genome_data/hg19/hg19/wgEncodeDacMapabilityConsensusExcludable.bed.gz \
--prom /mnt/NFS/homeGene/HarrisLab/sxx128/atac/bds_pipeline_genome_data/hg19/hg19/ataqc/reg2map_honeybadger2_dnase_prom_p2.bed.gz \
--enh /mnt/NFS/homeGene/HarrisLab/sxx128/atac/bds_pipeline_genome_data/hg19/hg19/ataqc/reg2map_honeybadger2_dnase_enh_p2.bed.gz \
--pbc /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.nodup.pbc.qc\
--fastq1 /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.fastq.gz \
--alignedbam /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.bam \
--alignmentlog /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.align.log \
--coordsortbam /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.bam \
--duplog /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.dup.qc \
--finalbam /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.nodup.bam \
--finalbed /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.tagAlign.gz \
--bigwig /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/signal/macs2/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.pf.pval.signal.bigwig \
--peaks /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/overlap/AH716_S6_L008_R1_001.trim.nodup.tn5.pf.500K.pval0.01.naive_overlap.filt.narrowPeak.gz --naive_overlap_peaks /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/overlap/AH716_S6_L008_R1_001.trim.nodup.tn5.pf.500K.pval0.01.naive_overlap.filt.narrowPeak.gz \
--reg2map /mnt/NFS/homeGene/HarrisLab/sxx128/atac/bds_pipeline_genome_data/hg19/hg19/ataqc/dnase_avgs_reg2map_p10_merged_named.pvals.gz --reg2map_bed /mnt/NFS/homeGene/HarrisLab/sxx128/atac/bds_pipeline_genome_data/hg19/hg19/ataqc/reg2map_honeybadger2_dnase_all_p10_ucsc.bed.gz --meta /mnt/NFS/homeGene/HarrisLab/sxx128/atac/bds_pipeline_genome_data/hg19/hg19/ataqc/eid_to_mnemonic.txt
# SYS command. line 147
rm -f test.log test.png
# SYS command. line 149
TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."; sleep 0
StdErr (100000000 lines) :
/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
Traceback (most recent call last):
File "/mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/ataqc/run_ataqc.py", line 14, in <module>
import metaseq
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/metaseq/__init__.py", line 7, in <module>
import plotutils
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/metaseq/plotutils.py", line 6, in <module>
from matplotlib import pyplot as plt
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/pyplot.py", line 36, in <module>
from matplotlib.figure import Figure, figaspect
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/figure.py", line 40, in <module>
from matplotlib.axes import Axes, SubplotBase, subplot_class_factory
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/axes/__init__.py", line 4, in <module>
from ._subplots import *
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/axes/_subplots.py", line 10, in <module>
from matplotlib.axes._axes import Axes
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/axes/_axes.py", line 22, in <module>
import matplotlib.dates as _ # <-registers a date unit converter
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/dates.py", line 126, in <module>
from dateutil.rrule import (rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY,
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/dateutil/rrule.py", line 19, in <module>
from six.moves import _thread, range
ImportError: cannot import name _thread
Fatal error: /mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/atac.bds, line 1665, pos 2. Task/s failed.
atac.bds, line 78 : main()
atac.bds, line 81 : void main() { // atac pipeline starts here
atac.bds, line 101 : ataqc()
atac.bds, line 1654 : void ataqc() {
atac.bds, line 1665 : wait
Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done.
leepc12
commented
6 years ago
i think it failed at run_ataqc.py i have the bigwig and narrowpeak files output already i put the log here if somebody could take a look. thank you very much the error is at the very end
this is my command line
bds /mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/atac.bds -species hg19 -se -nth 14 -fastq /mnt/NFS/projects/SOM_GEN_AXH575/raw/atac_seq_2017sep/AH716_S6_L008_R1_001.fastq.gz -auto_detect_adapter -out_dir atac_716_apr5
Task has finished (53 seconds). ################ ChIP data: /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.nodup.25M.tagAlign.gz Control data: NA strandshift(min): -500 strandshift(step): 5 strandshift(max) 1500 user-defined peak shift 0 exclusion(min): 10 exclusion(max): NaN num parallel nodes: 16 FDR threshold: 0.01 NumPeaks Threshold: NA Output Directory: /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1 narrowPeak output file name: NA regionPeak output file name: NA Rdata filename: NA plot pdf filename: /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.nodup.25M.cc.plot.pdf result filename: /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.nodup.25M.cc.qc Overwrite files?: TRUE
Minimum cross-correlation value 0.2597509 Minimum cross-correlation shift 1500 Top 3 cross-correlation values 0.285307089235863 Top 3 estimates for fragment length 0 Window half size 755 Phantom peak location 50 Phantom peak Correlation 0.2841252 Normalized Strand cross-correlation coefficient (NSC) 1.098387 Relative Strand cross-correlation Coefficient (RSC) 1.048491 Phantom Peak Quality Tag 1 null device 1 Task has finished (476 seconds).
== Done do_align() INFO @ Thu, 05 Apr 2018 19:21:21:
Command line: callpeak -t /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.tagAlign.gz -f BED -n /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.pf -g hs -p 0.01 --nomodel --shift -75 --extsize 150 -B --SPMR --keep-dup all --call-summits
ARGUMENTS LIST:
name = /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.pf
format = BED
ChIP-seq file = ['/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/pseudo_reps/rep1/pr2/AH716_S6_L008_R1_001.trim.nodup.tn5.pr2.tagAlign.gz']
control file = None
effective genome size = 2.70e+09
band width = 300
model fold = [5, 50]
pvalue cutoff = 1.00e-02
qvalue will not be calculated and reported as -1 in the final output.
Larger dataset will be scaled towards smaller dataset.
Range for calculating regional lambda is: 10000 bps
Broad region calling is off
Searching for subpeak summits is on
MACS will save fragment pileup signal per million reads
INFO @ Thu, 05 Apr 2018 19:21:21: #1 read tag files... INFO @ Thu, 05 Apr 2018 19:21:21: #1 read treatment tags... INFO @ Thu, 05 Apr 2018 19:21:21:
Command line: callpeak -t /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.tagAlign.gz -f BED -n /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.pf -g hs -p 0.01 --nomodel --shift -75 --extsize 150 -B --SPMR --keep-dup all --call-summits
ARGUMENTS LIST:
name = /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.pf
format = BED
ChIP-seq file = ['/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/pseudo_reps/rep1/pr1/AH716_S6_L008_R1_001.trim.nodup.tn5.pr1.tagAlign.gz']
control file = None
effective genome size = 2.70e+09
band width = 300
model fold = [5, 50]
pvalue cutoff = 1.00e-02
qvalue will not be calculated and reported as -1 in the final output.
Larger dataset will be scaled towards smaller dataset.
Range for calculating regional lambda is: 10000 bps
Broad region calling is off
Searching for subpeak summits is on
MACS will save fragment pileup signal per million reads
INFO @ Thu, 05 Apr 2018 19:21:21: #1 read tag files... INFO @ Thu, 05 Apr 2018 19:21:21: #1 read treatment tags... INFO @ Thu, 05 Apr 2018 19:21:21:
Command line: callpeak -t /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.tagAlign.gz -f BED -n /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.pf -g hs -p 0.01 --nomodel --shift -75 --extsize 150 -B --SPMR --keep-dup all --call-summits
ARGUMENTS LIST:
name = /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.pf
format = BED
ChIP-seq file = ['/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.tagAlign.gz']
control file = None
effective genome size = 2.70e+09
band width = 300
model fold = [5, 50]
pvalue cutoff = 1.00e-02
qvalue will not be calculated and reported as -1 in the final output.
Larger dataset will be scaled towards smaller dataset.
Range for calculating regional lambda is: 10000 bps
Broad region calling is off
Searching for subpeak summits is on
MACS will save fragment pileup signal per million reads
Task has finished (2358 seconds).
Task has finished (2 seconds). Task has finished (1 seconds). Task has finished (2 seconds). Task has finished (6 seconds).
== Done naive_overlap() species_ataqc: hg19 /mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment. warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.') Traceback (most recent call last): File "/mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/ataqc/run_ataqc.py", line 14, in
import metaseq
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/metaseq/init.py", line 7, in
import plotutils
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/metaseq/plotutils.py", line 6, in
from matplotlib import pyplot as plt
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/pyplot.py", line 36, in
from matplotlib.figure import Figure, figaspect
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/figure.py", line 40, in
from matplotlib.axes import Axes, SubplotBase, subplot_class_factory
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/axes/init.py", line 4, in
from ._subplots import *
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/axes/_subplots.py", line 10, in
from matplotlib.axes._axes import Axes
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/axes/axes.py", line 22, in
import matplotlib.dates as # <-registers a date unit converter
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/matplotlib/dates.py", line 126, in
from dateutil.rrule import (rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY,
File "/mnt/NFS/homeGene/HarrisLab/sxx128/anaconda3/envs/bds_atac/lib/python2.7/site-packages/dateutil/rrule.py", line 19, in
from six.moves import _thread, range
ImportError: cannot import name _thread
Task failed:
Program & line : '/mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/modules/ataqc.bds', line 109
Task Name : 'ataqc rep1'
Task ID : 'atac.bds.20180405_180322_656_parallel_88/task.ataqc.ataqc_rep1.line_109.id_41'
Task PID : '131434'
Task hint : 'if [ "${TMPDIR}" != "" ] && [ -d "${TMPDIR}" ]; then; fi; cd /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1; /mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/ataqc/run_ataqc.py; --workdir /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1; --outdir /mnt/'
Task resources : 'cpus: -1 mem: -1.0 B wall-timeout: 8640000'
State : 'ERROR'
Dependency state : 'ERROR'
Retries available : '1'
Input files : '[/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.fastq.gz, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.bam, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.align.log, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.nodup.pbc.qc, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim.dup.qc, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.nodup.bam, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/align/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.tagAlign.gz, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/signal/macs2/rep1/AH716_S6_L008_R1_001.trim.nodup.tn5.pf.pval.signal.bigwig, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/peak/macs2/overlap/AH716_S6_L008_R1_001.trim.nodup.tn5.pf.500K.pval0.01.naive_overlap.filt.narrowPeak.gz]'
Output files : '[/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim_qc.html, /mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac_716/qc/rep1/AH716_S6_L008_R1_001.trim_qc.txt]'
Script file : '/mnt/NFS/homeGene/HarrisLab/sxx128/atac_analysis/atac.bds.20180405_180322_656_parallel_88/task.ataqc.ataqc_rep1.line_109.id_41.sh'
Exit status : '1'
Program :
Fatal error: /mnt/NFS/homeGene/HarrisLab/sxx128/atac/atac_dnase_pipelines/atac.bds, line 1665, pos 2. Task/s failed. atac.bds, line 78 : main() atac.bds, line 81 : void main() { // atac pipeline starts here atac.bds, line 101 : ataqc() atac.bds, line 1654 : void ataqc() { atac.bds, line 1665 : wait
Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done.