Closed shmilyfhh closed 5 years ago
that's strange. can you attach the whole output file by chance? I can't tell what the command is trying to do...
Thu Mar 7 11:31:38 EST 2019
/home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/pipelines/pepatac.py -N -P 28 --single-or-paired paired --prealignments rCRSd human_repeats --genome hg19 --TSS-name hg19_TSS.tsv --sample-name 1_3577UPitt_A-MSC-1_ATAC_hs_i1 --input fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_r1.fastq --input2 fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_r2.fastq --genome-size hs --peak-caller macs2 --trimmer skewer --deduplicator samblaster -O pepATAC/
/primary/home/huihui.fan/.local/lib/python2.7/site-packages/piper-0.9.4-py2.7.egg/pypiper
/secondary/projects/shen/projects/tmp/piplines/pepatac/pipelines
input2
: ['fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_r2.fastq']
paired_end
: True
prealignments
: ['rCRSd', 'human_repeats']
new_start
: True
output_parent
: pepATAC/
frip_ref_peaks
: None
recover
: False
TSS_name
: hg19_TSS.tsv
deduplicator
: samblaster
sample_name
: 1_3577UPitt_A-MSC-1_ATAC_hs_i1
lite
: False
input
: ['fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_r1.fastq']
genome_size
: hs
config_file
: pepatac.yaml
mem
: 4000
force_follow
: False
cores
: 28
anno_name
: None
trimmer
: skewer
peak_caller
: macs2
keep
: False
single_or_paired
: paired
dirty
: False
genome_assembly
: hg19
no_fifo
: False
Changed status from initializing to running.
Using custom config file: /home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/pipelines/pepatac.yaml
Loading config file: /home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/pipelines/pepatac.yaml
/home/huihui.fan/.local/lib/python2.7/site-packages/piper-0.9.4-py2.7.egg/pypiper/manager.py:288: YAMLLoadWarning: Calling yaml.load() without Loader=... is deprecated. The default Loader is unsafe. *** Please read https://msg.pyyaml.org/load for full details. config = yaml.load(conf) Local input file: fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_r1.fastq Local input file: fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_r2.fastq
File_mb
9191.129 PEPATAC RES
Read_type
paired PEPATAC RES
Genome
hg19 PEPATAC RES
Number of input file sets: 2
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.fastq
ln -sf /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_r1.fastq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.fastq
Process 28555 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 0.0GB) Local input file: '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.fastq'
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.fastq
ln -sf /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_r2.fastq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.fastq
Process 28556 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 0.0GB) Local input file: '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.fastq' Found .fastq file; no conversion necessary Found .fastq file; no conversion necessary
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.fastq
ln -sf /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.fastq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.fastq
Process 28557 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 0.0GB)
ln -sf /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.fastq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.fastq
Process 28558 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 0.0GB) Follow: ['/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.fastq', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.fastq'] ['/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.fastq', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.fastq']
Raw_reads
66564988 PEPATAC RES
Fastq_reads
66564988 PEPATAC RES ['/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.fastq', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.fastq']
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq
skewer -f sanger -t 28 -m pe -x /home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/tools/NexteraPE-PE.fa --quiet -o /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.fastq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.fastq
.--. .-. : .--': :.-. `. `. : `'.' .--. .-..-..-. .--. .--. _`, :: . `.' '_.': `; `; :' '_.': ..' `.__.':_;:_;`.__.'`.__.__.'`.__.':_; skewer v0.2.2 [April 4, 2016] Parameters used: -- 3' end adapter sequences in file (-x): /home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/tools/NexteraPE-PE.fa A: [0;33mAGATGTGTATAAGAGACAG [0mB: [0;33mAGATGTGTATAAGAGACAG [0mC: [0;33mTCGTCGGCAGCGTCAGATGTGTATAAGAGACAG [0mD: [0;33mCTGTCTCTTATACACATCTGACGCTGCCGACGA [0mE: [0;33mGTCTCGTGGGCTCGGAGATGTGTATAAGAGACAG [0mF: [0;33mCTGTCTCTTATACACATCTCCGAGCCCACGAGA [0m-- maximum error ratio allowed (-r): 0.100 -- maximum indel error ratio allowed (-d): 0.030 -- minimum read length allowed after trimming (-l): 18 -- file format (-f): Sanger/Illumina 1.8+ FASTQ -- number of concurrent threads (-t): 28 Thu Mar 7 11:31:53 2019[0;32m >> started[0m Thu Mar 7 11:32:47 2019[0;32m >> done[0m (54.228s) 33282494 read pairs processed; of these: 21 ( 0.00%) short read pairs filtered out after trimming by size control 10 ( 0.00%) empty read pairs filtered out after trimming by size control 33282463 (100.00%) read pairs available; of these: 59948 ( 0.18%) trimmed read pairs available after processing 33222515 (99.82%) untrimmed read pairs available after processing log has been saved to "/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1-trimmed.log".
Process 28579 returned: (0). Elapsed: 0:00:54. Peak memory: (Process: 0.0GB; Pipeline: 0.0GB)
mv /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1-trimmed-pair1.fastq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq
Process 28619 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 0.0GB)
mv /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1-trimmed-pair2.fastq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq
Process 28620 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 0.0GB) Follow: Evaluating read trimming
Trimmed_reads
66564926 PEPATAC RES
Trim_loss_rate
0.0 PEPATAC RES
Targetless command, running...
/primary/vari/software/fastqc/default/fastqc --noextract --outdir /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastqc /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq
Started analysis of 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 5% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 10% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 15% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 20% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 25% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 30% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 35% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 40% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 45% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 50% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 55% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 60% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 65% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 70% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 75% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 80% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 85% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 90% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Approx 95% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq Analysis complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq
Process 28627 returned: (0). Elapsed: 0:01:38. Peak memory: (Process: 0.0GB; Pipeline: 0.0GB)
FastQC report r1
fastqc/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim_fastqc.html FastQC report r1 None PEPATAC OBJ
Targetless command, running...
/primary/vari/software/fastqc/default/fastqc --noextract --outdir /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastqc /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq
Started analysis of 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 5% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 10% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 15% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 20% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 25% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 30% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 35% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 40% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 45% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 50% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 55% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 60% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 65% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 70% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 75% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 80% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 85% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 90% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Approx 95% complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq Analysis complete for 1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq
Process 28668 returned: (0). Elapsed: 0:01:43. Peak memory: (Process: 0.0GB; Pipeline: 0.0GB)
FastQC report r2
fastqc/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim_fastqc.html FastQC report r2 None PEPATAC OBJ
Prealignment assemblies: ['rCRSd', 'human_repeats']
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/rCRSd_bt2
mkfifo /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/rCRSd_bt2
Process 28719 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 0.0GB)
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_unmap_R2.fq
perl /home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/tools/filter_paired_fq.pl /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/rCRSd_bt2 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R2.trim.fastq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_unmap_R1.fq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_unmap_R2.fq
Not waiting for subprocess(es): [28720]
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_bt_aln_summary.log
(bowtie2 -p 28 -k 1 -D 20 -R 3 -N 1 -L 20 -i S,1,0.50 -x /home/huihui.fan/shen-secondary/projects/tmp/genomes/hg19/refgenie/rCRSd/indexed_bowtie2/rCRSd --rg-id 1_3577UPitt_A-MSC-1_ATAC_hs_i1 -U /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1_R1.trim.fastq --un /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/rCRSd_bt2 > /dev/null) 2>/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_bt_aln_summary.log
Process 28721 returned: (0). Elapsed: 0:07:33. Peak memory: (Process: 0.372GB; Pipeline: 0.372GB)
grep 'aligned exactly 1 time' /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_bt_aln_summary.log | awk '{print $1}'
Aligned_reads_rCRSd
23298870.0 PEPATAC RES
Alignment_rate_rCRSd
35.0 PEPATAC RES
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/human_repeats_bt2
mkfifo /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/human_repeats_bt2
Process 28894 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 0.372GB)
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_unmap_R2.fq
perl /home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/tools/filter_paired_fq.pl /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/human_repeats_bt2 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_unmap_R1.fq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_unmap_R2.fq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_unmap_R1.fq /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_unmap_R2.fq
Not waiting for subprocess(es): [28895]
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_bt_aln_summary.log
(bowtie2 -p 28 -k 1 -D 20 -R 3 -N 1 -L 20 -i S,1,0.50 -x /home/huihui.fan/shen-secondary/projects/tmp/genomes/hg19/refgenie/human_repeats/indexed_bowtie2/human_repeats --rg-id 1_3577UPitt_A-MSC-1_ATAC_hs_i1 -U /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_unmap_R1.fq --un /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/human_repeats_bt2 > /dev/null) 2>/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_bt_aln_summary.log
11649435 reads skipped 0 reads lost
Process 28896 returned: (0). Elapsed: 0:05:05. Peak memory: (Process: 0.405GB; Pipeline: 0.405GB)
grep 'aligned exactly 1 time' /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_bt_aln_summary.log | awk '{print $1}'
Aligned_reads_human_repeats
2534060.0 PEPATAC RES
Alignment_rate_human_repeats
3.81 PEPATAC RES
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam
bowtie2 -p 28 --very-sensitive -X 2000 --rg-id 1_3577UPitt_A-MSC-1_ATAC_hs_i1 -x /home/huihui.fan/shen-secondary/projects/tmp/genomes/hg19/refgenie/hg19/indexed_bowtie2/hg19 -1 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_unmap_R1.fq -2 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_unmap_R2.fq | samtools view -bS - -@ 1 | samtools sort - -@ 1 -T /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/tmpF6Im_7 -o /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_temp.bam
1267030 reads skipped 0 reads lost 20365998 reads; of these: 20365998 (100.00%) were paired; of these: 572772 (2.81%) aligned concordantly 0 times 17213026 (84.52%) aligned concordantly exactly 1 time 2580200 (12.67%) aligned concordantly >1 times ---- 572772 pairs aligned concordantly 0 times; of these: 94577 (16.51%) aligned discordantly 1 time ---- 478195 pairs aligned 0 times concordantly or discordantly; of these: 956390 mates make up the pairs; of these: 704019 (73.61%) aligned 0 times 133818 (13.99%) aligned exactly 1 time 118553 (12.40%) aligned >1 times 98.27% overall alignment rate
Process 29018 returned: (0). Elapsed: 0:18:04. Peak memory: (Process: 4.13GB; Pipeline: 4.13GB)
Process 29019 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: None; Pipeline: 4.13GB)
[bam_sort_core] merging from 17 files...
Process 29020 returned: (0). Elapsed: 0:05:24. Peak memory: (Process: 0.0GB; Pipeline: 4.13GB)
samtools view -q 10 -b -@ 28 -U /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_fail_qc.bam -f 2 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_temp.bam > /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam
Process 29583 returned: (0). Elapsed: 0:01:20. Peak memory: (Process: 0.478GB; Pipeline: 4.13GB) Follow:
Mapped_reads
40027977 PEPATAC RES
QC_filtered_reads
2954949.0 PEPATAC RES
Aligned_reads
37073028 PEPATAC RES
Alignment_rate
55.69 PEPATAC RES
Total_efficiency
55.69 PEPATAC RES
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_unmap_R1.fq.gz
gzip /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_unmap_R1.fq
Process 29649 returned: (0). Elapsed: 0:04:28. Peak memory: (Process: 0.0GB; Pipeline: 4.13GB)
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_unmap_R2.fq.gz
gzip /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_rCRSd_unmap_R2.fq
Process 29765 returned: (0). Elapsed: 0:04:42. Peak memory: (Process: 0.0GB; Pipeline: 4.13GB)
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_unmap_R1.fq.gz
gzip /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_unmap_R1.fq
Process 29851 returned: (0). Elapsed: 0:04:22. Peak memory: (Process: 0.0GB; Pipeline: 4.13GB)
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_unmap_R2.fq.gz
gzip /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/1_3577UPitt_A-MSC-1_ATAC_hs_i1_human_repeats_unmap_R2.fq
Process 29932 returned: (0). Elapsed: 0:04:26. Peak memory: (Process: 0.0GB; Pipeline: 4.13GB)
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam.bai
samtools index /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_temp.bam
Process 30071 returned: (0). Elapsed: 0:00:25. Peak memory: (Process: 0.0GB; Pipeline: 4.13GB)
samtools index /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam
Process 30076 returned: (0). Elapsed: 0:00:24. Peak memory: (Process: 0.0GB; Pipeline: 4.13GB)
samtools idxstats /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_temp.bam | grep -we 'chrM' -we 'chrMT' -we 'M' -we 'MT'| cut -f 3
Mitochondrial_reads
15517.0 PEPATAC RES
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_noMT.bam
samtools idxstats /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam | cut -f 1 | grep -vwe 'chrM' -vwe 'chrMT' -vwe 'M' -vwe 'MT'| xargs samtools view -b -@ 28 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam > /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_noMT.bam
Process 30090 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 4.13GB)
Process 30091 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: None; Pipeline: 4.13GB)
Process 30092 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: None; Pipeline: 4.13GB)
Process 30093 returned: (0). Elapsed: 0:00:42. Peak memory: (Process: 0.486GB; Pipeline: 4.13GB)
mv /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_noMT.bam /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam
Process 30135 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 4.13GB)
samtools index /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam
Process 30136 returned: (0). Elapsed: 0:00:24. Peak memory: (Process: 0.0GB; Pipeline: 4.13GB)
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/QC_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_bamQC.tsv
/home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/tools/bamQC.py -i /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam -c 28 -o /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/QC_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_bamQC.tsv
Configured logger 'root' using pararead v0.5.0 Registering input file: '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam' Temporary files will be stored in: '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/QC_hg19/tmp_1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort_DL7THA' Processing with 28 cores... [Name: chr19_gl000208_random; Size: 92689] [Name: chr21_gl000210_random; Size: 27682] [Name: chr6_apd_hap1; Size: 4622290] [Name: chr13; Size: 115169878] [Name: chr12; Size: 133851895] [Name: chr11; Size: 135006516] [Name: chr10; Size: 135534747] [Name: chr17; Size: 81195210] [Name: chr16; Size: 90354753] [Name: chr15; Size: 102531392] [Name: chr14; Size: 107349540] [Name: chr19; Size: 59128983] [Name: chr18; Size: 78077248] [Name: chr9_gl000198_random; Size: 90085] [Name: chrUn_gl000239; Size: 33824] [Name: chrUn_gl000238; Size: 39939] [Name: chrUn_gl000233; Size: 45941] [Name: chrUn_gl000232; Size: 40652] [Name: chrUn_gl000231; Size: 27386] [Name: chrUn_gl000230; Size: 43691] [Name: chrUn_gl000237; Size: 45867] [Name: chrUn_gl000236; Size: 41934] [Name: chrUn_gl000235; Size: 34474] [Name: chrUn_gl000234; Size: 40531] [Name: chr6_qbl_hap6; Size: 4611984] [Name: chr11_gl000202_random; Size: 40103] [Name: chr17_gl000206_random; Size: 41001] [Name: chr6_cox_hap2; Size: 4795371] [Name: chr4_gl000193_random; Size: 189789] [Name: chrUn_gl000248; Size: 39786] [Name: chrUn_gl000246; Size: 38154] [Name: chrUn_gl000247; Size: 36422] [Name: chrUn_gl000244; Size: 39929] [Name: chrUn_gl000242; Size: 43523] [Name: chrUn_gl000243; Size: 43341] [Name: chrUn_gl000240; Size: 41933] [Name: chrUn_gl000241; Size: 42152] [Name: chr17_gl000204_random; Size: 81310] [Name: chr17_ctg5_hap1; Size: 1680828] [Name: chr17_gl000205_random; Size: 174588] [Name: chr9_gl000199_random; Size: 169874] [Name: chr9_gl000201_random; Size: 36148] [Name: chr8; Size: 146364022] [Name: chr6_ssto_hap7; Size: 4928567] [Name: chr8_gl000197_random; Size: 37175] [Name: chr6_dbb_hap3; Size: 4610396] [Name: chr7_gl000195_random; Size: 182896] [Name: chr1_gl000191_random; Size: 106433] [Name: chr4_ctg9_hap1; Size: 590426] [Name: chr3; Size: 198022430] [Name: chrUn_gl000229; Size: 19913] [Name: chr1; Size: 249250621] [Name: chr17_gl000203_random; Size: 37498] [Name: chr4_gl000194_random; Size: 191469] [Name: chrY; Size: 59373566] [Name: chrX; Size: 155270560] [Name: chr9; Size: 141213431] [Name: chr8_gl000196_random; Size: 38914] [Name: chr6_mann_hap4; Size: 4683263] [Name: chrUn_gl000211; Size: 166566] [Name: chrUn_gl000213; Size: 164239] [Name: chrUn_gl000212; Size: 186858] [Name: chrUn_gl000215; Size: 172545] [Name: chrUn_gl000214; Size: 137718] [Name: chrUn_gl000217; Size: 172149] [Name: chrUn_gl000216; Size: 172294] [Name: chrUn_gl000219; Size: 179198] [Name: chrUn_gl000218; Size: 161147] [Name: chr19_gl000209_random; Size: 159169] [Name: chr22; Size: 51304566] [Name: chr20; Size: 63025520] [Name: chr21; Size: 48129895] [Name: chr6_mcf_hap5; Size: 4833398] [Name: chr7; Size: 159138663] [Name: chr6; Size: 171115067] [Name: chr5; Size: 180915260] [Name: chr4; Size: 191154276] [Name: chrUn_gl000228; Size: 129120] [Name: chr2; Size: 243199373] [Name: chr1_gl000192_random; Size: 547496] [Name: chrUn_gl000224; Size: 179693] [Name: chrUn_gl000225; Size: 211173] [Name: chrUn_gl000226; Size: 15008] [Name: chrUn_gl000227; Size: 128374] [Name: chrUn_gl000220; Size: 161802] [Name: chrUn_gl000221; Size: 155397] [Name: chrUn_gl000222; Size: 186861] [Name: chrUn_gl000223; Size: 180455] [Name: chr18_gl000207_random; Size: 4262] Discarding 4 chunk(s) of reads: ['chrUn_gl000249', 'chrUn_gl000245', 'chr9_gl000200_random', 'chrM'] Keeping 89 chunk(s) of reads: ['chr19_gl000208_random', 'chr21_gl000210_random', 'chr6_apd_hap1', 'chr13', 'chr12', 'chr11', 'chr10', 'chr17', 'chr16', 'chr15', 'chr14', 'chr19', 'chr18', 'chr9_gl000198_random', 'chrUn_gl000239', 'chrUn_gl000238', 'chrUn_gl000233', 'chrUn_gl000232', 'chrUn_gl000231', 'chrUn_gl000230', 'chrUn_gl000237', 'chrUn_gl000236', 'chrUn_gl000235', 'chrUn_gl000234', 'chr6_qbl_hap6', 'chr11_gl000202_random', 'chr17_gl000206_random', 'chr6_cox_hap2', 'chr4_gl000193_random', 'chrUn_gl000248', 'chrUn_gl000246', 'chrUn_gl000247', 'chrUn_gl000244', 'chrUn_gl000242', 'chrUn_gl000243', 'chrUn_gl000240', 'chrUn_gl000241', 'chr17_gl000204_random', 'chr17_ctg5_hap1', 'chr17_gl000205_random', 'chr9_gl000199_random', 'chr9_gl000201_random', 'chr8', 'chr6_ssto_hap7', 'chr8_gl000197_random', 'chr6_dbb_hap3', 'chr7_gl000195_random', 'chr1_gl000191_random', 'chr4_ctg9_hap1', 'chr3', 'chrUn_gl000229', 'chr1', 'chr17_gl000203_random', 'chr4_gl000194_random', 'chrY', 'chrX', 'chr9', 'chr8_gl000196_random', 'chr6_mann_hap4', 'chrUn_gl000211', 'chrUn_gl000213', 'chrUn_gl000212', 'chrUn_gl000215', 'chrUn_gl000214', 'chrUn_gl000217', 'chrUn_gl000216', 'chrUn_gl000219', 'chrUn_gl000218', 'chr19_gl000209_random', 'chr22', 'chr20', 'chr21', 'chr6_mcf_hap5', 'chr7', 'chr6', 'chr5', 'chr4', 'chrUn_gl000228', 'chr2', 'chr1_gl000192_random', 'chrUn_gl000224', 'chrUn_gl000225', 'chrUn_gl000226', 'chrUn_gl000227', 'chrUn_gl000220', 'chrUn_gl000221', 'chrUn_gl000222', 'chrUn_gl000223', 'chr18_gl000207_random'] Reduce step (merge files)... Merging 89 files into output file: '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/QC_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_bamQC.tsv'
Process 30142 returned: (0). Elapsed: 0:00:25. Peak memory: (Process: 7.362GB; Pipeline: 7.362GB) Follow:
awk '{ for (i=1; i<=NF; ++i) { if ($i ~ "NRF") c=i } getline; print $c }' /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/QC_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_bamQC.tsv
awk '{ for (i=1; i<=NF; ++i) { if ($i ~ "PBC1") c=i } getline; print $c }' /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/QC_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_bamQC.tsv
awk '{ for (i=1; i<=NF; ++i) { if ($i ~ "PBC2") c=i } getline; print $c }' /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/QC_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_bamQC.tsv
NRF
0.93 PEPATAC RES
PBC1
0.96 PEPATAC RES
PBC2
26.51 PEPATAC RES
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_unmap.bam
samtools view -b -@ 28 -f 12 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_temp.bam > /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_unmap.bam
Process 30187 returned: (0). Elapsed: 0:00:26. Peak memory: (Process: 0.031GB; Pipeline: 7.362GB) Follow:
samtools view -c -f 4 -@ 28 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_temp.bam
Unmapped_reads
704019.0 PEPATAC RES
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort_dedup.bam
samtools sort -n -@ 28 -T /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/tmpmb3Bu2 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort.bam | samtools view -h - -@ 28 | samblaster -r 2> /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_dedup_metrics_log.txt | samtools view -b - -@ 28 | samtools sort - -@ 28 -T /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/tmpmb3Bu2 -o /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort_dedup.bam
Process 30232 returned: (0). Elapsed: 0:07:14. Peak memory: (Process: 0.0GB; Pipeline: 7.362GB)
Process 30233 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: None; Pipeline: 7.362GB)
Process 30234 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: None; Pipeline: 7.362GB)
Process 30235 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: None; Pipeline: 7.362GB)
Process 30237 returned: (0). Elapsed: 0:01:00. Peak memory: (Process: 0.0GB; Pipeline: 7.362GB)
samtools index /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort_dedup.bam
Process 30469 returned: (0). Elapsed: 0:00:24. Peak memory: (Process: 0.0GB; Pipeline: 7.362GB) Follow:
grep 'Removed' /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_dedup_metrics_log.txt | cut -f 3 -d ' '
Duplicate_reads
808827 PEPATAC RES
Dedup_aligned_reads
36264201.0 PEPATAC RES
Dedup_alignment_rate
54.48 PEPATAC RES
Dedup_total_efficiency
54.48 PEPATAC RES
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/temp/flag_completed
/home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/tools/bamSitesToWig.py -i /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort_dedup.bam -c /home/huihui.fan/shen-secondary/projects/tmp/genomes/hg19/refgenie/hg19/hg19.chromSizes -b /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/1_3577UPitt_A-MSC-1_ATAC_hs_i1_shift.bed -o /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/1_3577UPitt_A-MSC-1_ATAC_hs_i1_exact.bw -w /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_smooth.bw -p 18
Configured logger 'root' using pararead v0.5.0 Cutting parallel chroms in half to accommodate smooth track. Registering input file: '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort_dedup.bam' Temporary files will be stored in: '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/tmp_1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort_dedup_cuttrace_7aYgNe' Processing with 9 cores... [Name: chr19_gl000208_random; Size: 92689] [Name: chr13; Size: 115169878] [Name: chr10; Size: 135534747] [Name: chr15; Size: 102531392] [Name: chr18; Size: 78077248] [Name: chrUn_gl000238; Size: 39939] [Name: chrUn_gl000231; Size: 27386] [Name: chrUn_gl000236; Size: 41934] [Name: chr6_qbl_hap6; Size: 4611984] [Name: chrUn_gl000230; Size: 43691] [Name: chrUn_gl000233; Size: 45941] [Name: chrUn_gl000235; Size: 34474] [Name: chr21_gl000210_random; Size: 27682] [Name: chrUn_gl000232; Size: 40652] [Name: chrUn_gl000237; Size: 45867] [Name: chrUn_gl000234; Size: 40531] [Name: chr6_apd_hap1; Size: 4622290] [Name: chr6_cox_hap2; Size: 4795371] [Name: chrUn_gl000246; Size: 38154] [Name: chrUn_gl000242; Size: 43523] [Name: chrUn_gl000247; Size: 36422] [Name: chrUn_gl000243; Size: 43341] [Name: chrUn_gl000244; Size: 39929] [Name: chrUn_gl000240; Size: 41933] [Name: chrUn_gl000241; Size: 42152] [Name: chr17_gl000205_random; Size: 174588] [Name: chr17_gl000204_random; Size: 81310] [Name: chr17_ctg5_hap1; Size: 1680828] [Name: chr9_gl000199_random; Size: 169874] [Name: chr9_gl000201_random; Size: 36148] [Name: chr8; Size: 146364022] [Name: chr6_dbb_hap3; Size: 4610396] [Name: chr11_gl000202_random; Size: 40103] [Name: chr17_gl000206_random; Size: 41001] [Name: chr4_ctg9_hap1; Size: 590426] [Name: chr1; Size: 249250621] [Name: chr3; Size: 198022430] [Name: chr4_gl000193_random; Size: 189789] [Name: chrUn_gl000248; Size: 39786] [Name: chrY; Size: 59373566] [Name: chr7_gl000195_random; Size: 182896] [Name: chr1_gl000191_random; Size: 106433] [Name: chr8_gl000196_random; Size: 38914] [Name: chr6_mann_hap4; Size: 4683263] [Name: chrUn_gl000211; Size: 166566] [Name: chrUn_gl000213; Size: 164239] [Name: chrUn_gl000212; Size: 186858] [Name: chrUn_gl000215; Size: 172545] [Name: chrUn_gl000214; Size: 137718] [Name: chrUn_gl000217; Size: 172149] [Name: chrUn_gl000216; Size: 172294] [Name: chrUn_gl000219; Size: 179198] [Name: chrUn_gl000218; Size: 161147] [Name: chr19_gl000209_random; Size: 159169] [Name: chr22; Size: 51304566] [Name: chrX; Size: 155270560] [Name: chr20; Size: 63025520] [Name: chr9_gl000198_random; Size: 90085] [Name: chrUn_gl000239; Size: 33824] [Name: chr6_mcf_hap5; Size: 4833398] [Name: chr7; Size: 159138663] [Name: chr14; Size: 107349540] [Name: chr12; Size: 133851895] [Name: chr21; Size: 48129895] [Name: chr17; Size: 81195210] [Name: chr6_ssto_hap7; Size: 4928567] [Name: chr8_gl000197_random; Size: 37175] [Name: chr5; Size: 180915260] [Name: chr9; Size: 141213431] [Name: chr2; Size: 243199373] [Name: chr19; Size: 59128983] [Name: chrUn_gl000229; Size: 19913] [Name: chrUn_gl000225; Size: 211173] [Name: chrUn_gl000226; Size: 15008] [Name: chrUn_gl000227; Size: 128374] [Name: chrUn_gl000220; Size: 161802] [Name: chrUn_gl000221; Size: 155397] [Name: chrUn_gl000222; Size: 186861] [Name: chrUn_gl000223; Size: 180455] [Name: chr18_gl000207_random; Size: 4262] [Name: chr11; Size: 135006516] [Name: chr16; Size: 90354753] [Name: chr6; Size: 171115067] [Name: chr17_gl000203_random; Size: 37498] [Name: chr4_gl000194_random; Size: 191469] [Name: chr4; Size: 191154276] [Name: chr1_gl000192_random; Size: 547496] [Name: chrUn_gl000224; Size: 179693] [Name: chrUn_gl000228; Size: 129120] Discarding 4 chunk(s) of reads: ['chrUn_gl000249', 'chrUn_gl000245', 'chr9_gl000200_random', 'chrM'] Keeping 89 chunk(s) of reads: ['chr19_gl000208_random', 'chr21_gl000210_random', 'chr6_apd_hap1', 'chr13', 'chr12', 'chr11', 'chr10', 'chr17', 'chr16', 'chr15', 'chr14', 'chr19', 'chr18', 'chr9_gl000198_random', 'chrUn_gl000239', 'chrUn_gl000238', 'chrUn_gl000233', 'chrUn_gl000232', 'chrUn_gl000231', 'chrUn_gl000230', 'chrUn_gl000237', 'chrUn_gl000236', 'chrUn_gl000235', 'chrUn_gl000234', 'chr6_qbl_hap6', 'chr11_gl000202_random', 'chr17_gl000206_random', 'chr6_cox_hap2', 'chr4_gl000193_random', 'chrUn_gl000248', 'chrUn_gl000246', 'chrUn_gl000247', 'chrUn_gl000244', 'chrUn_gl000242', 'chrUn_gl000243', 'chrUn_gl000240', 'chrUn_gl000241', 'chr17_gl000204_random', 'chr17_ctg5_hap1', 'chr17_gl000205_random', 'chr9_gl000199_random', 'chr9_gl000201_random', 'chr8', 'chr6_ssto_hap7', 'chr8_gl000197_random', 'chr6_dbb_hap3', 'chr7_gl000195_random', 'chr1_gl000191_random', 'chr4_ctg9_hap1', 'chr3', 'chrUn_gl000229', 'chr1', 'chr17_gl000203_random', 'chr4_gl000194_random', 'chrY', 'chrX', 'chr9', 'chr8_gl000196_random', 'chr6_mann_hap4', 'chrUn_gl000211', 'chrUn_gl000213', 'chrUn_gl000212', 'chrUn_gl000215', 'chrUn_gl000214', 'chrUn_gl000217', 'chrUn_gl000216', 'chrUn_gl000219', 'chrUn_gl000218', 'chr19_gl000209_random', 'chr22', 'chr20', 'chr21', 'chr6_mcf_hap5', 'chr7', 'chr6', 'chr5', 'chr4', 'chrUn_gl000228', 'chr2', 'chr1_gl000192_random', 'chrUn_gl000224', 'chrUn_gl000225', 'chrUn_gl000226', 'chrUn_gl000227', 'chrUn_gl000220', 'chrUn_gl000221', 'chrUn_gl000222', 'chrUn_gl000223', 'chr18_gl000207_random'] Reduce step (merge files)... Merging 89 files into output file: '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/1_3577UPitt_A-MSC-1_ATAC_hs_i1_exact.bw' Merging 89 files into output file: '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_smooth.bw'
Process 30480 returned: (0). Elapsed: 0:05:53. Peak memory: (Process: 14.606GB; Pipeline: 14.606GB)
touch /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/temp/flag_completed
Process 31412 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 14.606GB) Skipping TSS -- TSS enrichment requires TSS annotation file: /home/huihui.fan/shen-secondary/projects/tmp/genomes/hg19/refgenie/hg19/hg19_TSS.tsv
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.narrowPeak
/home/huihui.fan/.local/bin/macs2 callpeak -t /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/1_3577UPitt_A-MSC-1_ATAC_hs_i1_shift.bed -f BED -g hs --outdir /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19 -n 1_3577UPitt_A-MSC-1_ATAC_hs_i1 -q 0.01 --shift 0 --nomodel
INFO @ Thu, 07 Mar 2019 12:50:08: # Command line: callpeak -t /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/1_3577UPitt_A-MSC-1_ATAC_hs_i1_shift.bed -f BED -g hs --outdir /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19 -n 1_3577UPitt_A-MSC-1_ATAC_hs_i1 -q 0.01 --shift 0 --nomodel # ARGUMENTS LIST: # name = 1_3577UPitt_A-MSC-1_ATAC_hs_i1 # format = BED # ChIP-seq file = ['/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/1_3577UPitt_A-MSC-1_ATAC_hs_i1_shift.bed'] # control file = None # effective genome size = 2.70e+09 # band width = 300 # model fold = [5, 50] # qvalue cutoff = 1.00e-02 # The maximum gap between significant sites is assigned as the read length/tag size. # The minimum length of peaks is assigned as the predicted fragment length "d". # Larger dataset will be scaled towards smaller dataset. # Range for calculating regional lambda is: 10000 bps # Broad region calling is off # Paired-End mode is off INFO @ Thu, 07 Mar 2019 12:50:08: #1 read tag files... INFO @ Thu, 07 Mar 2019 12:50:08: #1 read treatment tags... INFO @ Thu, 07 Mar 2019 12:50:09: 1000000 INFO @ Thu, 07 Mar 2019 12:50:10: 2000000 INFO @ Thu, 07 Mar 2019 12:50:11: 3000000 INFO @ Thu, 07 Mar 2019 12:50:13: 4000000 INFO @ Thu, 07 Mar 2019 12:50:14: 5000000 INFO @ Thu, 07 Mar 2019 12:50:15: 6000000 INFO @ Thu, 07 Mar 2019 12:50:16: 7000000 INFO @ Thu, 07 Mar 2019 12:50:17: 8000000 INFO @ Thu, 07 Mar 2019 12:50:18: 9000000 INFO @ Thu, 07 Mar 2019 12:50:20: 10000000 INFO @ Thu, 07 Mar 2019 12:50:21: 11000000 INFO @ Thu, 07 Mar 2019 12:50:22: 12000000 INFO @ Thu, 07 Mar 2019 12:50:23: 13000000 INFO @ Thu, 07 Mar 2019 12:50:24: 14000000 INFO @ Thu, 07 Mar 2019 12:50:25: 15000000 INFO @ Thu, 07 Mar 2019 12:50:26: 16000000 INFO @ Thu, 07 Mar 2019 12:50:27: 17000000 INFO @ Thu, 07 Mar 2019 12:50:28: 18000000 INFO @ Thu, 07 Mar 2019 12:50:29: 19000000 INFO @ Thu, 07 Mar 2019 12:50:30: 20000000 INFO @ Thu, 07 Mar 2019 12:50:31: 21000000 INFO @ Thu, 07 Mar 2019 12:50:32: 22000000 INFO @ Thu, 07 Mar 2019 12:50:33: 23000000 INFO @ Thu, 07 Mar 2019 12:50:34: 24000000 INFO @ Thu, 07 Mar 2019 12:50:35: 25000000 INFO @ Thu, 07 Mar 2019 12:50:36: 26000000 INFO @ Thu, 07 Mar 2019 12:50:37: 27000000 INFO @ Thu, 07 Mar 2019 12:50:38: 28000000 INFO @ Thu, 07 Mar 2019 12:50:39: 29000000 INFO @ Thu, 07 Mar 2019 12:50:40: 30000000 INFO @ Thu, 07 Mar 2019 12:50:41: 31000000 INFO @ Thu, 07 Mar 2019 12:50:42: 32000000 INFO @ Thu, 07 Mar 2019 12:50:43: 33000000 INFO @ Thu, 07 Mar 2019 12:50:44: 34000000 INFO @ Thu, 07 Mar 2019 12:50:45: 35000000 INFO @ Thu, 07 Mar 2019 12:50:46: #1 tag size is determined as 50 bps INFO @ Thu, 07 Mar 2019 12:50:46: #1 tag size = 50.0 INFO @ Thu, 07 Mar 2019 12:50:46: #1 total tags in treatment: 35453413 INFO @ Thu, 07 Mar 2019 12:50:46: #1 user defined the maximum tags... INFO @ Thu, 07 Mar 2019 12:50:46: #1 filter out redundant tags at the same location and the same strand by allowing at most 1 tag(s) INFO @ Thu, 07 Mar 2019 12:50:46: #1 tags after filtering in treatment: 30887984 INFO @ Thu, 07 Mar 2019 12:50:46: #1 Redundant rate of treatment: 0.13 INFO @ Thu, 07 Mar 2019 12:50:46: #1 finished! INFO @ Thu, 07 Mar 2019 12:50:46: #2 Build Peak Model... INFO @ Thu, 07 Mar 2019 12:50:46: #2 Skipped... INFO @ Thu, 07 Mar 2019 12:50:46: #2 Use 200 as fragment length INFO @ Thu, 07 Mar 2019 12:50:46: #3 Call peaks... INFO @ Thu, 07 Mar 2019 12:50:46: #3 Pre-compute pvalue-qvalue table... INFO @ Thu, 07 Mar 2019 12:52:10: #3 Call peaks for each chromosome... INFO @ Thu, 07 Mar 2019 12:53:15: #4 Write output xls file... /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.xls INFO @ Thu, 07 Mar 2019 12:53:16: #4 Write peak in narrowPeak format file... /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.narrowPeak INFO @ Thu, 07 Mar 2019 12:53:16: #4 Write summits bed file... /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_summits.bed INFO @ Thu, 07 Mar 2019 12:53:17: Done!
Process 31413 returned: (0). Elapsed: 0:03:09. Peak memory: (Process: 0.0GB; Pipeline: 14.606GB) Follow:
Peak_count
126763 PEPATAC RES
samtools view -@ 4 -c -L /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.narrowPeak /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort_dedup.bam
('37073028', '12914714\n')
FRiP
0.348358758286 PEPATAC RES
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.bigBed
Rscript /home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/tools/narrowPeakToBigBed.R /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.narrowPeak /home/huihui.fan/shen-secondary/projects/tmp/genomes/hg19/refgenie/hg19/hg19.chromSizes bedToBigBed /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.bigBed
pass1 - making usageList (49 chroms): 53 millis pass2 - checking and writing primary data (126764 records, 10 fields): 668 millis [1] TRUE [1] TRUE
Process 31488 returned: (0). Elapsed: 0:00:06. Peak memory: (Process: 0.0GB; Pipeline: 14.606GB)
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks_sort.bed
cut -f 1-3 /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.narrowPeak > /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.bed
Process 31522 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.003GB; Pipeline: 14.606GB)
samtools view -H /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort_dedup.bam | grep 'SN:' | awk -F':' '{print $2,$3}' | awk -F' ' -v OFS=' ' '{print $1,$3}' > /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/chr_order.txt
Process 31524 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: None; Pipeline: 14.606GB)
Process 31525 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: None; Pipeline: 14.606GB)
Process 31526 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: None; Pipeline: 14.606GB)
Process 31527 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 14.606GB)
bedtools sort -i /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.bed -faidx /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/chr_order.txt > /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks_sort.bed
Process 31529 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 14.606GB)
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks_coverage.bed
bedtools coverage -sorted -counts -a /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks_sort.bed -b /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_sort_dedup.bam -g /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/chr_order.txt > /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks_coverage.bed
Process 31531 returned: (0). Elapsed: 0:01:01. Peak memory: (Process: 0.008GB; Pipeline: 14.606GB)
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/hg19_annotations.bed.gz
ln -sf /home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/anno/hg19_annotations.bed.gz /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/hg19_annotations.bed.gz
Process 31551 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: 0.0GB; Pipeline: 14.606GB)
gunzip -c /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/hg19_annotations.bed.gz | cut -f 4 | sort -u
Target to produce: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/QC_hg19/3' UTR
gunzip -c /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/raw/hg19_annotations.bed.gz | awk -F' ' '{print>"/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/QC_hg19/"$4}'
Process 31558 returned: (0). Elapsed: 0:00:01. Peak memory: (Process: 0.0GB; Pipeline: 14.606GB)
Process 31559 returned: (0). Elapsed: 0:00:00. Peak memory: (Process: None; Pipeline: 14.606GB)
Target exists: /secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/QC_hg19/3_UTR
New start mode: run anyway
Targetless command, running...
Traceback (most recent call last):
File "/home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/pipelines/pepatac.py", line 1411, in " + cmd + "
\n")
TypeError: cannot concatenate 'str' and 'NoneType' objects
Pipeline status: running
Conditional flag found: []
These conditional files were left in place:['/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/1_3577UPitt_A-MSC-1_ATAC_hs_i1.fastq', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/.fq', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/fastq/*.log', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/tmpjHOgSC', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/rCRSd_bt2', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/tmpxm8Dt3', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/prealignments/human_repeats_bt2', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/tmpF6Im_7', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_temp.bam.bai', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_temp.bam', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19/tmpmb3Bu2', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/temp/flag_completed', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/aligned_hg19_exact/temp', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks.bed', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/chr_order.txt', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/1_3577UPitt_A-MSC-1_ATAC_hs_i1/peak_calling_hg19/1_3577UPitt_A-MSC-1_ATAC_hs_i1_peaks_sort.bed'] Could not produce cleanup script for item(s):
Total time: 1:23:37
Changed status from running to failed. Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/home/huihui.fan/.local/lib/python2.7/site-packages/piper-0.9.4-py2.7.egg/pypiper/manager.py", line 1632, in _exit_handler self.fail_pipeline(Exception("Pipeline failure. See details above.")) File "/home/huihui.fan/.local/lib/python2.7/site-packages/piper-0.9.4-py2.7.egg/pypiper/manager.py", line 1528, in fail_pipeline raise e Exception: Pipeline failure. See details above.
Pypiper terminating spawned child process 28540...(tee) Child not responding to SIGTERM, trying SIGKILL...Thu Mar 7 12:55:20 EST 2019 Done.
this is a bug in pepatac. You can get around it for now by turning off new start mode. you'll have to just erase your output manually for now, and then re-run.
we will work on solving the bug for the future, though.
sounds great! Thank you so much.
The most recent release includes the expected fix to this issue.
Hi there,
I attached the error printout of one sample run. Any one could help? Thanks a lot.
Huihui
Target exists:
/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/QC_hg19/3_UTR
New start mode: run anywayTargetless command, running...
Traceback (most recent call last): File "/home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/pipelines/pepatac.py", line 1411, in
sys.exit(main())
File "/home/huihui.fan/shen-secondary/projects/tmp/piplines/pepatac/pipelines/pepatac.py", line 1287, in main
container=pm.container)
File "/home/huihui.fan/.local/lib/python2.7/site-packages/piper-0.9.4-py2.7.egg/pypiper/manager.py", line 716, in run
self.callprint(cmd, nofail, container) # Run command
File "/home/huihui.fan/.local/lib/python2.7/site-packages/piper-0.9.4-py2.7.egg/pypiper/manager.py", line 826, in callprint
self._report_command(cmd)
File "/home/huihui.fan/.local/lib/python2.7/site-packages/piper-0.9.4-py2.7.egg/pypiper/manager.py", line 1231, in _report_command
print("\n>
" + cmd + "
\n") TypeError: cannot concatenate 'str' and 'NoneType' objects Pipeline status: runningConditional flag found: []
These conditional files were left in place:['/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/fastq/2_3577UPitt_OM-MSC-1_ATAC_hs_i2.fastq', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/fastq/.fq', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/fastq/*.log', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/prealignments/tmp77IJ3D', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/prealignments/rCRSd_bt2', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/prealignments/tmpXeh9bk', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/prealignments/human_repeats_bt2', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/aligned_hg19/tmpFxujBi', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/aligned_hg19/2_3577UPitt_OM-MSC-1_ATAC_hs_i2_temp.bam.bai', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/aligned_hg19/2_3577UPitt_OM-MSC-1_ATAC_hs_i2_temp.bam', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/aligned_hg19/tmpP7Jl6D', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/aligned_hg19_exact/temp/flag_completed', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/aligned_hg19_exact/temp', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/peak_calling_hg19/2_3577UPitt_OM-MSC-1_ATAC_hs_i2_peaks.bed', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/peak_calling_hg19/chr_order.txt', '/secondary/projects/shen/projects/2019_01_07_MSC_Lan/ATACseq/pepATAC/2_3577UPitt_OM-MSC-1_ATAC_hs_i2/peak_calling_hg19/2_3577UPitt_OM-MSC-1_ATAC_hs_i2_peaks_sort.bed'] Could not produce cleanup script for item(s):
Pipeline failed at: (03-07 15:11:15) elapsed: 1.0 TIME
Total time: 1:49:49
Changed status from running to failed. Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs func(*targs, **kargs) File "/home/huihui.fan/.local/lib/python2.7/site-packages/piper-0.9.4-py2.7.egg/pypiper/manager.py", line 1632, in _exit_handler self.fail_pipeline(Exception("Pipeline failure. See details above.")) File "/home/huihui.fan/.local/lib/python2.7/site-packages/piper-0.9.4-py2.7.egg/pypiper/manager.py", line 1528, in fail_pipeline raise e Exception: Pipeline failure. See details above.
Pypiper terminating spawned child process 34088...(tee) Thu Mar 7 15:11:20 EST 2019 Done.