Closed yingsun-ucsd closed 7 years ago
It looks like Picard tools failed to create temporary files on your $TMPDIR
and /tmp
.
No space left on device
How did you set environment variables ($TMPDIR
and $TMP
) for temporary directories? Can you check if you have enough space on them?
$ env | grep TMP
Yes, you are right. After adding "export TMPDIR=/home/ysun/tmp/" to .bash_profile, it works as expected. Thanks.
Hi,
I am using atac_dnase_pipelines for both DNase-seq and ATAC-seq. And DNase-seq is running perfectly without any issue. Unfortunately, I tried several times on my ATAC-seq data but never got it run successfully. Here is the log as well as the html file. If anyone can give any advise. I will really appreciate it1.
Please REMOVE ".txt" before open it since the website dose not accept html file. [===========================================================] atac.bds.20170904_140521_807.report.html.txt
nohup: ignoring input Picked up _JAVA_OPTIONS: -Xms256M -Xmx728M -XX:ParallelGCThreads=1
== git info Latest git commit : f23d942ddb4e56961ecb33cb32baf9a5f084d883 (Thu Aug 17 15:19:29 2017) Reading parameters from section (default) in file(*****/atac_dnase_pipelines/default.env)...
== configuration file info Hostname : **** Configuration file : Environment file : /atac_dnase_pipelines/default.env
== parallelization info No parallel jobs : false Maximum # threads : 12
== cluster/system info Walltime (general) : 5h50m Max. memory (general) : 7G Force to use a system : local Process priority (niceness) : 0 Retiral for failed tasks : 0 Submit tasks to a cluster queue : Unlimited cluster mem./walltime : false
== shell environment info Conda env. : bds_atac Conda env. for python3 : bds_atac_py3 Conda bin. directory :
Shell cmd. for init. : if [[ -f $(which conda) && $(conda env list | grep bds_atac | wc -l) != "0" ]]; then source activate bds_atac; sleep 5; fi; export PATH=/h ome/ysun/atac_dnase_pipelines/.:/atac_dnase_pipelines/modules:/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipef ail; STARTTIME=$(date +%s)
Shell cmd. for init.(py3) : if [[ -f $(which conda) && $(conda env list | grep bds_atac_py3 | wc -l) != "0" ]]; then source activate bds_atac_py3; sleep 5; fi; export PATH=/atac_dnase_pipelines/.:/atac_dnase_pipelines/modules:*****/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)
Shell cmd. for fin. : TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."; sleep 0
Cluster task min. len. : 60
Cluster task delay : 0
== output directory/title info Output dir. : /home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell Title (prefix) : Q363alphaCell Reading parameters from section (default) in file(/home/ysun/atac_dnase_pipelines/default.env)... Reading parameters from section (hg38) in file(/home/ysun/bds_pipeline_genome_data/bds_atac_species.conf)...
== species settings Species : hg38 Species file : /home/ysun/bds_pipeline_genome_data/bds_atac_species.conf
Species name (WashU browser) : hg38 Ref. genome seq. fasta : /home/ysun/bds_pipeline_genome_data/hg38/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta Chr. sizes file : /home/ysun/bds_pipeline_genome_data/hg38/hg38.chrom.sizes Black list bed : /home/ysun/bds_pipeline_genome_data/hg38/hg38.blacklist.bed.gz Ref. genome seq. dir. :
== ENCODE accession settings ENCODE experiment accession : ENCODE award RFA : ENCODE assay category : ENCODE assay title : ENCODE award : ENCODE lab : ENCODE assembly genome : ENCODE alias prefix : KLAB_PIPELINE ENCODE alias suffix :
== report settings URL root for output directory : Genome coord. for browser tracks :
== align multimapping settings
alignments reported for multimapping : 0
== align bowtie2 settings Bowtie2 index : /home/ysun/bds_pipeline_genome_data/hg38/bowtie2_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta Replacement --score-min for bowtie2 : Walltime (bowtie2) : 47h Max. memory (bowtie2) : 12G Extra param. (bowtie2) :
== adapter trimmer settings Maximum allowed error rate for cutadapt : 0.10 Minimum trim. length for cutadapt -m : 5 Walltime (adapter trimming) : 23h Max. memory (adapter trimming) : 12G
== postalign bam settings MAPQ reads rm thresh. : 30 Rm. tag reads with str. : No dupe removal in filtering raw bam : false Walltime (bam filter) : 23h Max. memory (bam filter) : 12G Dup marker : picard Use sambamba markdup (instead of picard) : false
== postalign bed/tagalign settings Max. memory for UNIX shuf : 12G
== postalign cross-corr. analysis settings Max. memory for UNIX shuf : 12G User-defined cross-corr. peak strandshift : -1 Extra parameters for cross-corr. analysis :
== callpeak macs2 settings Genome size (hs,mm) : hs Walltime (macs2) : 23h Max. memory (macs2) : 15G Extra parameters for macs2 callpeak :
== callpeak naiver overlap settings Bedtools intersect -nonamecheck : false
== callpeak etc settings
of top peaks to pick up in peak files : 500000
== IDR settings Append IDR threshold to IDR out_dir : false
== ATAQC settings TSS enrichment bed : /home/ysun/bds_pipeline_genome_data/hg38/ataqc/hg38_gencode_tss_unique.bed.gz DNase bed for ataqc : /home/ysun/bds_pipeline_genome_data/hg38/ataqc/reg2map_honeybadger2_dnase_all_p10_ucsc.hg19_to_hg38.bed.gz Promoter bed for ataqc : /home/ysun/bds_pipeline_genome_data/hg38/ataqc/reg2map_honeybadger2_dnase_prom_p2.hg19_to_hg38.bed.gz Enhancer bed for ataqc : /home/ysun/bds_pipeline_genome_data/hg38/ataqc/reg2map_honeybadger2_dnase_enh_p2.hg19_to_hg38.bed.gz Reg2map for ataqc : /home/ysun/bds_pipeline_genome_data/hg38/ataqc/hg38_dnase_avg_fseq_signal_formatted.txt.gz Reg2map_bed for ataqc : /home/ysun/bds_pipeline_genome_data/hg38/ataqc/hg38_celltype_compare_subsample.bed.gz Roadmap metadata for ataqc : /home/ysun/bds_pipeline_genome_data/hg38/ataqc/hg38_dnase_avg_fseq_signal_metadata.txt Max. memory for ATAQC : 20G Walltime for ATAQC : 47h
== atac pipeline settings Type of pipeline : atac-seq Fastqs are trimmed? : true Align only : false
reads to subsample replicates (0 if no subsampling) : 0
reads to subsample for cross-corr. analysis : 25000000
No pseudo replicates : false No ATAQC (advanced QC report) : false No Cross-corr. analysis : false Use CSEM for alignment : false Smoothing window for MACS2 : 150 DNase Seq : false IDR threshold : 0.1 Use old trim adapters : false Force to use ENCODE3 parameter set : false Force to use ENCODE parameter set : false Disable genome browser tracks : false p-val thresh. for overlapped peaks : 0.01 MACS2 p-val thresh. for peaks : 0.01 MACS2 p-val thresh. for BIGWIGs : 0.01 Enable IDR on called peaks : false
== checking atac parameters ...
Checking parameters and data files for ATAQC.
== checking adapters to be trimmed ...
== checking input files ...
Rep1 fastq (PE) : /home/ysun/published/GEO/GSE76268_Ackerman_MM/SRR3048039_1.fastq.gz /home/ysun/published/GEO/GSE76268_Ackerman_MM/SRR3048039_2.fastq.gz Rep2 fastq (PE) : /home/ysun/published/GEO/GSE76268_Ackerman_MM/SRR3048040_1.fastq.gz /home/ysun/published/GEO/GSE76268_Ackerman_MM/SRR3048040_2.fastq.gz Distributing 12 to ... {1=7, 2=5} Picked up _JAVA_OPTIONS: -Xms256M -Xmx12G -XX:ParallelGCThreads=1 [Mon Sep 04 14:05:38 PDT 2017] picard.sam.markduplicates.MarkDuplicates INPUT=[/home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep1/SRR3048039_1.PE2SE.filt .bam] OUTPUT=/home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep1/SRR3048039_1.PE2SE.dupmark.bam METRICS_FILE=/home/ysun/published/GEO/GSE76268_Ackerman_MM /Q363alphaCell/qc/rep1/SRR3048039_1.PE2SE.dup.qc REMOVE_DUPLICATES=false ASSUME_SORTED=true VALIDATION_STRINGENCY=LENIENT MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_F ILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates DUPLICATE_SCORING_STRATEGY=SUM_OF_BA SE_QUALITIES READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAXRECORDS IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false [Mon Sep 04 14:05:38 PDT 2017] Executing as ysun@gatsby.ucsd.edu on Linux 3.10.0-514.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_131-b12; Picard version: 1.126(4691ee61 1ac205d4afe2a1b7a2ea975a6f997426_1417447214) IntelDeflater INFO 2017-09-04 14:05:38 MarkDuplicates Start of doWork freeMemory: 254859848; totalMemory: 257425408; maxMemory: 11453595648 INFO 2017-09-04 14:05:38 MarkDuplicates Reading input file and constructing read end information. INFO 2017-09-04 14:05:38 MarkDuplicates Will retain up to 44052290 data points before spilling to disk. [Mon Sep 04 14:05:38 PDT 2017] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=434110464 To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp Exception in thread "main" htsjdk.samtools.SAMException: Exception creating temporary directory. at htsjdk.samtools.util.IOUtil.createTempDir(IOUtil.java:677) at htsjdk.samtools.CoordinateSortedPairInfoMap.(CoordinateSortedPairInfoMap.java:59)
at picard.sam.markduplicates.util.DiskBasedReadEndsForMarkDuplicatesMap.(DiskBasedReadEndsForMarkDuplicatesMap.java:57)
at picard.sam.markduplicates.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:250)
at picard.sam.markduplicates.MarkDuplicates.doWork(MarkDuplicates.java:114)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:187)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:89)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:99)
Caused by: java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at java.io.File.createTempFile(File.java:2070)
at htsjdk.samtools.util.IOUtil.createTempDir(IOUtil.java:668)
... 7 more
Task failed:
Program & line : '/home/ysun/atac_dnase_pipelines/modules/postalign_bam.bds', line 391
Task Name : 'markdup_bam_picard rep1'
Task ID : 'atac.bds.20170904_140521_807_parallel_41/task.postalign_bam.markdup_bam_picard_rep1.line_391.id_11'
Task PID : '29452'
Task hint : 'if [ -d "${TMPDIR}" ]; then; fi; if [ -f "${PICARDROOT}/picard.jar" ]; then; MARKDUP="${PICARDROOT}/picard.jar MarkDuplicates";; elif [ -f "${P
ICARDROOT}/MarkDuplicates.jar" ]; then; MARKDUP="${PICARDROOT}/MarkDuplicates.jar";; elif [ $(which picard 2> /dev/null | wc -l || echo) == "1" ]; then; MARK'
Task resources : 'cpus: 1 mem: -1.0 B wall-timeout: 8640000'
State : 'ERROR'
Dependency state : 'ERROR'
Retries available : '1'
Input files : '[/home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep1/SRR3048039_1.PE2SE.filt.bam]'
Output files : '[/home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep1/SRR3048039_1.PE2SE.dupmark.bam, /home/ysun/published/GEO/GSE76268_Acker
man_MM/Q363alphaCell/qc/rep1/SRR3048039_1.PE2SE.dup.qc]'
Script file : '/home/ysun/published/GEO/GSE76268_Ackerman_MM/atac.bds.20170904_140521_807_parallel_41/task.postalign_bam.markdup_bam_picard_rep1.line_391.id_1
1.sh'
Exit status : '1'
Program :
se_pipelines/.:/home/ysun/atac_dnase_pipelines/modules:/home/ysun/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$( date +%s)
/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep1/SRR3048039_1.PE2SE.dupmark.bam" \ METRICS_FILE="/home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/qc/rep1/SRR3048039_1.PE2SE.dup.qc" VALIDATION_STRIN GENCY=LENIENT \ ASSUME_SORTED=true REMOVE_DUPLICATES=false; \ else \ picard MarkDuplicates \ INPUT="/home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep1/SRR3048039_1.PE2SE.filt.bam" OUTPUT="/home/ysun /published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep1/SRR3048039_1.PE2SE.dupmark.bam" \ METRICS_FILE="/home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/qc/rep1/SRR3048039_1.PE2SE.dup.qc" VALIDATION_STRIN GENCY=LENIENT \ ASSUME_SORTED=true REMOVE_DUPLICATES=false; \ fi
039_1.PE2SE.filt.bam] OUTPUT=/home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep1/SRR3048039_1.PE2SE.dupmark.bam METRICS_FILE=/home/ysun/published/GEO/GSE7 6268_Ackerman_MM/Q363alphaCell/qc/rep1/SRR3048039_1.PE2SE.dup.qc REMOVE_DUPLICATES=false ASSUME_SORTED=true VALIDATION_STRINGENCY=LENIENT MAX_SEQUENCES_FOR_DISK_READ_ENDS _MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates DUPLICATE_SCORING_ST RATEGY=SUM_OF_BASE_QUALITIES READ_NAME_REGEX=[a-zA-Z0-9]+:[0-9]:([0-9]+):([0-9]+):([0-9]+).* OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 VERBOSITY=INFO QUIET=false COMPRESSION_LEVE L=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false [Mon Sep 04 14:05:38 PDT 2017] Executing as ysun@gatsby.ucsd.edu on Linux 3.10.0-514.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_131-b12; Picard version : 1.126(4691ee611ac205d4afe2a1b7a2ea975a6f997426_1417447214) IntelDeflater INFO 2017-09-04 14:05:38 MarkDuplicates Start of doWork freeMemory: 254859848; totalMemory: 257425408; maxMemory: 11453595648 INFO 2017-09-04 14:05:38 MarkDuplicates Reading input file and constructing read end information. INFO 2017-09-04 14:05:38 MarkDuplicates Will retain up to 44052290 data points before spilling to disk. [Mon Sep 04 14:05:38 PDT 2017] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=434110464 To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp Exception in thread "main" htsjdk.samtools.SAMException: Exception creating temporary directory. at htsjdk.samtools.util.IOUtil.createTempDir(IOUtil.java:677) at htsjdk.samtools.CoordinateSortedPairInfoMap.(CoordinateSortedPairInfoMap.java:59)
at picard.sam.markduplicates.util.DiskBasedReadEndsForMarkDuplicatesMap.(DiskBasedReadEndsForMarkDuplicatesMap.java:57)
at picard.sam.markduplicates.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:250)
at picard.sam.markduplicates.MarkDuplicates.doWork(MarkDuplicates.java:114)
at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:187)
at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:89)
at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:99)
Caused by: java.io.IOException: No space left on device
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at java.io.File.createTempFile(File.java:2070)
at htsjdk.samtools.util.IOUtil.createTempDir(IOUtil.java:668)
... 7 more
Fatal error: /home/ysun/atac_dnase_pipelines/modules/postalign_bam.bds, line 313, pos 4. Task/s failed.
Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done. /home/ysun/published/GEO/GSE76268_Ackerman_MM/atac.bds.20170904_140521_807_parallel_42/task.postalign_bam.nmsrt_bam_rep2.line_507.id_10.sh: line 11: 29586 Segmentation fault sambamba sort -t 1 -n /home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep2/SRR3048040_1.PE2SE.nodup.bam -o /home/ysun/published/GEO/GSE76268_Ackerman _MM/Q363alphaCell/align/rep2/SRR3048040_1.PE2SE.nodup.nmsrt.bam Task failed: Program & line : '/home/ysun/atac_dnase_pipelines/modules/postalign_bam.bds', line 507 Task Name : 'nmsrt_bam rep2' Task ID : 'atac.bds.20170904_140521_807_parallel_42/task.postalign_bam.nmsrt_bam_rep2.line_507.id_10' Task PID : '29447' Task hint : 'sambamba sort -t 1 -n /home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep2/SRR3048040_1.PE2SE.nodup.bam -o /home/ysun/publish ed/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep2/SRR3048040_1.PE2SE.nodup.nmsrt.bam; TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TAS' Task resources : 'cpus: -1 mem: -1.0 B wall-timeout: 8640000' State : 'ERROR' Dependency state : 'ERROR' Retries available : '1' Input files : '[/home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep2/SRR3048040_1.PE2SE.nodup.bam]' Output files : '[/home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep2/SRR3048040_1.PE2SE.nodup.nmsrt.bam]' Script file : '/home/ysun/published/GEO/GSE76268_Ackerman_MM/atac.bds.20170904_140521_807_parallel_42/task.postalign_bam.nmsrt_bam_rep2.line_507.id_10.sh' Exit status : '1' Program :
se_pipelines/.:/home/ysun/atac_dnase_pipelines/modules:/home/ysun/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$( date +%s)
68_Ackerman_MM/Q363alphaCell/align/rep2/SRR3048040_1.PE2SE.nodup.nmsrt.bam
gmentation fault sambamba sort -t 1 -n /home/ysun/published/GEO/GSE76268_Ackerman_MM/Q363alphaCell/align/rep2/SRR3048040_1.PE2SE.nodup.bam -o /home/ysun/published/GEO/G SE76268_Ackerman_MM/Q363alphaCell/align/rep2/SRR3048040_1.PE2SE.nodup.nmsrt.bam
Fatal error: /home/ysun/atac_dnase_pipelines/modules/postalign_bam.bds, line 608, pos 3. Task/s failed.
Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done. Fatal error: /home/ysun/atac_dnase_pipelines/atac.bds, line 393, pos 2. Task/s failed. atac.bds, line 79 : main() atac.bds, line 82 : void main() { // atac pipeline starts here atac.bds, line 92 : do_align() atac.bds, line 360 : void do_align() { atac.bds, line 393 : wait
Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done.