search

nservant / HiC-Pro

HiC-Pro: An optimized and flexible pipeline for Hi-C data processing
Other
382 stars 183 forks source link

#HiC-Pro3 #samtools error samtools sort: fail to open "***.bam" Too many open files #392

Closed Dawe890311 closed 3 years ago

Dawe890311 commented 3 years ago

Hey, I'm glad to use hicpro3, but when I use it to analysis data like hicpro which based python2, the error [samtools error samtools sort: fail to open "***.bam" Too many open files] occured.

mapping_combine.log /home/ZhangW/miniconda3/bin/samtools merge -@ 20 -n -f bowtie_results/bwt2/9WT/9WT_R1_9311_genome.bwt2merged.bam bowtie_results/bwt2_global/9WT/9WT_R1_9311_genome.bwt2glob.bam bowtie_results/bwt2_local/9WT/9WT_R1_9311_genome.bwt2glob.unmap_bwt2loc.bam /home/ZhangW/miniconda3/bin/samtools merge -@ 20 -n -f bowtie_results/bwt2/9WT/9WT_R2_9311_genome.bwt2merged.bam bowtie_results/bwt2_global/9WT/9WT_R2_9311_genome.bwt2glob.bam bowtie_results/bwt2_local/9WT/9WT_R2_9311_genome.bwt2glob.unmap_bwt2loc.bam /home/ZhangW/miniconda3/bin/samtools sort -@ 20 -m 38M -n -T tmp/9WT_R1_9311_genome -o bowtie_results/bwt2/9WT/9WT_R1_9311_genome.bwt2merged.sorted.bam bowtie_results/bwt2/9WT/9WT_R1_9311_genome.bwt2merged.bam /home/ZhangW/miniconda3/bin/samtools sort -@ 20 -m 38M -n -T tmp/9WT_R2_9311_genome -o bowtie_results/bwt2/9WT/9WT_R2_9311_genome.bwt2merged.sorted.bam bowtie_results/bwt2/9WT/9WT_R2_9311_genome.bwt2merged.bam [bam_sort_core] merging from 2520 files and 20 in-memory blocks... [E::hts_open_format] Failed to open file "tmp/9WT_R1_9311_genome.1020.bam" : Too many open files samtools sort: fail to open "tmp/9WT_R1_9311_genome.1020.bam": Too many open files [bam_sort_core] merging from 2580 files and 20 in-memory blocks... [E::hts_open_format] Failed to open file "tmp/9WT_R2_9311_genome.1020.bam" : Too many open files samtools sort: fail to open "tmp/9WT_R2_9311_genome.1020.bam": Too many open files

mapping_stats.log /home/ZhangW/miniconda3/bin/samtools view -c bowtie_results/bwt2/9WT/9WT_R2_9311_genome.bwt2merged.bam /home/ZhangW/miniconda3/bin/samtools view -c bowtie_results/bwt2/9WT/9WT_R1_9311_genome.bwt2merged.bam /home/ZhangW/miniconda3/bin/samtools view -c -F 4 bowtie_results/bwt2/9WT/9WT_R1_9311_genome.bwt2merged.bam /home/ZhangW/miniconda3/bin/samtools view -c -F 4 bowtie_results/bwt2/9WT/9WT_R2_9311_genome.bwt2merged.bam /home/ZhangW/miniconda3/bin/samtools view -c -F 4 bowtie_results/bwt2_global/9WT/9WT_R1_9311_genome.bwt2glob.bam /home/ZhangW/miniconda3/bin/samtools view -c -F 4 bowtie_results/bwt2_global/9WT/9WT_R2_9311_genome.bwt2glob.bam /home/ZhangW/miniconda3/bin/samtools view -c -F 4 bowtie_results/bwt2_local/9WT/9WT_R1_9311_genome.bwt2glob.unmap_bwt2loc.bam /home/ZhangW/miniconda3/bin/samtools view -c -F 4 bowtie_results/bwt2_local/9WT/9WT_R2_9311_genome.bwt2glob.unmap_bwt2loc.bam

mergeSAM.log /home/ZhangW/miniconda3/bin/python /data1/usr/ZhangW/workspace/hic/software/HiCPro3/HiC-Pro_3.0.0/scripts/mergeSAM.py -q 10 -t -v -f bowtie_results/bwt2/9WT/9WT_R1_9311_genome.bwt2merged.bam -r bowtie_results/bwt2/9WT/9WT_R2_9311_genome.bwt2merged.bam -o bowtie_results/bwt2/9WT/9WT_9311_genome.bwt2pairs.bam [E::idx_find_and_load] Could not retrieve index file for 'bowtie_results/bwt2/9WT/9WT_R1_9311_genome.bwt2merged.bam' [E::idx_find_and_load] Could not retrieve index file for 'bowtie_results/bwt2/9WT/9WT_R2_9311_genome.bwt2merged.bam'

mergeBAM.py

forward= bowtie_results/bwt2/9WT/9WT_R1_9311_genome.bwt2merged.bam

reverse= bowtie_results/bwt2/9WT/9WT_R2_9311_genome.bwt2merged.bam

output= bowtie_results/bwt2/9WT/9WT_9311_genome.bwt2pairs.bam

min mapq= 10

report_single= False

report_multi= False

verbose= True

Merging forward and reverse tags ...

Forward and reverse reads not paired. Check that BAM files have the same read names and are sorted.

AND the nohup file Run HiC-Pro 3.0.0

2020年 12月 26日 星期六 20:06:19 CST Bowtie2 alignment step1 ... Logs: logs/9WT/mapping_step1.log

2020年 12月 27日 星期日 01:27:25 CST Bowtie2 alignment step2 ... Logs: logs/9WT/mapping_step2.log

2020年 12月 27日 星期日 02:38:32 CST Combine R1/R2 alignment files ... Logs: logs/9WT/mapping_combine.log

2020年 12月 27日 星期日 02:50:02 CST Mapping statistics for R1 and R2 tags ... Logs: logs/9WT/mapping_stats.log

2020年 12月 27日 星期日 03:04:29 CST Pairing of R1 and R2 tags ... Logs: logs/9WT/mergeSAM.log make: *** [bowtie_pairing] 错误 1 9311 Hic-Pro3 is done

nservant commented 3 years ago

Hi

I think this is related to the RAM available for samtools. We recently fixed a bug with the SORT_RAM parameter, which is now divided by the number of threads. So here, I guess you have something like 768M of RAM, divided by 20 CPUs, which means 38M of RAM per thread. My feeling is that this is too low, and may explain the error.

So you should change the SORT_RAMparameter. For instance, let's say that you want to use 500M of RAM per thread, you should fix SORT_RAM=10000M Of course, be sure that you have enough memory. Then, be aware that the value should be in M (with the M at the end).

Hope it's clear Nicolas

Dawe890311 commented 3 years ago

Hi I did what you said, you are right, the program works perfectly, thank you, Merry Christmas🎄