How to solve the ‘Too many open files’ problem

[shenlinyong]

My CPU version HiCPro reported an error：

Fri Jul 16 02:13:33 EDT 2021
Mapping statistics for R1 and R2 tags ...
Logs: logs/fat1/mapping_stats.log
Logs: logs/fat2/mapping_stats.log
Logs: logs/fat3/mapping_stats.log
Logs: logs/lean1/mapping_stats.log
Logs: logs/lean2/mapping_stats.log
Logs: logs/lean3/mapping_stats.log
--------------------------------------------
Fri Jul 16 04:11:58 EDT 2021
Pairing of R1 and R2 tags ...
Logs: logs/fat1/mergeSAM.log
make: *** [HiC-Pro_3.0.0/bin/../scripts//Makefile:144: bowtie_pairing] Error 1

I found by checking pipline that it was caused by "Too many open files", how should I modify it.

/home/SLY68/anaconda3/envs/hicpro/bin/samtools sort -@ 80 -m 9M -n -T tmp/AF224727_R1_galGal6 -o bowtie_results/bwt2/*/*bwt2merged.sorted.bam bowtie_results/bwt2/*/*.bwt2merged.bam
[bam_sort_core] merging from 23920 files and 80 in-memory blocks...
[E::hts_open_format] Failed to open file tmp*.1020.bam
samtools sort: fail to open "tmp/*.bam": Too many open files
[bam_sort_core] merging from 24080 files and 80 in-memory blocks...
[E::hts_open_format] Failed to open file tmp/*.1020.bam
samtools sort: fail to open "tmp/*.1020.bam": Too many open files

[nservant]

Hi, This is samtools issue. I guess you fix the SORT_RAM parameter to 9M in the HiC-Pro config. Thus, samtools is run with 9M of RAM, and is swapping a lot with many small files ... I think that increasing the SORT_RAM option to 1000M for instance, should fix the issue N

[yanchunzhang]

Hi, This is samtools issue. I guess you fix the SORT_RAM parameter to 9M in the HiC-Pro config. Thus, samtools is run with 9M of RAM, and is swapping a lot with many small files ... I think that increasing the SORT_RAM option to 1000M for instance, should fix the issue N

Hi, Where should I set the SORT_RAM option in the configure file? I didn't see it in the documentation. Thanks, Yanchun