Lexogen-Tools / idemux

Idemux is a command line tool designed to demultiplex paired-end fastq files from QuantSeq-Pool.
https://www.lexogen.com/
Other
8 stars 1 forks source link

OSError: [Errno 24] Too many open files #5

Open zihhuafang opened 3 years ago

zihhuafang commented 3 years ago

Hi,

I have a question regarding this particular error for HPC users.

We wanted to run quantseqpool_analysis using the script with the repo here (https://github.com/Lexogen-Tools/quantseqpool_analysis). I ran into the known issue for idemux as too many open files as we pooled 22 libraries which consisted of 2112 samples on a Novaseq S4 flowcell. With the number of samples, I would need to run ulimit -n 5120. For an HPC cluster user, I am unable to increase the limit since I am non-root user (The OS of our cluster is CentOS Linux release 7.9.2009 with max user processes (-u) = 4096). ​

What would be the alternative here? Run idemux per library after demultiplexed i5/i7 with bcl2fastq? Would that be an potential issue for the undetermined reads?

Appreciate your insight. Thanks. Zih-Hia