search

karel-brinda / rnftools

RNF framework for NGS: simulation of reads, evaluation of mappers, conversion of RNF-compliant data.
http://karel-brinda.github.io/rnftools
MIT License
14 stars 5 forks source link

FqMerger - too many open files #32

Open karel-brinda opened 8 years ago

karel-brinda commented 8 years ago

Limit number of concurrently open files in the merging step of read simulation.

Error in job 42 while creating output file simulation.fq.
RuleException:
OSError in line 36 of /home/karel/anaconda3/lib/python3.4/site-packages/rnftools/mishmash/mishmash.snake:
[Errno 24] Too many open files: 'simulation/951/_final_reads.fq'
  File "/home/karel/anaconda3/lib/python3.4/site-packages/rnftools/mishmash/mishmash.snake", line 36, in __None
  File "/home/karel/anaconda3/lib/python3.4/site-packages/rnftools/mishmash/Sample.py", line 81, in create_fq
  File "/home/karel/anaconda3/lib/python3.4/site-packages/rnftools/rnfformat/FqMerger.py", line 37, in __init__
  File "/home/karel/anaconda3/lib/python3.4/site-packages/rnftools/rnfformat/FqMerger.py", line 37, in <listcomp>
karel-brinda commented 8 years ago

How to get to know the limit:

import resource

# the soft limit imposed by the current configuration
# the hard limit imposed by the operating system.
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print 'Soft limit is ', soft 

# For the following line to run, you need to execute the Python script as root.
resource.setrlimit(resource.RLIMIT_NOFILE, (3000, hard))
karel-brinda commented 8 years ago

On my Linux machine it still does not work:

...
Error in job 42 while creating output files test_metagenomic_pe.1.fq, test_metagenomic_pe.2.fq.
   RuleException:
   OSError in line 36 of /home/karel/anaconda3/lib/python3.4/site-packages/rnftools/mishmash/mishmash.snake:
   [Errno 24] Too many open files: 'test_metagenomic_pe.2.fq'
   Removing output files of failed job 42 since they might be corrupted:
   test_metagenomic_pe.1.fq
   Will exit after finishing currently running jobs.
   Exiting because a job execution failed. Look above for error message