search

davidemms / OrthoFinder

Phylogenetic orthology inference for comparative genomics
https://davidemms.github.io/
GNU General Public License v3.0
715 stars 189 forks source link

IOError: [Errno 24] Too many open files #384

Closed davidemms closed 4 years ago

davidemms commented 4 years ago

Issue If the system limits on the number of files that a user can open is too low then OrthoFinder is unable to open all the required files and it fails late in the run. Instead it should fail immediately so that the user can adjust the limits and thus fix the problem immediately.

Error message The error produced if not enough files can be opened is of the form:

Reconciling gene trees and species tree
---------------------------------------
Outgroup: Anncaliia_algerae_pra109.Annc_alge_PRA109_V4.pep.all
2020-05-07 09:26:57 : Starting Recon and orthologues
2020-05-07 09:26:57 : Starting OF Orthologues
Traceback (most recent call last):
  File "/home/emms/workspace/git/OrthoFinder/orthofinder.py", line 7, in <module>
    main(args)
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/__main__.py", line 1664, in main
    GetOrthologues(speciesInfoObj, options, prog_caller)   
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/__main__.py", line 1511, in GetOrthologues
    options.name)
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/orthologues.py", line 1020, in OrthologuesWorkflow
    ReconciliationAndOrthologues(recon_method, db.ogSet, nHighParallel, i if qMultiple else None, all_stride_dup_genes=all_stride_dup_genes) 
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/orthologues.py", line 818, in ReconciliationAndOrthologues
    nOrthologues_SpPair = trees2ologs_of.DoOrthologuesForOrthoFinder(ogSet, speciesTree_ids_fn, trees2ologs_of.GeneToSpecies_dash, all_stride_dup_genes, qNoRecon)
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/trees2ologs_of.py", line 631, in DoOrthologuesForOrthoFinder
    with open(files.FileHandler.GetDuplicationsFN(), csv_write_mode) as outfile, OrthologsFiles(dResultsOrthologues, speciesDict, ogSet.speciesToUse, nspecies, sp_to_index) as (ortholog_file_writers, suspect_genes_file_writers):
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/trees2ologs_of.py", line 583, in __enter__
    self.ortholog_file_handles[j][i] = open(d1 + '%s__v__%s.tsv' % (self.speciesDict[sp1], self.speciesDict[sp0]), csv_write_mode)
IOError: [Errno 24] Too many open files: '/data/NOBACKUP/open_files/33_proteomes/OrthoFinder/Results_May06_11/../Results_May07_5/Orthologues/Orthologues_Botrytis_cinerea.ASM83294v1.pep.all/Botrytis_cinerea.ASM83294v1.pep.all__v__Blastomyces_dermatitidis_atcc_26199.BD_ATCC26199_V2.pep.all.tsv'

Or, potentially:

Reconciling gene and species trees
----------------------------------
2020-05-07 09:13:42 : Starting OF Orthologues
Traceback (most recent call last):
  File "/home/emms/workspace/git/OrthoFinder/orthofinder.py", line 7, in <module>
    main(args)
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/__main__.py", line 1713, in main
    GetOrthologues_FromTrees(options)
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/__main__.py", line 1515, in GetOrthologues_FromTrees
    orthologues.OrthologuesFromTrees(options.recon_method, options.nBlast, options.speciesTreeFN, options.qAddSpeciesToIDs)
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/orthologues.py", line 842, in OrthologuesFromTrees
    ReconciliationAndOrthologues(recon_method, ogSet, nHighParallel)
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/orthologues.py", line 818, in ReconciliationAndOrthologues
    nOrthologues_SpPair = trees2ologs_of.DoOrthologuesForOrthoFinder(ogSet, speciesTree_ids_fn, trees2ologs_of.GeneToSpecies_dash, all_stride_dup_genes, qNoRecon)
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/trees2ologs_of.py", line 635, in DoOrthologuesForOrthoFinder
    rooted_tree_ids, qHaveSupport = CheckAndRootTree(files.FileHandler.GetOGsTreeFN(iog), species_tree_rooted, GeneToSpecies) # this can be parallelised easily
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/trees2ologs_of.py", line 325, in CheckAndRootTree
    tree = tree_lib.Tree(treeFN, format=3)
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/tree.py", line 221, in __init__
    read_newick(newick, root_node = self, format=format)
  File "/home/emms/workspace/git/OrthoFinder/scripts_of/newick.py", line 208, in read_newick
    nw = open(newick, 'rU').read()
IOError: [Errno 24] Too many open files: '/data/NOBACKUP/open_files/33_proteomes/OrthoFinder/Results_May06/../Results_May06_11/WorkingDirectory/Trees_ids/OG0000000_tree_id.txt'

Work around If this occurs then take the following steps:

  1. Increase the system limit number of files the user can open
  2. Restarted OrthoFinder from the point it reached using the '-ft' (='from trees') restart option. E.g.: python orthofinder.py -ft OrthoFinder/Results_May06_11/

(In the updated version of OrthoFinder this problem will be identified right at the start rather than at the "Reconciling gene and species trees" stage. It will print a message detailing the steps that need to be taken. If this occurs you will need to update the limits and rerun the original command rather than use the '-ft' option.)

davidemms commented 4 years ago

Updated version In the updated version OrthoFinder will detect the issue at the start of a run and print a message more or less the same as the steps described here:

If you are running an analysis on n species then OrthoFinder will need to be able to open approximately r=n^2+50 files. The n-squared files are the .tsv files of orthologs that are updated with the orthologs as each new gene tree is analysed. The 50 is for extra files opened by the process. As I don't understand the full intricacies of how how many files linux opens to run a process I'd suggest being safe and using a number considerably bigger than 50.

I'll use r=1000000 in the steps below

  1. Check the hard and soft limits on the number of open files for your system with the commands: ulimit -Hn ulimit -Sn
  2. If hard limit, h > r already, then you just need to increase the soft limit: ulimit -n 1000000
  3. Alternatively, if h < r then you need to edit the file /etc/security/limits.conf, this requires root privileges. To increase the limit to 1000000 for user called 'emms' add the lines: emms hard nofile 1000000 emms soft nofile 1000000 (edit these lines to match your username)
  4. Check the limit has now been updated (if you changed the hard limit you'll need to open a new session and confirm it's updated): ulimit -Sn
  5. Once the limit is updated restart OrthoFinder with the original command or, if the error occurred at the stage "Reconciling gene and species trees" then use the option '-ft PREVIOUS_RESULTS_DIRECTORY' to restart 'from trees' instead of '-f' or '-b' (which starts from fasta files or blast results respectively).
davidemms commented 4 years ago

There's also lots of information online about ulimit & adjusting the number of open files limit: https://www.google.co.uk/search?q=linux+ulimit+nofile

davidemms commented 4 years ago

Update:

I've recently found cases where the above changes on linux don't initially work and calling ulimit -n continues to show the old limit rather than the new, higher limit. This looks like it's an operating system error which should hopefully be resolved at some point. A workaround for me was to use the linux su command to 'switch' user to myself even though the current user on the terminal was already me. For some reason this works to update the nofile limit when things like restarting the computer didn't.

My username is 'emms' so I ran su emms and provided my password at the prompt. Then when I called ulimit -n I saw that the limit had been successfully updated.

TaehyungKwon commented 4 years ago

@davidemms Hi David, I was having the open limit error on our cluster, but my ulimit commands give back a soft and hard limit of 131072. _ERROR: The system limits on the number of files a process can open is too low. For 214 species OrthoFinder needs to be able t o open at least r=45896 files. Please increase the limit and restart OrthoFinder

  1. Check the hard and soft limits on the number of open files for your system: $ ulimit -Hn $ ulimit -Sn
  2. If hard limit, h > r already, then you just need to increase the soft limit: $ ulimit -n 45896
  3. Alternatively, if h < r then you need to edit the file '/etc/security/limits.conf', this requires root privileges. To incr ease the limit to 45896 for user called 'emms' add the lines: emms hard nofile 45896 emms soft nofile 45896 (edit these lines to match your username)
  4. Check the limit has now been updated (if you changed the hard limit you'll need to open a new session and confirm it's upd ated): $ ulimit -Sn
  5. Once the limit is updated restart OrthoFinder with the original command_

The administrator answered that they never changed the open limit. What could cause this problem?

Thanks,

TaehyungKwon commented 3 years ago

FIXED. I fixed the problem with ulimit. This problem comes with qsub command sge module, applying different file open limits from the one assigned to my account. I managed to run it with a direct command run.

XK4959 commented 3 years ago

Hi，David I encountered the same error, but I don’t have administrator rights. Is there any way to run the software with the maximum number of open files of 1024?

ERROR: The system limits on the number of files a process can open is too low. For 176 species OrthoFinder needs to be able to open at least r=31076 files. Please increase the limit and restart OrthoFinder

  1. Check the hard and soft limits on the number of open files for your system: $ ulimit -Hn $ ulimit -Sn
  2. If hard limit, h > r already, then you just need to increase the soft limit: $ ulimit -n 31076
  3. Alternatively, if h < r then you need to edit the file '/etc/security/limits.conf', this requires root privileges. To increase the limit to 31076 for user called 'emms' add the lines: emms hard nofile 31076 emms soft nofile 31076 (edit these lines to match your username)
  4. Check the limit has now been updated (if you changed the hard limit you'll need to open a new session and confirm it's updated): $ ulimit -Sn
  5. Once the limit is updated restart OrthoFinder with the original command

Xinlong

jolespin commented 2 years ago

Here's the error: 

ERROR: The system limits on the number of files a process can open is too low. For 33 species OrthoFinder needs to be able to open at least r=1189 files. Please increase the limit and restart OrthoFinder
1. Check the hard and soft limits on the number of open files for your system:
    $ ulimit -Hn
    $ ulimit -Sn
2. If hard limit, h > r already, then you just need to increase the soft limit:
    $ ulimit -n 1189
3. Alternatively, if h < r then you need to edit the file '/etc/security/limits.conf', this requires root privileges. To increase the limit to 1189 for user  called 'emms' add the lines:
    emms hard nofile 1189
    emms soft nofile 1189
    (edit these lines to match your username)
4. Check the limit has now been updated (if you changed the hard limit you'll need to open a new session and confirm it's updated):
    $ ulimit -Sn
5. Once the limit is updated restart OrthoFinder with the original command

For full details see: https://github.com/davidemms/OrthoFinder/issues/384

I'm developing a pipeline that uses orthofinder as one of its step and will be run via qsub on our grid. I had a few questions:

  1. Is r specific to my run or is it hardcoded for all users as an example?
  2. Is h the hard limit or soft limit?
  3. Is ulimit -Hn the hard limit and ulimit -Sn the soft limit? 
    bash-4.2$  ulimit -Hn
4096 # Is this the hard limit?
bash-4.2$ ulimit -Sn
1024 # Is this the soft limit?
  4. Is it possible to determine the number of files needed to open before running and then set the ulimit -n ?
  5. Does ulimit -n 1189 reset after starting a new ssh session?
thiagogenez commented 2 years ago

For those running Orthofinder on CentOS Linux release 7.9.2009 (Core) with a demand of r>1024*1024 (=1048576), a previous step is required before editing the file /etc/security/limits.conf as root. The number you set in /etc/security/limits.conf for nofile (number of open files) cannot be greater than the value found in /proc/sys/fs/nr_open [2]

[thiagogenez@login001 ~]$ cat /proc/sys/fs/nr_open
1048576
[thiagogenez@login001 ~]$ ulimit -Hn
4096
[thiagogenez@login001 ~]$ ulimit -Sn
1024

To increase the number in /proc/sys/fs/nr_open to, for instance, 1186021 > 1048576, run the following command line with root privilegies:

[centos@login001 ~]$ sudo sysctl -w fs.nr_open=1186021
fs.nr_open = 1186021

Then, edit the file /etc/security/limits.conf with root privileges:

thiagogenez hard nofile 1186021
thiagogenez soft nofile 1186021

Finally checking the results:

[thiagogenez@login001 ~]$ ulimit -Sn
1186021
[thiagogenez@login001 ~]$ ulimit -Hn
1186021

PS: I haven't rebooted the machine to check if the value in /proc/sys/fs/nr_open remains intact.

Sources: [1] https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ [2] https://serverfault.com/questions/583820/cant-log-in-when-nofile-is-set-to-unlimited-in-etc-security-limits-conf