search

ctSkennerton / minced

Mining CRISPRs in Environmental Datasets
GNU General Public License v3.0
99 stars 17 forks source link

java.io.FileNotFoundException: <file_name> (Too many open files) Error #37

Open tparket opened 1 year ago

tparket commented 1 year ago

First of all, thank you for your wonderful work. Minced has been very helpful in my recent work and it is much appreciated.

I'm running minced 0.4.2 with the following options on a cluster: minced -gffFull

For some reason, minced crashes with the following error messages (>200,000 times) for each of the output files (the text- and gff-output files which are marked by <>).

java.io.FileNotFoundException: (Too many open files) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:292) at java.base/java.io.FileOutputStream.(FileOutputStream.java:235) at CRISPRFinder.findRepeats(CRISPRFinder.java:293) at CRISPRFinder.goCRISPRFinder(CRISPRFinder.java:138) at minced.main(minced.java:259) --Error writing to file--

java.io.FileNotFoundException: (Too many open files) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:292) at java.base/java.io.FileOutputStream.(FileOutputStream.java:235) at CRISPRFinder.findRepeats(CRISPRFinder.java:279) at CRISPRFinder.goCRISPRFinder(CRISPRFinder.java:138) at minced.main(minced.java:259) --Error writing to file--

I'm afraid that minced is creating too many files without closing them, which results in the exhaustion of the ulimit of allowed open files per user.

I would really appreciate your help.

ctSkennerton commented 1 year ago

Hi there,

Thanks for the bug report. Indeed I think I forgot to close the gff file handle. Can you try using the code in the GH-37 branch to see if it fixes it?

tparket commented 1 year ago

@ctSkennerton it fixed it, thank you!

I also noticed another minor issue - the string "##gff-version 3" is printed many times to the .gff output, making it ~99% of the text in the file. I could easily solve it with - grep '^[^##]' {gff_file} > tmpfile && mv tmpfile {gff_file} - but I figured you might want to take a look at it in the sourcecode.

ctSkennerton commented 1 year ago

could you tell me a bit more about your input? I don't think I've seen minced print the gff version string more than once to the output for an input file.

ctSkennerton commented 1 year ago

Oh, I can reproduce it. Looks like it only happens when outputting to a file rather than stdout. Okay, I'll take a look