search

ghpaetzold / questplusplus

Pipelined quality estimation.
49 stars 14 forks source link

FileNotFoundException: Too many open files #25

Open varvara-l opened 8 years ago

varvara-l commented 8 years ago

I'm getting the following error when trying to extract the full blackbox feature set from a big (>2000 lines) file.

\ Producing output ** java.io.FileNotFoundException: input/active_learning/english/2000.source.en.cased.tok.pos (Too many open files) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileInputStream.(FileInputStream.java:93) at java.io.FileReader.(FileReader.java:58) at shef.mt.tools.POSTaggerProcessor.processNextSentence(Unknown Source) at shef.mt.SentenceLevelFeatureExtractor.run(Unknown Source) at shef.mt.SentenceLevelFeatureExtractor.main(Unknown Source) java.io.FileNotFoundException: input/active_learning/spanish/2000.target.es.cased.tok.pos (Too many open files) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileInputStream.(FileInputStream.java:93) at java.io.FileReader.(FileReader.java:58) at shef.mt.tools.POSTaggerProcessor.processNextSentence(Unknown Source) at shef.mt.SentenceLevelFeatureExtractor.run(Unknown Source) at shef.mt.SentenceLevelFeatureExtractor.main(Unknown Source) Exception in thread "main" java.lang.ClassCastException: java.lang.Float cannot be cast to java.lang.Integer at shef.mt.features.impl.bb.Feature1083.run(Unknown Source) at shef.mt.features.util.FeatureManager.runFeatures(Unknown Source) at shef.mt.SentenceLevelFeatureExtractor.run(Unknown Source) at shef.mt.SentenceLevelFeatureExtractor.main(Unknown Source)

What could be the reason of that? I've just tried with a file with 2140 lines and feature extraction failed, then just removed the last 140 lines - and this error didn't occur. That doesn't happen with 17 baseline feature set.

fredblain commented 8 years ago

Hi, usually this error occurs due to system limitations (manageable at both system and shell levels) so we need to investigate on how many files are opened at the same time during this process (not only the 2k input files but also the ones produced by the tool at intermediate steps I suppose).

varvara-l commented 8 years ago

I bet there are places in the code where files are opened and never closed. Maybe @ghpaetzold could check that?

ghpaetzold commented 8 years ago

Whenever I find some time to fix this issue, I will look into it. :)

Gustavo Henrique PaetzoldPh.D. Candidate in Computer ScienceUniversity of Sheffield

Date: Tue, 2 Feb 2016 04:28:38 -0800 From: notifications@github.com To: questplusplus@noreply.github.com CC: ghpaetzold@outlook.com Subject: Re: [questplusplus] FileNotFoundException: Too many open files (#25)

I bet there are places in the code where files are opened and never closed.

Maybe @ghpaetzold could check that?

— Reply to this email directly or view it on GitHub.

fredblain commented 8 years ago

In POSTaggerProcessor.java, a buffer reader is created each time the function processNextSentence() is called but never closed, which explains why the "too many open files" exception is raised if a (too) large amount of input sentences are given. By closing the buffer reader with the close() function, this issue does not occur anymore. Nevertheless, we need to investigate the potential side effects of this fix before pushing it on the repo.