tx.close() causing "FileNotFound exception: Too many open files in system"

[jerdeb]

I am noticing that over 10K transactions, the tx.close() function is throwing a "FileNotFound exception...: (Too many open files in system)". My initial thought is that lucene files are not being closed after a transection, but I could be wrong.

I am running my test application locally using a Mac (10.9.5) and Eclipse Kepler.

Code:

public void compute(Quad quad){
        try ( Transaction tx = graphDb.beginTx() )
        {
               // check if quad.subject exists as a node and return it, else create a new one
               // check if quad.object exists as a node and return it, else create a new one
               // create relationship between quad.subject and quad.object
               tx.success();
               tx.close(); // added this just to check if this would actually work, but made no difference
        }
}

I am using maven and the Neo4j version I'm using is 2.1.5.

Message.log can be found here: https://gist.github.com/jerdeb/410b00b7f67cfa0e0d7a

For testing purposes, I can also provide access to the current code as it can be found in a public github.

[lutovich]

Hello @jerdeb, how many indexes do you have? Can you please ensure that max number of opened file descriptors is set to >= 40K, as described in manual: http://neo4j.com/docs/stable/linux-performance-guide.html#_setting_the_number_of_open_files

[jerdeb]

I only have one index, defined this way:

    private static IndexDefinition indexDefinition;
    private static Label value_Label = DynamicLabel.label("Value");
    static{
        registerShutdownHook(graphDb);

        try ( Transaction tx = graphDb.beginTx() )
        {
            Schema schema = graphDb.schema();
            indexDefinition = schema.indexFor(value_Label).on("value").create();
            tx.success();
        }

        try ( Transaction tx = graphDb.beginTx() )
        {
            Schema schema = graphDb.schema();
            schema.awaitIndexOnline( indexDefinition, 10, TimeUnit.SECONDS );
        }
    }

With regard to the max number of opened file descriptors, how should I ensure that anyone using the project does not need to set this descriptor?

[lutovich]

I assume tweak of opened files can be included in shell script that launches the app. Under high insert load Lucene opens many small files with different information in them and after some time merges them into one. Peak usage of descriptors might benefit from: https://github.com/neo4j/neo4j/pull/3302

[jerdeb]

I have set the descriptors to 40K as advised by http://neo4j.com/docs/stable/linux-performance-guide.html#_setting_the_number_of_open_files , but this still caused errors. Could it be due to the reason that when checking ulimit -n, it still gave me 256?

[jexp]

Yes, that could be def. the reason. Please check your unix system documentation for upping the file limits. Perhaps something is different there.

[jerdeb]

Ok, I've updated the limits (following: http://unix.stackexchange.com/questions/108174/how-to-persist-ulimit-settings-in-osx-mavericks -> maybe this could be added to the documentation for completeness?). But this still gave that exception.

[lutovich]

@jerdeb, messages.log contains Max number of file descriptors: some_x >= 40K under Operating system information section during startup, right? It would be great if you could provide code that reveals the problem.

[jerdeb]

The code revealing the problem can be found here: https://gist.github.com/jerdeb/9cff012383a0f3239fa4 (line 101)

Strangely enough, even though I've updated the limited, the message.log still gave the following:

    Operating System: Mac OS X; version: 10.9.5; arch: x86_64; cpus: 4
    Max number of file descriptors: 10240
    Number of open file descriptors: 158

When i do the following: ulimit -n it gives 16384

green-tea:~ jeremy$ ulimit -n
50000

Whilst in /etc/security/limits.conf I have the following: neo4j soft nofile 40000 neo4j hard nofile 40000

I am starting to think that this is something related to the OS now...

Edit: I've updated the Mac file limit to 50000 - still same results

[lutovich]

@jerdeb, sorry for a long pause. I've tried many ways to up the number of file descriptors on mac but messages.log indeed always shows 10240. Not sure if this is MBean issue or smth else. We'll look into this problem.

It would also be amazing if you could extract a simple example that reproduces the problem. Thanks!

[klaren]

According to the internet:

By default, the maximum number of files that Mac OS X can open is set to 12,288 and 
the maximum number of files a given process can open is 10,240.

How to actually change the maximum value differs from each major version, a semi-complete list can be found in the answers here: https://superuser.com/questions/433746/is-there-a-fix-for-the-too-many-open-files-in-system-error-on-os-x-10-7-1