search

jprante / elasticsearch-jdbc

JDBC importer for Elasticsearch
Apache License 2.0
2.84k stars 709 forks source link

java.io.IOException: Too many open files #881

Open gvsrini opened 8 years ago

gvsrini commented 8 years ago

I have been running JDBC Importer consistently and I get this error after every 1 or 2 days of running it.

My task is configured to run every 5 minutes. Here is the stack trace. Please help.

org.jboss.netty.channel.ChannelException: Failed to create a selector.
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:362) ~[netty-3.10.5.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100) ~[netty-3.10.5.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52) ~[netty-3.10.5.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45) ~[netty-3.10.5.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44) ~[netty-3.10.5.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28) ~[netty-3.10.5.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80) ~[netty-3.10.5.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39) ~[netty-3.10.5.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33) ~[netty-3.10.5.Final.jar:?]
        at org.elasticsearch.transport.netty.NettyTransport.createClientBootstrap(NettyTransport.java:346) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.transport.netty.NettyTransport.doStart(NettyTransport.java:279) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:68) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.transport.TransportService.doStart(TransportService.java:182) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:68) ~[elasticsearch-2.3.4.jar:2.3.4]
        at org.xbib.elasticsearch.helper.client.TransportClient$Builder.build(TransportClient.java:139) ~[elasticsearch-helper-2.3.4.0.jar:?]
        at org.xbib.elasticsearch.helper.client.BaseTransportClient.createClient(BaseTransportClient.java:73) ~[elasticsearch-helper-2.3.4.0.jar:?]
        at org.xbib.elasticsearch.helper.client.BaseMetricTransportClient.init(BaseMetricTransportClient.java:40) ~[elasticsearch-helper-2.3.4.0.jar:?]
        at org.xbib.elasticsearch.helper.client.BulkTransportClient.init(BulkTransportClient.java:96) ~[elasticsearch-helper-2.3.4.0.jar:?]
        at org.xbib.elasticsearch.helper.client.ClientBuilder.toBulkTransportClient(ClientBuilder.java:113) ~[elasticsearch-helper-2.3.4.0.jar:?]
        at org.xbib.elasticsearch.jdbc.strategy.standard.StandardSink.createClient(StandardSink.java:348) ~[elasticsearch-jdbc-2.3.4.0.jar:?]
        at org.xbib.elasticsearch.jdbc.strategy.standard.StandardSink.beforeFetch(StandardSink.java:100) ~[elasticsearch-jdbc-2.3.4.0.jar:?]
        at org.xbib.elasticsearch.jdbc.strategy.standard.StandardContext.beforeFetch(StandardContext.java:183) ~[elasticsearch-jdbc-2.3.4.0.jar:?]
        at org.xbib.elasticsearch.jdbc.strategy.standard.StandardContext.execute(StandardContext.java:164) ~[elasticsearch-jdbc-2.3.4.0.jar:?]
        at org.xbib.tools.JDBCImporter.process(JDBCImporter.java:199) ~[elasticsearch-jdbc-2.3.4.0.jar:?]
        at org.xbib.tools.JDBCImporter.newRequest(JDBCImporter.java:185) [elasticsearch-jdbc-2.3.4.0.jar:?]
        at org.xbib.tools.JDBCImporter.newRequest(JDBCImporter.java:51) [elasticsearch-jdbc-2.3.4.0.jar:?]
        at org.xbib.pipeline.AbstractPipeline.call(AbstractPipeline.java:50) [elasticsearch-jdbc-2.3.4.0.jar:?]
        at org.xbib.pipeline.AbstractPipeline.call(AbstractPipeline.java:16) [elasticsearch-jdbc-2.3.4.0.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
Caused by: java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method) ~[?:1.8.0_60]
        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65) ~[?:1.8.0_60]
        at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) ~[?:1.8.0_60]
        at java.nio.channels.Selector.open(Selector.java:227) ~[?:1.8.0_60]
        at org.jboss.netty.channel.socket.nio.SelectorUtil.open(SelectorUtil.java:63) ~[netty-3.10.5.Final.jar:?]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:360) ~[netty-3.10.5.Final.jar:?]
        ... 31 more
jprante commented 8 years ago

I just pushed version 2.3.4.1, can you please try if it improves the situation?

gvsrini commented 8 years ago

Hi,

I cloned and built the repository using: ./gradlew clean build

I have attached snapshots of process files at three time intervals 11:25 and 11:38 AM and 11:51 AM. Initially, it showed 76 files (11:25 AM) , then 106 (11:38 AM) and then 136 files (11:48 AM) and then 151 files (11:51 AM). There seems to be a File leak somewhere.

Surprisingly, time stamps of all files changed at 11:51 whereas in the first two snapshots, that did not happen.

The jar is still showing as 2.3.4.0. So, I am not sure if this is the latest version I built.

Here is my Git log

[user@srini elasticsearch-jdbc]$ git log|head -20
commit 7ec2dde68d023cb1498907e30bbb07764d527824
Author: Jörg Prante <joergprante@gmail.com>
Date:   Sun Aug 28 15:19:48 2016 +0200

    2.3.4.1

commit 153692505251d5f8b84146e83e3b4a66614d473b
Author: Jörg Prante <joergprante@gmail.com>
Date:   Sun Aug 28 15:19:25 2016 +0200

    moving executor service management to JDBCImporter

commit 37f3c5a36cad9bf45afe323efec2c54066013726
Author: Jörg Prante <joergprante@gmail.com>
Date:   Sun Aug 28 15:18:55 2016 +0200

    update to Gradle 3

commit 072c82386e1dcd64ceed4284b68bbd3c740a4c61
Author: Jörg Prante <joergprante@gmail.com>

Jar file size:

-rw-r--r-- 1 user user 129248 Aug 29 11:06 elasticsearch-jdbc-2.3.4.0.jar

File list at 11:25 AM

[user@srini lib]$ ls -l /proc/6598/fd
total 0
l-wx------ 1 user user 64 Aug 29 11:23 0 -> /dev/null
l-wx------ 1 user user 64 Aug 29 11:23 1 -> /home/user/elasticsearch-jdbc/bin/nohup.out
lr-x------ 1 user user 64 Aug 29 11:23 10 -> /home/user/elasticsearch-jdbc/lib/lucene-core-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 11 -> /home/user/elasticsearch-jdbc/lib/lucene-backward-codecs-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 12 -> /home/user/elasticsearch-jdbc/lib/lucene-analyzers-common-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 13 -> /home/user/elasticsearch-jdbc/lib/lucene-queries-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 14 -> /home/user/elasticsearch-jdbc/lib/lucene-memory-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 15 -> /home/user/elasticsearch-jdbc/lib/lucene-highlighter-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 16 -> /home/user/elasticsearch-jdbc/lib/lucene-queryparser-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 17 -> /home/user/elasticsearch-jdbc/lib/lucene-suggest-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 18 -> /home/user/elasticsearch-jdbc/lib/lucene-join-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 19 -> /home/user/elasticsearch-jdbc/lib/lucene-spatial-5.5.0.jar
l-wx------ 1 user user 64 Aug 29 11:22 2 -> /home/user/elasticsearch-jdbc/bin/nohup.out
lr-x------ 1 user user 64 Aug 29 11:23 20 -> /home/user/elasticsearch-jdbc/lib/guava-18.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 21 -> /home/user/elasticsearch-jdbc/lib/securesm-1.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 22 -> /home/user/elasticsearch-jdbc/lib/hppc-0.7.1.jar
lr-x------ 1 user user 64 Aug 29 11:23 23 -> /home/user/elasticsearch-jdbc/lib/joda-time-2.9.4.jar
lr-x------ 1 user user 64 Aug 29 11:23 24 -> /home/user/elasticsearch-jdbc/lib/joda-convert-1.2.jar
lr-x------ 1 user user 64 Aug 29 11:23 25 -> /home/user/elasticsearch-jdbc/lib/jackson-core-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:23 26 -> /home/user/elasticsearch-jdbc/lib/jackson-dataformat-smile-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:23 27 -> /home/user/elasticsearch-jdbc/lib/jackson-dataformat-yaml-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:23 28 -> /home/user/elasticsearch-jdbc/lib/jackson-dataformat-cbor-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:23 29 -> /home/user/elasticsearch-jdbc/lib/netty-3.10.5.Final.jar
lr-x------ 1 user user 64 Aug 29 11:23 3 -> /usr/java/jdk1.8.0_60/jre/lib/rt.jar
lr-x------ 1 user user 64 Aug 29 11:23 30 -> /home/user/elasticsearch-jdbc/lib/compress-lzf-1.0.2.jar
lr-x------ 1 user user 64 Aug 29 11:23 31 -> /home/user/elasticsearch-jdbc/lib/t-digest-3.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 32 -> /home/user/elasticsearch-jdbc/lib/HdrHistogram-2.1.6.jar
lr-x------ 1 user user 64 Aug 29 11:23 33 -> /home/user/elasticsearch-jdbc/lib/commons-cli-1.3.1.jar
lr-x------ 1 user user 64 Aug 29 11:23 34 -> /home/user/elasticsearch-jdbc/lib/jsr166e-1.1.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 35 -> /home/user/elasticsearch-jdbc/lib/slf4j-api-1.7.12.jar
lr-x------ 1 user user 64 Aug 29 11:23 36 -> /home/user/elasticsearch-jdbc/lib/log4j-api-2.5.jar
lr-x------ 1 user user 64 Aug 29 11:23 37 -> /home/user/elasticsearch-jdbc/lib/lucene-sandbox-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 38 -> /home/user/elasticsearch-jdbc/lib/lucene-misc-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 39 -> /home/user/elasticsearch-jdbc/lib/lucene-grouping-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 4 -> /home/user/elasticsearch-jdbc/lib/elasticsearch-jdbc-2.3.4.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 40 -> /home/user/elasticsearch-jdbc/lib/lucene-spatial3d-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 41 -> /home/user/elasticsearch-jdbc/lib/spatial4j-0.5.jar
lr-x------ 1 user user 64 Aug 29 11:23 42 -> /home/user/elasticsearch-jdbc/lib/snakeyaml-1.15.jar
lr-x------ 1 user user 64 Aug 29 11:23 43 -> /home/user/elasticsearch-jdbc/lib/jdbc-driver-csv-1.0.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 44 -> /home/user/elasticsearch-jdbc/lib/derby-10.12.1.1.jar
lr-x------ 1 user user 64 Aug 29 11:23 45 -> /home/user/elasticsearch-jdbc/lib/h2-1.4.191.jar
lr-x------ 1 user user 64 Aug 29 11:23 46 -> /home/user/elasticsearch-jdbc/lib/hsqldb-2.3.3.jar
lr-x------ 1 user user 64 Aug 29 11:23 47 -> /home/user/elasticsearch-jdbc/lib/mysql-connector-java-5.1.38.jar
lr-x------ 1 user user 64 Aug 29 11:23 48 -> /home/user/elasticsearch-jdbc/lib/postgresql-9.4.1208.jre7.jar
lr-x------ 1 user user 64 Aug 29 11:23 49 -> /home/user/elasticsearch-jdbc/lib/sqljdbc42.jar
lr-x------ 1 user user 64 Aug 29 11:23 5 -> /home/user/elasticsearch-jdbc/lib/elasticsearch-2.3.4.jar
lr-x------ 1 user user 64 Aug 29 11:23 50 -> /usr/java/jdk1.8.0_60/jre/lib/ext/nashorn.jar
l-wx------ 1 user user 64 Aug 29 11:23 51 -> /home/user/elasticsearch-jdbc/bin/logs/jdbc.log
lr-x------ 1 user user 64 Aug 29 11:23 52 -> /usr/java/jdk1.8.0_60/jre/lib/jsse.jar
lr-x------ 1 user user 64 Aug 29 11:23 53 -> /dev/random
lr-x------ 1 user user 64 Aug 29 11:23 54 -> /dev/urandom
lr-x------ 1 user user 64 Aug 29 11:23 55 -> /dev/random
lr-x------ 1 user user 64 Aug 29 11:23 56 -> /dev/random
lr-x------ 1 user user 64 Aug 29 11:23 57 -> /dev/urandom
lr-x------ 1 user user 64 Aug 29 11:23 58 -> /dev/urandom
lr-x------ 1 user user 64 Aug 29 11:25 59 -> pipe:[328515000]
lr-x------ 1 user user 64 Aug 29 11:23 6 -> /home/user/elasticsearch-jdbc/lib/elasticsearch-helper-2.3.4.0.jar
l-wx------ 1 user user 64 Aug 29 11:25 60 -> pipe:[328515000]
lrwx------ 1 user user 64 Aug 29 11:25 61 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:25 62 -> pipe:[328515001]
l-wx------ 1 user user 64 Aug 29 11:25 63 -> pipe:[328515001]
lrwx------ 1 user user 64 Aug 29 11:25 64 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:25 65 -> pipe:[328515002]
l-wx------ 1 user user 64 Aug 29 11:25 66 -> pipe:[328515002]
lrwx------ 1 user user 64 Aug 29 11:25 67 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:25 68 -> pipe:[328515003]
l-wx------ 1 user user 64 Aug 29 11:25 69 -> pipe:[328515003]
lr-x------ 1 user user 64 Aug 29 11:23 7 -> /home/user/elasticsearch-jdbc/lib/log4j-slf4j-impl-2.5.jar
lrwx------ 1 user user 64 Aug 29 11:25 70 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:25 71 -> pipe:[328515004]
l-wx------ 1 user user 64 Aug 29 11:25 72 -> pipe:[328515004]
lrwx------ 1 user user 64 Aug 29 11:25 73 -> anon_inode:[eventpoll]
lrwx------ 1 user user 64 Aug 29 11:25 75 -> socket:[328515010]
lr-x------ 1 user user 64 Aug 29 11:23 8 -> /home/user/elasticsearch-jdbc/lib/log4j-core-2.5.jar
lr-x------ 1 user user 64 Aug 29 11:23 9 -> /home/user/elasticsearch-jdbc/lib/jts-1.13.jar

File list at 11:38 AM

[user@srini lib]$ ls -l /proc/6598/fd|wc -l
106
[user@srini lib]$ ls -l /proc/6598/fd
total 0
l-wx------ 1 user user 64 Aug 29 11:23 0 -> /dev/null
l-wx------ 1 user user 64 Aug 29 11:23 1 -> /home/user/elasticsearch-jdbc/bin/nohup.out
lr-x------ 1 user user 64 Aug 29 11:23 10 -> /home/user/elasticsearch-jdbc/lib/lucene-core-5.5.0.jar
l-wx------ 1 user user 64 Aug 29 11:38 100 -> pipe:[328561902]
lrwx------ 1 user user 64 Aug 29 11:38 101 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:38 102 -> pipe:[328563563]
l-wx------ 1 user user 64 Aug 29 11:38 103 -> pipe:[328563563]
lrwx------ 1 user user 64 Aug 29 11:38 104 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:23 11 -> /home/user/elasticsearch-jdbc/lib/lucene-backward-codecs-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 12 -> /home/user/elasticsearch-jdbc/lib/lucene-analyzers-common-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 13 -> /home/user/elasticsearch-jdbc/lib/lucene-queries-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 14 -> /home/user/elasticsearch-jdbc/lib/lucene-memory-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 15 -> /home/user/elasticsearch-jdbc/lib/lucene-highlighter-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 16 -> /home/user/elasticsearch-jdbc/lib/lucene-queryparser-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 17 -> /home/user/elasticsearch-jdbc/lib/lucene-suggest-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 18 -> /home/user/elasticsearch-jdbc/lib/lucene-join-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 19 -> /home/user/elasticsearch-jdbc/lib/lucene-spatial-5.5.0.jar
l-wx------ 1 user user 64 Aug 29 11:22 2 -> /home/user/elasticsearch-jdbc/bin/nohup.out
lr-x------ 1 user user 64 Aug 29 11:23 20 -> /home/user/elasticsearch-jdbc/lib/guava-18.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 21 -> /home/user/elasticsearch-jdbc/lib/securesm-1.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 22 -> /home/user/elasticsearch-jdbc/lib/hppc-0.7.1.jar
lr-x------ 1 user user 64 Aug 29 11:23 23 -> /home/user/elasticsearch-jdbc/lib/joda-time-2.9.4.jar
lr-x------ 1 user user 64 Aug 29 11:23 24 -> /home/user/elasticsearch-jdbc/lib/joda-convert-1.2.jar
lr-x------ 1 user user 64 Aug 29 11:23 25 -> /home/user/elasticsearch-jdbc/lib/jackson-core-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:23 26 -> /home/user/elasticsearch-jdbc/lib/jackson-dataformat-smile-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:23 27 -> /home/user/elasticsearch-jdbc/lib/jackson-dataformat-yaml-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:23 28 -> /home/user/elasticsearch-jdbc/lib/jackson-dataformat-cbor-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:23 29 -> /home/user/elasticsearch-jdbc/lib/netty-3.10.5.Final.jar
lr-x------ 1 user user 64 Aug 29 11:23 3 -> /usr/java/jdk1.8.0_60/jre/lib/rt.jar
lr-x------ 1 user user 64 Aug 29 11:23 30 -> /home/user/elasticsearch-jdbc/lib/compress-lzf-1.0.2.jar
lr-x------ 1 user user 64 Aug 29 11:23 31 -> /home/user/elasticsearch-jdbc/lib/t-digest-3.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 32 -> /home/user/elasticsearch-jdbc/lib/HdrHistogram-2.1.6.jar
lr-x------ 1 user user 64 Aug 29 11:23 33 -> /home/user/elasticsearch-jdbc/lib/commons-cli-1.3.1.jar
lr-x------ 1 user user 64 Aug 29 11:23 34 -> /home/user/elasticsearch-jdbc/lib/jsr166e-1.1.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 35 -> /home/user/elasticsearch-jdbc/lib/slf4j-api-1.7.12.jar
lr-x------ 1 user user 64 Aug 29 11:23 36 -> /home/user/elasticsearch-jdbc/lib/log4j-api-2.5.jar
lr-x------ 1 user user 64 Aug 29 11:23 37 -> /home/user/elasticsearch-jdbc/lib/lucene-sandbox-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 38 -> /home/user/elasticsearch-jdbc/lib/lucene-misc-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 39 -> /home/user/elasticsearch-jdbc/lib/lucene-grouping-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 4 -> /home/user/elasticsearch-jdbc/lib/elasticsearch-jdbc-2.3.4.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 40 -> /home/user/elasticsearch-jdbc/lib/lucene-spatial3d-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 41 -> /home/user/elasticsearch-jdbc/lib/spatial4j-0.5.jar
lr-x------ 1 user user 64 Aug 29 11:23 42 -> /home/user/elasticsearch-jdbc/lib/snakeyaml-1.15.jar
lr-x------ 1 user user 64 Aug 29 11:23 43 -> /home/user/elasticsearch-jdbc/lib/jdbc-driver-csv-1.0.0.jar
lr-x------ 1 user user 64 Aug 29 11:23 44 -> /home/user/elasticsearch-jdbc/lib/derby-10.12.1.1.jar
lr-x------ 1 user user 64 Aug 29 11:23 45 -> /home/user/elasticsearch-jdbc/lib/h2-1.4.191.jar
lr-x------ 1 user user 64 Aug 29 11:23 46 -> /home/user/elasticsearch-jdbc/lib/hsqldb-2.3.3.jar
lr-x------ 1 user user 64 Aug 29 11:23 47 -> /home/user/elasticsearch-jdbc/lib/mysql-connector-java-5.1.38.jar
lr-x------ 1 user user 64 Aug 29 11:23 48 -> /home/user/elasticsearch-jdbc/lib/postgresql-9.4.1208.jre7.jar
lr-x------ 1 user user 64 Aug 29 11:23 49 -> /home/user/elasticsearch-jdbc/lib/sqljdbc42.jar
lr-x------ 1 user user 64 Aug 29 11:23 5 -> /home/user/elasticsearch-jdbc/lib/elasticsearch-2.3.4.jar
lr-x------ 1 user user 64 Aug 29 11:23 50 -> /usr/java/jdk1.8.0_60/jre/lib/ext/nashorn.jar
l-wx------ 1 user user 64 Aug 29 11:23 51 -> /home/user/elasticsearch-jdbc/bin/logs/jdbc.log
lr-x------ 1 user user 64 Aug 29 11:23 52 -> /usr/java/jdk1.8.0_60/jre/lib/jsse.jar
lr-x------ 1 user user 64 Aug 29 11:23 53 -> /dev/random
lr-x------ 1 user user 64 Aug 29 11:23 54 -> /dev/urandom
lr-x------ 1 user user 64 Aug 29 11:23 55 -> /dev/random
lr-x------ 1 user user 64 Aug 29 11:23 56 -> /dev/random
lr-x------ 1 user user 64 Aug 29 11:23 57 -> /dev/urandom
lr-x------ 1 user user 64 Aug 29 11:23 58 -> /dev/urandom
lr-x------ 1 user user 64 Aug 29 11:25 59 -> pipe:[328515000]
lr-x------ 1 user user 64 Aug 29 11:23 6 -> /home/user/elasticsearch-jdbc/lib/elasticsearch-helper-2.3.4.0.jar
l-wx------ 1 user user 64 Aug 29 11:25 60 -> pipe:[328515000]
lrwx------ 1 user user 64 Aug 29 11:25 61 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:25 62 -> pipe:[328515001]
l-wx------ 1 user user 64 Aug 29 11:25 63 -> pipe:[328515001]
lrwx------ 1 user user 64 Aug 29 11:25 64 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:25 65 -> pipe:[328515002]
l-wx------ 1 user user 64 Aug 29 11:25 66 -> pipe:[328515002]
lrwx------ 1 user user 64 Aug 29 11:25 67 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:25 68 -> pipe:[328515003]
l-wx------ 1 user user 64 Aug 29 11:25 69 -> pipe:[328515003]
lr-x------ 1 user user 64 Aug 29 11:23 7 -> /home/user/elasticsearch-jdbc/lib/log4j-slf4j-impl-2.5.jar
lrwx------ 1 user user 64 Aug 29 11:25 70 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:25 71 -> pipe:[328515004]
l-wx------ 1 user user 64 Aug 29 11:25 72 -> pipe:[328515004]
lrwx------ 1 user user 64 Aug 29 11:25 73 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:38 74 -> pipe:[328537438]
lrwx------ 1 user user 64 Aug 29 11:25 75 -> socket:[328515010]
l-wx------ 1 user user 64 Aug 29 11:38 76 -> pipe:[328537438]
lrwx------ 1 user user 64 Aug 29 11:38 77 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:38 78 -> pipe:[328537439]
l-wx------ 1 user user 64 Aug 29 11:38 79 -> pipe:[328537439]
lr-x------ 1 user user 64 Aug 29 11:23 8 -> /home/user/elasticsearch-jdbc/lib/log4j-core-2.5.jar
lrwx------ 1 user user 64 Aug 29 11:38 80 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:38 81 -> pipe:[328537440]
l-wx------ 1 user user 64 Aug 29 11:38 82 -> pipe:[328537440]
lrwx------ 1 user user 64 Aug 29 11:38 83 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:38 84 -> pipe:[328538745]
l-wx------ 1 user user 64 Aug 29 11:38 85 -> pipe:[328538745]
lrwx------ 1 user user 64 Aug 29 11:38 86 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:38 87 -> pipe:[328538746]
l-wx------ 1 user user 64 Aug 29 11:38 88 -> pipe:[328538746]
lrwx------ 1 user user 64 Aug 29 11:38 89 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:23 9 -> /home/user/elasticsearch-jdbc/lib/jts-1.13.jar
lr-x------ 1 user user 64 Aug 29 11:38 90 -> pipe:[328561899]
l-wx------ 1 user user 64 Aug 29 11:38 91 -> pipe:[328561899]
lrwx------ 1 user user 64 Aug 29 11:38 92 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:38 93 -> pipe:[328561900]
l-wx------ 1 user user 64 Aug 29 11:38 94 -> pipe:[328561900]
lrwx------ 1 user user 64 Aug 29 11:38 95 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:38 96 -> pipe:[328561901]
l-wx------ 1 user user 64 Aug 29 11:38 97 -> pipe:[328561901]
lrwx------ 1 user user 64 Aug 29 11:38 98 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:38 99 -> pipe:[328561902]

File list at 11:51

[user@srini lib]$ ls -l /proc/6598/fd|wc -l
151
[user@srini lib]$ ls -l /proc/6598/fd
total 0
l-wx------ 1 user user 64 Aug 29 11:51 0 -> /dev/null
l-wx------ 1 user user 64 Aug 29 11:51 1 -> /home/user/elasticsearch-jdbc/bin/nohup.out
lr-x------ 1 user user 64 Aug 29 11:51 10 -> /home/user/elasticsearch-jdbc/lib/lucene-core-5.5.0.jar
l-wx------ 1 user user 64 Aug 29 11:51 100 -> pipe:[328561902]
lrwx------ 1 user user 64 Aug 29 11:51 101 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 102 -> pipe:[328563563]
l-wx------ 1 user user 64 Aug 29 11:51 103 -> pipe:[328563563]
lrwx------ 1 user user 64 Aug 29 11:51 104 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 105 -> pipe:[328586090]
l-wx------ 1 user user 64 Aug 29 11:51 106 -> pipe:[328586090]
lrwx------ 1 user user 64 Aug 29 11:51 107 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 108 -> pipe:[328586091]
l-wx------ 1 user user 64 Aug 29 11:51 109 -> pipe:[328586091]
lr-x------ 1 user user 64 Aug 29 11:51 11 -> /home/user/elasticsearch-jdbc/lib/lucene-backward-codecs-5.5.0.jar
lrwx------ 1 user user 64 Aug 29 11:51 110 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 111 -> pipe:[328586092]
l-wx------ 1 user user 64 Aug 29 11:51 112 -> pipe:[328586092]
lrwx------ 1 user user 64 Aug 29 11:51 113 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 114 -> pipe:[328586093]
l-wx------ 1 user user 64 Aug 29 11:51 115 -> pipe:[328586093]
lrwx------ 1 user user 64 Aug 29 11:51 116 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 117 -> pipe:[328586094]
l-wx------ 1 user user 64 Aug 29 11:51 118 -> pipe:[328586094]
lrwx------ 1 user user 64 Aug 29 11:51 119 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 12 -> /home/user/elasticsearch-jdbc/lib/lucene-analyzers-common-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 120 -> pipe:[328609481]
l-wx------ 1 user user 64 Aug 29 11:51 121 -> pipe:[328609481]
lrwx------ 1 user user 64 Aug 29 11:51 122 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 123 -> pipe:[328609482]
l-wx------ 1 user user 64 Aug 29 11:51 124 -> pipe:[328609482]
lrwx------ 1 user user 64 Aug 29 11:51 125 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 126 -> pipe:[328609483]
l-wx------ 1 user user 64 Aug 29 11:51 127 -> pipe:[328609483]
lrwx------ 1 user user 64 Aug 29 11:51 128 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 129 -> pipe:[328609484]
lr-x------ 1 user user 64 Aug 29 11:51 13 -> /home/user/elasticsearch-jdbc/lib/lucene-queries-5.5.0.jar
l-wx------ 1 user user 64 Aug 29 11:51 130 -> pipe:[328609484]
lrwx------ 1 user user 64 Aug 29 11:51 131 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 132 -> pipe:[328609485]
l-wx------ 1 user user 64 Aug 29 11:51 133 -> pipe:[328609485]
lrwx------ 1 user user 64 Aug 29 11:51 134 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 135 -> pipe:[328634490]
l-wx------ 1 user user 64 Aug 29 11:51 136 -> pipe:[328634490]
lrwx------ 1 user user 64 Aug 29 11:51 137 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 138 -> pipe:[328634491]
l-wx------ 1 user user 64 Aug 29 11:51 139 -> pipe:[328634491]
lr-x------ 1 user user 64 Aug 29 11:51 14 -> /home/user/elasticsearch-jdbc/lib/lucene-memory-5.5.0.jar
lrwx------ 1 user user 64 Aug 29 11:51 140 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 141 -> pipe:[328634492]
l-wx------ 1 user user 64 Aug 29 11:51 142 -> pipe:[328634492]
lrwx------ 1 user user 64 Aug 29 11:51 143 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 144 -> pipe:[328634493]
l-wx------ 1 user user 64 Aug 29 11:51 145 -> pipe:[328634493]
lrwx------ 1 user user 64 Aug 29 11:51 146 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 147 -> pipe:[328634494]
l-wx------ 1 user user 64 Aug 29 11:51 148 -> pipe:[328634494]
lrwx------ 1 user user 64 Aug 29 11:51 149 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 15 -> /home/user/elasticsearch-jdbc/lib/lucene-highlighter-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 16 -> /home/user/elasticsearch-jdbc/lib/lucene-queryparser-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 17 -> /home/user/elasticsearch-jdbc/lib/lucene-suggest-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 18 -> /home/user/elasticsearch-jdbc/lib/lucene-join-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 19 -> /home/user/elasticsearch-jdbc/lib/lucene-spatial-5.5.0.jar
l-wx------ 1 user user 64 Aug 29 11:51 2 -> /home/user/elasticsearch-jdbc/bin/nohup.out
lr-x------ 1 user user 64 Aug 29 11:51 20 -> /home/user/elasticsearch-jdbc/lib/guava-18.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 21 -> /home/user/elasticsearch-jdbc/lib/securesm-1.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 22 -> /home/user/elasticsearch-jdbc/lib/hppc-0.7.1.jar
lr-x------ 1 user user 64 Aug 29 11:51 23 -> /home/user/elasticsearch-jdbc/lib/joda-time-2.9.4.jar
lr-x------ 1 user user 64 Aug 29 11:51 24 -> /home/user/elasticsearch-jdbc/lib/joda-convert-1.2.jar
lr-x------ 1 user user 64 Aug 29 11:51 25 -> /home/user/elasticsearch-jdbc/lib/jackson-core-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:51 26 -> /home/user/elasticsearch-jdbc/lib/jackson-dataformat-smile-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:51 27 -> /home/user/elasticsearch-jdbc/lib/jackson-dataformat-yaml-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:51 28 -> /home/user/elasticsearch-jdbc/lib/jackson-dataformat-cbor-2.6.6.jar
lr-x------ 1 user user 64 Aug 29 11:51 29 -> /home/user/elasticsearch-jdbc/lib/netty-3.10.5.Final.jar
lr-x------ 1 user user 64 Aug 29 11:51 3 -> /usr/java/jdk1.8.0_60/jre/lib/rt.jar
lr-x------ 1 user user 64 Aug 29 11:51 30 -> /home/user/elasticsearch-jdbc/lib/compress-lzf-1.0.2.jar
lr-x------ 1 user user 64 Aug 29 11:51 31 -> /home/user/elasticsearch-jdbc/lib/t-digest-3.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 32 -> /home/user/elasticsearch-jdbc/lib/HdrHistogram-2.1.6.jar
lr-x------ 1 user user 64 Aug 29 11:51 33 -> /home/user/elasticsearch-jdbc/lib/commons-cli-1.3.1.jar
lr-x------ 1 user user 64 Aug 29 11:51 34 -> /home/user/elasticsearch-jdbc/lib/jsr166e-1.1.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 35 -> /home/user/elasticsearch-jdbc/lib/slf4j-api-1.7.12.jar
lr-x------ 1 user user 64 Aug 29 11:51 36 -> /home/user/elasticsearch-jdbc/lib/log4j-api-2.5.jar
lr-x------ 1 user user 64 Aug 29 11:51 37 -> /home/user/elasticsearch-jdbc/lib/lucene-sandbox-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 38 -> /home/user/elasticsearch-jdbc/lib/lucene-misc-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 39 -> /home/user/elasticsearch-jdbc/lib/lucene-grouping-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 4 -> /home/user/elasticsearch-jdbc/lib/elasticsearch-jdbc-2.3.4.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 40 -> /home/user/elasticsearch-jdbc/lib/lucene-spatial3d-5.5.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 41 -> /home/user/elasticsearch-jdbc/lib/spatial4j-0.5.jar
lr-x------ 1 user user 64 Aug 29 11:51 42 -> /home/user/elasticsearch-jdbc/lib/snakeyaml-1.15.jar
lr-x------ 1 user user 64 Aug 29 11:51 43 -> /home/user/elasticsearch-jdbc/lib/jdbc-driver-csv-1.0.0.jar
lr-x------ 1 user user 64 Aug 29 11:51 44 -> /home/user/elasticsearch-jdbc/lib/derby-10.12.1.1.jar
lr-x------ 1 user user 64 Aug 29 11:51 45 -> /home/user/elasticsearch-jdbc/lib/h2-1.4.191.jar
lr-x------ 1 user user 64 Aug 29 11:51 46 -> /home/user/elasticsearch-jdbc/lib/hsqldb-2.3.3.jar
lr-x------ 1 user user 64 Aug 29 11:51 47 -> /home/user/elasticsearch-jdbc/lib/mysql-connector-java-5.1.38.jar
lr-x------ 1 user user 64 Aug 29 11:51 48 -> /home/user/elasticsearch-jdbc/lib/postgresql-9.4.1208.jre7.jar
lr-x------ 1 user user 64 Aug 29 11:51 49 -> /home/user/elasticsearch-jdbc/lib/sqljdbc42.jar
lr-x------ 1 user user 64 Aug 29 11:51 5 -> /home/user/elasticsearch-jdbc/lib/elasticsearch-2.3.4.jar
lr-x------ 1 user user 64 Aug 29 11:51 50 -> /usr/java/jdk1.8.0_60/jre/lib/ext/nashorn.jar
l-wx------ 1 user user 64 Aug 29 11:51 51 -> /home/user/elasticsearch-jdbc/bin/logs/jdbc.log
lr-x------ 1 user user 64 Aug 29 11:51 52 -> /usr/java/jdk1.8.0_60/jre/lib/jsse.jar
lr-x------ 1 user user 64 Aug 29 11:51 53 -> /dev/random
lr-x------ 1 user user 64 Aug 29 11:51 54 -> /dev/urandom
lr-x------ 1 user user 64 Aug 29 11:51 55 -> /dev/random
lr-x------ 1 user user 64 Aug 29 11:51 56 -> /dev/random
lr-x------ 1 user user 64 Aug 29 11:51 57 -> /dev/urandom
lr-x------ 1 user user 64 Aug 29 11:51 58 -> /dev/urandom
lr-x------ 1 user user 64 Aug 29 11:51 59 -> pipe:[328515000]
lr-x------ 1 user user 64 Aug 29 11:51 6 -> /home/user/elasticsearch-jdbc/lib/elasticsearch-helper-2.3.4.0.jar
l-wx------ 1 user user 64 Aug 29 11:51 60 -> pipe:[328515000]
lrwx------ 1 user user 64 Aug 29 11:51 61 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 62 -> pipe:[328515001]
l-wx------ 1 user user 64 Aug 29 11:51 63 -> pipe:[328515001]
lrwx------ 1 user user 64 Aug 29 11:51 64 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 65 -> pipe:[328515002]
l-wx------ 1 user user 64 Aug 29 11:51 66 -> pipe:[328515002]
lrwx------ 1 user user 64 Aug 29 11:51 67 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 68 -> pipe:[328515003]
l-wx------ 1 user user 64 Aug 29 11:51 69 -> pipe:[328515003]
lr-x------ 1 user user 64 Aug 29 11:51 7 -> /home/user/elasticsearch-jdbc/lib/log4j-slf4j-impl-2.5.jar
lrwx------ 1 user user 64 Aug 29 11:51 70 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 71 -> pipe:[328515004]
l-wx------ 1 user user 64 Aug 29 11:51 72 -> pipe:[328515004]
lrwx------ 1 user user 64 Aug 29 11:51 73 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 74 -> pipe:[328537438]
lrwx------ 1 user user 64 Aug 29 11:51 75 -> socket:[328515010]
l-wx------ 1 user user 64 Aug 29 11:51 76 -> pipe:[328537438]
lrwx------ 1 user user 64 Aug 29 11:51 77 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 78 -> pipe:[328537439]
l-wx------ 1 user user 64 Aug 29 11:51 79 -> pipe:[328537439]
lr-x------ 1 user user 64 Aug 29 11:51 8 -> /home/user/elasticsearch-jdbc/lib/log4j-core-2.5.jar
lrwx------ 1 user user 64 Aug 29 11:51 80 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 81 -> pipe:[328537440]
l-wx------ 1 user user 64 Aug 29 11:51 82 -> pipe:[328537440]
lrwx------ 1 user user 64 Aug 29 11:51 83 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 84 -> pipe:[328538745]
l-wx------ 1 user user 64 Aug 29 11:51 85 -> pipe:[328538745]
lrwx------ 1 user user 64 Aug 29 11:51 86 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 87 -> pipe:[328538746]
l-wx------ 1 user user 64 Aug 29 11:51 88 -> pipe:[328538746]
lrwx------ 1 user user 64 Aug 29 11:51 89 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 9 -> /home/user/elasticsearch-jdbc/lib/jts-1.13.jar
lr-x------ 1 user user 64 Aug 29 11:51 90 -> pipe:[328561899]
l-wx------ 1 user user 64 Aug 29 11:51 91 -> pipe:[328561899]
lrwx------ 1 user user 64 Aug 29 11:51 92 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 93 -> pipe:[328561900]
l-wx------ 1 user user 64 Aug 29 11:51 94 -> pipe:[328561900]
lrwx------ 1 user user 64 Aug 29 11:51 95 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 96 -> pipe:[328561901]
l-wx------ 1 user user 64 Aug 29 11:51 97 -> pipe:[328561901]
lrwx------ 1 user user 64 Aug 29 11:51 98 -> anon_inode:[eventpoll]
lr-x------ 1 user user 64 Aug 29 11:51 99 -> pipe:[328561902]