search

humio / issues

Issue Tracker for Humio
4 stars 2 forks source link

IOException: Too many open files #42

Closed tobh-stibo closed 6 years ago

tobh-stibo commented 6 years ago

Humio stopped receiving logs, a restart fixed the issue. The Humio logs shows a bunch of these errors:

[839703-akka.actor.default-dispatcher-10559] ERROR a.i.TcpListener - Accept error: could not accept new connection java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:424) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:252) ~[?:?] at akka.io.TcpListener.acceptAllPending(TcpListener.scala:112) ~[humio-assembly.jar:0.1] at akka.io.TcpListener$$anonfun$bound$1.applyOrElse(TcpListener.scala:85) ~[humio-assembly.jar:0.1] at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[humio-assembly.jar:0.1] at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[humio-assembly.jar:0.1] at akka.io.TcpListener.aroundReceive(TcpListener.scala:34) ~[humio-assembly.jar:0.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) ~[humio-assembly.jar:0.1] at akka.actor.ActorCell.invoke(ActorCell.scala:559) ~[humio-assembly.jar:0.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) ~[humio-assembly.jar:0.1] at akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[humio-assembly.jar:0.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) ~[humio-assembly.jar:0.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[humio-assembly.jar:0.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[humio-assembly.jar:0.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[humio-assembly.jar:0.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[humio-assembly.jar:0.1]

pmech commented 6 years ago

Hi.

Do you know how many sockets/files that was open? What is the limit on open files on the server?

pmech commented 6 years ago

cat /proc/<humio-pid>/limits | grep "files"

tobh-stibo commented 6 years ago

We are running it in the your docker container.

It appears to be 4096.

PS. Our devops is off this week

pmech commented 6 years ago

4096 is too low. I would set this to 250k.

Open files comes from open sockets and open data source files. Not sure which is hitting you, but I would raise it and maybe look at it from time to time.

pmech commented 6 years ago

Also, look at our doc here: https://docs.humio.com/operation/installation/configuration_options/#raising-system-limits-for-humio

Closing this issue.