Closed tobh-stibo closed 6 years ago
Hi.
Do you know how many sockets/files that was open? What is the limit on open files on the server?
cat /proc/<humio-pid>/limits | grep "files"
We are running it in the your docker container.
It appears to be 4096.
PS. Our devops is off this week
4096 is too low. I would set this to 250k.
Open files comes from open sockets and open data source files. Not sure which is hitting you, but I would raise it and maybe look at it from time to time.
Also, look at our doc here: https://docs.humio.com/operation/installation/configuration_options/#raising-system-limits-for-humio
Closing this issue.
Humio stopped receiving logs, a restart fixed the issue. The Humio logs shows a bunch of these errors:
[839703-akka.actor.default-dispatcher-10559] ERROR a.i.TcpListener - Accept error: could not accept new connection java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:424) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:252) ~[?:?] at akka.io.TcpListener.acceptAllPending(TcpListener.scala:112) ~[humio-assembly.jar:0.1] at akka.io.TcpListener$$anonfun$bound$1.applyOrElse(TcpListener.scala:85) ~[humio-assembly.jar:0.1] at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[humio-assembly.jar:0.1] at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[humio-assembly.jar:0.1] at akka.io.TcpListener.aroundReceive(TcpListener.scala:34) ~[humio-assembly.jar:0.1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590) ~[humio-assembly.jar:0.1] at akka.actor.ActorCell.invoke(ActorCell.scala:559) ~[humio-assembly.jar:0.1] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) ~[humio-assembly.jar:0.1] at akka.dispatch.Mailbox.run(Mailbox.scala:224) ~[humio-assembly.jar:0.1] at akka.dispatch.Mailbox.exec(Mailbox.scala:234) ~[humio-assembly.jar:0.1] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[humio-assembly.jar:0.1] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[humio-assembly.jar:0.1] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[humio-assembly.jar:0.1] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[humio-assembly.jar:0.1]