search

benmur / riemann-scala-client

Scala client library for sending events to Riemann
MIT License
19 stars 7 forks source link

Never recovers from connection failure #20

Open chris-martin opened 8 years ago

chris-martin commented 8 years ago

I just restarted my Riemann server and saw my client app enter a failure state wherein every attempt to send data to Riemann failed, logging this:

2016-01-19 04:34:51,895 ERROR n.b.r.c.ReliableIO$TcpConnectionActor akka://foo/user/riemann-tcp-client-1/io - could not send or receive data
java.io.EOFException: null
        at java.io.DataInputStream.readInt(DataInputStream.java:392) ~[na:1.7.0_65]
        at net.benmur.riemann.client.ReliableIO$TcpConnectionActor$$anonfun$receive$2.applyOrElse(ReliableIO.scala:90) ~[foo-249.jar:0.0]
        at akka.actor.Actor$class.aroundReceive(Actor.scala:467) ~[foo-249.jar:0.0]
        at net.benmur.riemann.client.ReliableIO$TcpConnectionActor.aroundReceive(ReliableIO.scala:79) ~[foo-249.jar:0.0]
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [foo-249.jar:0.0]
        at akka.actor.ActorCell.invoke(ActorCell.scala:487) [foo-249.jar:0.0]
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [foo-249.jar:0.0]
        at akka.dispatch.Mailbox.run(Mailbox.scala:220) [foo-249.jar:0.0]
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [foo-249.jar:0.0]
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [foo-249.jar:0.0]
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [foo-249.jar:0.0]
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [foo-249.jar:0.0]
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [foo-249.jar:0.0]
2016-01-19 04:34:51,896 INFO  akka.actor.DeadLetterActorRef akka://foo/deadLetters - Message [com.aphyr.riemann.Proto$Msg] from Actor[akka://foo/user/riemann-tcp-client-1/io#916728015] to Actor[akka://foo/deadLetters] was not delivered. [270] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

Restarting the client app fixed the problem, leading me to assume that there's some state in the Riemann client library that isn't getting reset after a connection fails.

riemann-scala-client version 0.3.4

chris-martin commented 8 years ago

I am creating a single RiemannDestination[Reliable] and reusing it throughout the lifetime of my application. I'm unclear from reading the docs, is that how this library is intended to be used?

benmur commented 8 years ago

It's supposed to be used this way, and there is code to reconnect to Riemann when the connection is in a bad state. I will have a look.