Frequent socket closed errors when adding event.

[iandotkelly]

I'm using v5.2.0 of the Java Client - from a Scala application. I'm running Java 8u171 from OSX.

I'm attempting to send data to a stream, and most of the time I get a "Socket closed" error when sending. Then once in a while I can send a whole sequence of successful events, then it goes back to failing.

I turned on logging and get the following:

FINER: Adding event to collection: laksjdflskjdf
May 10, 2018 4:46:43 PM io.keen.client.java.KeenLogging log
FINER: Sent request '{"keen":{"timestamp":"2018-05-10T16:46:43.039-0400"},"eventId":"fe7c9a10-5491-11e8-9ab9-5bf7447501b3","triggeredRule":"29c87ed0-53bd-11e8-bec9-c148352c0304","messageId":"fdcde381-5491-11e8-9ab9-5bf7447501b3","message":{"body":{"footer":194,"message":{"maxAcceleration":67,"finalSpeed":50,"initialSpeed":33,"type":"HardAccelerationMessage"},"header":{"timestamp":"2018-05-10T16:58:39-04:00","latitude":41.29999923706055,"tripType":"Idling","vehicleProtocolId":"ISO15765_11_BIT_CAN","longitude":81.4000015258789,"messageType":5,"odo":123456,"tripNumber":1,"fixQuality":"StoredFix","messageLength":4}}},"receivedAt":"2018-05-10T16:37:43.992Z","deviceId":"2c207890-53bd-11e8-b4e6-d92220b211a4","tags":[],"deliveredAt":"2018-05-10T16:46:43.039Z"}' to URL 'https://api.keen.io/3.0/projects/59e0d1acc9e77c0001573537/events/laksjdflskjdf'
May 10, 2018 4:46:43 PM io.keen.client.java.KeenLogging log
FINER: Encountered error: Socket closed

I'm using a singleton client, and the addEvent() method to send the events. I know the project-id and write key are correct as once in a while it all works.

What am I doing wrong here, or is this an issue with the library on Java 8 or OSX?

[Geeber]

Hmm, that's definitely strange. My first instinct is some kind of networking issue - are you using a proxy, or is there anything else unusual about your networking situation? One thing you could try is submitting the events with curl (or something equivalent). If you don't see the same issues then it's a good indication that something's off in the library, your usage of it, your java version, or some combination. If you do see the same issues then it's very likely networking, or at least it's pretty clearly not the Java client.

You can see an example of submitting an event via curl here: https://keen.io/docs/api/?shell#record-a-single-event.

[iandotkelly]

@Geeber thanks for the quick reply. I assumed networking issues but tried both at work and at home - neither place has any http proxy that I am aware of, and I don't have a proxy or firewall rules enabled on a Mac. In any case I would expect it to either work or not work.

Yeah, I'll try generating the request some other way, like curl and see how I get on.

[iandotkelly]

@Geeber ... so today both curl and my application are working perfectly, with no code or environment changes. This wasn't an isolated incident though, last week when I first played with the Java client I had Socket closed issues.

If I experience this again I will compare Java client to curl performance and let you know.

[iandotkelly]

@Geeber

Ok, I spoke a little too soon ... I'm still getting occasional errors, nothing like the rate that I got yesterday.

I've had zero errors with curl, even in periods where I have had 4-5 Socket closed errors when retrying sending with the client. My code uses an exponential backoff when errors occur, and I've had periods of 1 minute unable to send with the client but curl is working.

However in that period, its only making a small number of attempts. Unfortunately since the rate of errors today is so low, its difficult to draw any definite conclusion from the fact that I've had no issues with curl. It could be a networking issue but it does seem like the client is quite unreliable.

[Geeber]

That's very strange. I'm pretty sure there are other people using this version of the Java SDK at scale and not reporting issues. @masojus do you remember seeing or hearing anything like this?

I wonder if it's worth trying with a different JDK, or a different HTTP library (see https://github.com/keenlabs/KeenClient-Java#keenclient-interfaces)? I would try to reproduce locally but it's hard to prioritize it based on only one report, so it might be a while before we can get to it. It would be great if we could further isolate the cause somehow though... :/

[masojus]

I don't recall seeing this behavior, unfortunately, so I'm afraid I can't provide too much insight.

[iandotkelly]

@Geeber & @masojus

I switched to an entirely different http client and generated the POST requests myself, and encountered a similar issue when generating a number of requests a second. This was due to a limit of datagram sockets at the JVM level when DNS lookups were happening when the requests were generated - clearly I was a little close to the limit, which I think is 25. It could be fixed with the following JVM property:

-Dsun.net.maxDatagramSockets=1024

I've not gone back and tried this with the Keen client, but I suspect it was the same cause.

Many thanks.

[Geeber]

Good to know, and glad you got it sorted out!