search

x-cubed / event-store-client

JS client library for connecting to Event Store over TCP/IP
The Unlicense
71 stars 24 forks source link

Subscription dropout #24

Closed PaulGrimshaw closed 7 years ago

PaulGrimshaw commented 7 years ago

We are getting subscriptions dropping intermittently using this library. Although we log errors on both connection drop and on Error, the subscription just drops silently, not giving any hint as to why it has stopped receiving events. This doesn't appear to be after any set period of time, sometimes it is almost straight away, sometimes after a few hours. With DEBUG mode on, we can see the heardbeats continue on without fail after the last event before the drop.

Any idea what might be going on and how to debug this further?

x-cubed commented 7 years ago

At minimum, a packet capture of the traffic on port 1113 would give us some insight into what is going on.

Is the subscription (rather than the connection) onDropped handler being invoked with a reason why the subscription was cancelled?

PaulGrimshaw commented 7 years ago

I haven't managed to do a packet capture yet. But I have found an issue on the server where the DNS was dropping periodically, so maybe this was the issue? I connect by direct IP however so I wouldn't have expected that.

I was logging in all the callbacks (onDropped / onErrror) in both the connection and subscription, but nothing was logged. I have switched to a poll based subscription for now, will revisit when i have more time.

x-cubed commented 7 years ago

It might be that the Event Store needs to be pinged regularly for it to consider the connection to be alive alive, and that's not something that the client does automatically, but those methods are exposed on the connection for your application to use.

PaulGrimshaw commented 7 years ago

Ok have updated our code to fire a ping every 10 seconds. Will keep an eye on it and see if this prevents the dropout.

PaulGrimshaw commented 7 years ago

Just to confirm, firing a regular ping to keep the connection alive appears to have resolved this issue. Would this not be something you'd have the client do on connect?

Anyhow thanks for the solution.

x-cubed commented 7 years ago

It would need to be done regularly so that the server knows that the client is still there. Ideally, that would be done on a timer in your application.

PaulGrimshaw commented 7 years ago

Yes. I have a wrapper class around the API with a connect and disconnect method. The connect method starts a timer pinging every 10 seconds, the disconnect clears the timer. Just thought this could maybe be behaviour the library could take over, as it's not immediately obvious for others that you'd need to do this for a long subscription.

x-cubed commented 7 years ago

Yeah, if there's a way we can get the library to do it that would be good. It's been a while since I've looked at timers and threading in Node.js, so I'm not too familiar with how we can do it without potentially affecting the application itself.