search

oddin-gg / netcoresdk

BSD 3-Clause "New" or "Revised" License
4 stars 1 forks source link

Feed not reconnecting after restart #8

Open Seuntjie900 opened 3 years ago

Seuntjie900 commented 3 years ago

Our integration feed went offline and didn't reconnect. From our logs:

10:25:00,047     Feed    WARN    The AMQP connection was shut down. Cause: System.IO.EndOfStreamException: Reached the end of the stream. Possible authentication failure.
   at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(Stream reader, Byte[] frameHeaderBuffer)
   at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()
   at RabbitMQ.Client.Framing.Impl.Connection.MainLoop() 
10:25:14,106     ProducerRecoveryManager     INFO    Alive message wasn't received in time! 
10:25:14,106     ProducerRecoveryManager     INFO    Alive message wasn't received in time! 
10:25:14,106     ProducerRecoveryManager     INFO    Starting recovery... 
10:25:14,106     ProducerRecoveryManager     INFO    Starting recovery... 
10:25:14,542     CompleteInfo    WARN    Producer Oddin.OddsFeedSdk.API.Entities.Producer is down 
10:25:14,542     ProducerRecoveryManager     INFO    Sending recovery request (producer name: pre, request ID: 3, node ID: 1, UTC timestamp: 10/4/2021 10:24:59 AM) 
10:25:14,678     CompleteInfo    WARN    Producer Oddin.OddsFeedSdk.API.Entities.Producer is down 
10:25:14,678     ProducerRecoveryManager     INFO    Sending recovery request (producer name: live, request ID: 4, node ID: 1, UTC timestamp: 10/4/2021 10:24:59 AM)

The feed didn't log anything again or receive any messages for about 6 minutes, then started logging the following every 10 seconds:

10:31:47,465     ProducerRecoveryManager     INFO    An alive message with subscribed == 0 was received! 
10:31:47,466     ProducerRecoveryManager     INFO    An alive message with subscribed == 0 was received! 
10:31:57,467     ProducerRecoveryManager     INFO    An alive message with subscribed == 0 was received! 
10:31:57,468     ProducerRecoveryManager     INFO    An alive message with subscribed == 0 was received!

It didn't reconnect until we stopped the service completely and restarted it about 2 hours later. During this time, it doesn't look like we were receiving any data, (I can't see from the log that any odds changes was sent over our socket or bet any settlements were received). The producerupevents were never fired, which is why we weren't accepting bets until they manually enabled them.

Neither the OnClosed nor ConnectionException events were fired.

michal-fuleky commented 3 years ago

We were unable to reproduce this issue. We have release an update (1.1.1). If this problem happens again we will have more info. For now, we've changed the logic of Disconnect/Closed events, so that you should receive Closed event if this happens again.