eosdac / eosdac-api

DAC API which takes data from state history
11 stars 11 forks source link

Filler amq connection reset handler #17

Closed rkamysz closed 2 years ago

rkamysz commented 2 years ago

This PR contains a fix for the problem that the 'filler' process crashes when AMQ loses connection.

There are several reasons for the problem ... the first is that AMQ does not inform the environment about the lack of connection to the server. Each command was unnecessarily preceded by an init method which was not in the try catch block which caused an exception. Whenever the connection is lost, AMQ tries to reestablish it and all processes should stop until then.

In this hot-fix no method (send, listen, ack...) is invoked if AMQ is unavailable, instead a "no connection" message is logged.

The next thing is that we have a lot of nested code in which we pass instances of AMQ and still do nothing in case of connection changes. I added simple callbacks (onDisconnect and onReconnect) to handle the AMQ status change in some way.

The problem is also StateReceiver and Connection which use AMQ and WebSocket in a chaotic and poorly controlled way. Despite connection issues on AMQ, the messages were still pushed. To avoid this I had to terminate the web socket connection in an ugly way (we have no access to do it softly - because it's external package) and dispose the stateReceiver handlers. When the AMQ connection is restored, the handlers and the stateReceiver / webSocket itself are recreated.

Please check if nothing breaks