paritytech / bench-bot

ISC License
9 stars 19 forks source link

bench-bot is not responding to commands sometimes #83

Closed joao-paulo-parity closed 2 years ago

joao-paulo-parity commented 2 years ago

e.g. https://github.com/paritytech/polkadot/pull/4777#issuecomment-1022412673

might be related to https://github.com/paritytech/bench-bot/issues/68

or might be related to some other problem which is not visible from the logs at the moment https://github.com/paritytech/bench-bot/issues/82

joao-paulo-parity commented 2 years ago

My current guess is related to the proxy we're using for receiving events (https://github.com/paritytech/devops/issues/1070). Specifically after we reconnect:

ERROR (server):                                                                                        
    type: "error"                                                                                      
INFO (server): Connected    

It seems that although the server says it's "Connected" again, we might no longer receive events.

Here's the trail of code sources leading up to this assumption:

If the guess is correct then we can try to restart the whole server when an error happens instead of reconnecting, but that implies a refactor in the server's setup.

This problem might originate from the library we're using, https://github.com/EventSource/eventsource, which does not appear to be well-maintained. If this bot were to be rewritten in another language as suggested by #54 then the error could end up being solved by using a (better and) different implementation of EventSource, e.g. in processbot we're using a different library (https://github.com/paritytech/parity-processbot/blob/2f8ffedb62a2abc1a3b50c983087e80c1d8677be/src/main.rs#L72).