search

xmtp / example-notification-server-go

Example push notification server, written in Golang
MIT License
15 stars 8 forks source link

Better handle downtime #22

Closed neekolas closed 1 year ago

neekolas commented 1 year ago

Summary

During the XMTP incident this morning, the nodes were briefly unavailable. Two undesirable things happened with the notification listener at that time.

  1. We continued trying to reconnect so rapidly that the notification server was throttled by the WAF
  2. Once the XMTP nodes came back online, the connection was still not restored in the notification server and errors continued

The fix

When there is an error in the stream other than io.EOF, create a new connection to the XMTP node server and wait 100ms before retrying.

neekolas commented 1 year ago

Tested this locally, and refreshing the client does seem to make it recover gracefully from connection failures