During the XMTP incident this morning, the nodes were briefly unavailable. Two undesirable things happened with the notification listener at that time.
We continued trying to reconnect so rapidly that the notification server was throttled by the WAF
Once the XMTP nodes came back online, the connection was still not restored in the notification server and errors continued
The fix
When there is an error in the stream other than io.EOF, create a new connection to the XMTP node server and wait 100ms before retrying.
Summary
During the XMTP incident this morning, the nodes were briefly unavailable. Two undesirable things happened with the notification listener at that time.
The fix
When there is an error in the stream other than
io.EOF
, create a new connection to the XMTP node server and wait 100ms before retrying.