2023-03-28 outage of the Libera bridge

matrix-org / matrix-appservice-irc

Node.js IRC bridge for Matrix

Apache License 2.0

461 stars 149 forks source link

2023-03-28 outage of the Libera bridge #1693

Closed progval closed 1 year ago

progval commented 1 year ago

(timestamps in UTC)

Somewhere between 8:51 and 8:54, the bridge stopped relaying IRC->Matrix messages.
From 8:56 to 9:02, puppets timed out
At 9:06, the bridge sent an IRC->Matrix message, timestamped to 9:06, even though I sent it at 8:54
at 9:07, puppets started reconnecting
at 9:21, about a tenth of the puppets are back online, some channels still did not have any puppet at all

Matrix users did not get any notification of the outage, and https://status.matrix.org/ is green.

Half-Shot commented 1 year ago

Looks like we had a shudder as a result of some admin actions. I'm having a look, mostly seems to be reconnections.

Half-Shot commented 1 year ago

Bridge restarted. Should be back in a bit. I'll do a quick status line.

Half-Shot commented 1 year ago

Status page update is in https://status.matrix.org/incidents/ztp9sq3570pd.

Half-Shot commented 1 year ago

Well I spoke too soon. It had another issue after recovering.

Half-Shot commented 1 year ago

Issue logged as #1694 , we're watching the recovery now.

alanz commented 1 year ago

Is it possible to inject something equivalent to a ChanServ message to the affected rooms saying this has happened? So we can know to look at alternate logs for anything possibly missing

progval commented 1 year ago

You can make your client display hidden events (eg. via /devtools in Element); it will show org.matrix.appservice-irc.connection events when this happens.

Half-Shot commented 1 year ago

This issue did recover, so closing.