Planned server restart on Libera causes the bridge to be silently unavailable for 2.5 hours

[progval]

Libera had a planned netsplit today at 19:55 UTC, and still has not recovered at the time I am writing this. No message is going from IRC to Matrix, and most Matrix users' messages are not sent to IRC, because their puppets did not reconnect.

Additionally, the org.matrix.appservice-irc.connection event is not set on the Matrix side until someone speaks on the IRC side.

[progval]

It's working again since ~22:35 UTC; looks like the interruption was because of the privacy filter kicking in while the bridge took 2.5h to reconnect all the puppets

[Half-Shot]

Needs investigation from the team to see why this took so long.

[tadzik]

I see a wave of these in the logs:

Aug 30 20:30:04 liberairc[12912]: 2022-08-30 20:30:04 ERROR:client-connection Server: irc.libera.chat (XXX) Network Error: {
Aug 30 20:30:04 liberairc[12912]:   "errno": -110,
Aug 30 20:30:04 liberairc[12912]:   "code": "ETIMEDOUT",
Aug 30 20:30:04 liberairc[12912]:   "syscall": "read"
Aug 30 20:30:04 liberairc[12912]: }
Aug 30 20:30:04 liberairc[12912]: 2022-08-30 20:30:04 INFO:client-connection disconnect()ing XXX@irc.libera.chat - net_error
Aug 30 20:30:04 liberairc[12912]: 2022-08-30 20:30:04 WARN:ClientPool Client bzzzzzzzzzzz (@XXX:XXX.com) disconnected with reason net_error

But nothing after 20:30 – seems like it took over half an hour for the bridge to notice its old connections timing out, at which point it reconnected the puppets – and the connections themselves took as short as they usually do.

I see nothing hinting at a 2.5 hour delay though.

[progval]

This is happening again today (2022-09-07). Timeline (in UTC) in #matrix-irc as far as I can tell:

• 06:47:13 to 06:47:27: netsplit
• 06:50:22 to 06:50:22: netjoin
• 06:50:58 to 06:56:20: puppets time out
• 07:05:04 to 09:30:31: puppets reconnect
• 09:35:36: someone asks a question from Matrix (relayed to IRC)
• 09:40:11: I reply (not relayed to Matrix), org.matrix.appservice-irc.connection gets set to {"blocked": true}
• 09:46:35 to 12:15:01: more puppets reconnect
• 12:35:23: @tadzik replies from Matrix (relayed to IRC)
• 12:40:36: I reply (again, not relayed)
• 16:28:57: ditto

Still no IRC->Matrix relaying on #matrix-irc last I checked (2022-09-12 08:18:33)

[podiki]

Seeing the same thing here, I can send from Matrix, it goes through, but do not get messages from IRC->Matrix. One channel (stumpwm) is working fine, while another (guix), has been this way for about 12 hours as of now. I've tried disconnecting and reconnecting from Matrix to the channel, with no changes.

[podiki]

Another day later and still only have one way connection in #guix (two other rooms seem fine though). Anything we can do?

[simonmichael]

I confirm an outage still in progress, eg #haskell IRC channel has not been relaying to matrix since 2022-09-08 0330 UTC.

[exarkun]

The last message I saw come through on Matrix from #python was at 2022-09-08 2116 (I don't know what TZ Element is reporting this timestamp in).

[eeickmeyer]

Confirming outage is still in progress. All #ubuntustudio rooms are one-way Matrix -> IRC.

[CairnThePerson]

Still having issues in #emacs and #guix.

[podiki]

Is there anyone to contact to restart these bridges? Given it has been days now, I'm not holding my breath they will fix themselves.

Edit: they know https://fosstodon.org/@liberachat/108976211933737326

[progval]

Element replied on Twitter https://twitter.com/element_hq/status/1568710974210719745

[progval]

Element just restarted the bridge, it's slowly coming back up

[progval]

Seems to be working fine in rooms that were working a week ago. You should probably open a new issue if you see a room that doesn't work

[progval]

It's happening again. Netsplit today at 11:46, and all IRC->Matrix and some Matrix->IRC messages are still missing at 12:32.

EDIT: and it fixed itself at 13:21:14 (possibly a little later for other channels), so about 2.5 hours later

EDIT2: #matrix-irc was missing a puppet until I spoke in the channel, much later than that:

15:36:26 <val> jA_cOp: it's not restarted several times a week, it just crashes when it sees too much traffic at once
15:36:26 --> psydroid [psydroid] (@psydroid:matrix.org) (~psydroid@user/psydroid) has joined #matrix-irc
15:37:54 <val> test

only the second message went through.

[Half-Shot]

While we're still experiencing shaky bridge behavior, the project has moved on somewhat and this is better tracked in the later issues.