Recover from disconnects

[pushcx]

A couple times in the last few weeks mockturtle has lost its connection to Libera. It continues running but can’t reconnect.

Maybe the straightforward fix here is to detect the net connection has dropped and exit the process? I’d rather not build complicated reconnect logic, let a new process rebuild. It’s not so bad if we lose a single post along the way, though depending when and where the first exception happens, maybe the cache file won’t be updated so the new process will handle it?

Sep 16 19:25:28 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"defaultMeta":{"account":false},"message":"Activating scheduled tasks.","level":"info","timestamp":"2024-09-16 19:25:28"}
Sep 16 19:29:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"defaultMeta":{"account":false},"message":"Activating scheduled tasks.","level":"info","timestamp":"2024-09-16 19:29:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Client Ping timeout (120 seconds)","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Connection.end() connected=true with data=true had_error=true","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Connection.end() connected=true with data=false had_error=true","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport close() destroying","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketClose()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Socket closed. was_connected=true safely_registered=true requested_disconnect=false","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:46 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Scheduling reconnect. Attempt: 1/3 Wait: 6894ms","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:46"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Connection.setEncoding() encoding=utf8","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport connect()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport disposeSocket() connected=false","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport Connection.setEncoding() encoding=utf8","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport Connecting socket..","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:53 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketRawConnected()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:53"}
Sep 16 19:31:54 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketFullyConnected()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:54"}
Sep 16 19:31:54 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Socket fully connected","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:31:54"}
Sep 16 19:32:27 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"NetTransport socketClose()","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:32:27"}
Sep 16 19:32:27 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"Socket closed. was_connected=true safely_registered=false requested_disconnect=false","level":"debug","message":"Debug Event","timestamp":"2024-09-16 19:32:27"}
Sep 16 21:21:37 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"defaultMeta":{"account":"mockturtle"},"itemDate":"2024-09-16T21:21:01.000Z","itemGuid":"https://lobste.rs/s/mun2dn","lastSeen":"2024-09-16T21:21:01.000Z","level":"info","message":"Broadcasting story","timestamp":"2024-09-16 21:21:37"}
Sep 16 21:21:37 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"write() called when not connected","level":"debug","message":"Debug Event","timestamp":"2024-09-16 21:21:37"}
Sep 16 21:21:37 ubuntu-s-1vcpu-1gb-nyc1-01 sh[3854313]: {"event":"write() called when not connected","level":"debug","message":"Debug Event","timestamp":"2024-09-16 21:21:37"}

[OscarFKE]

I'm having a look through the code now, my first guess is that the bot is not responding to PING requests from the libera irc server. Have you updated the dependencies recently?

I have some free time tomorrow, I'll try my hand at updating the bot to gracefully exit the process on connection end.

[pushcx]

It's been quite some time since we bumped dependencies on this bot. I don't know if Libera changed something about pings recently.

[pushcx]

I asked in #libera-communities and an op said they don't believe anything has changed in this area.