Bot stuck in loop: "Heartbeat not answered properly."

ZacharyTalis / AltTextBot

A Discord bot designed to reduce the message clutter associated with image alt-text. Also includes leaderboard functionality for alt-text submitters!

https://zacharytalis.com/blog/atb

Apache License 2.0

2 stars 0 forks source link

Bot stuck in loop: "Heartbeat not answered properly." #18

Open ZacharyTalis opened 3 years ago

ZacharyTalis commented 3 years ago

Issue: Bot was online but unresponsive, with the following message repeatedly appearing in the log:

[Javacord - Central Scheduler - 1] DEBUG org.javacord.core.util.gateway.Heart - Heartbeat not answered properly. This might be because of a busy websocket

The bot responded normally upon restart.

Proposed solution: (Somehow) detect that this issue has occurred, and restart the bot's container.

glossawy commented 3 years ago

This seems to be related to Javacord/Javacord#627, the simplest way I see to detect this is to see how long its been since DiscordApi#getLatestGatewayLatency was updated indicating time since the last heartbeat and restarting the bot if that exceeds some threshold. To be extra sure I would say if it exceeds 5 seconds. Maybe more.

glossawy commented 1 year ago

May eventually fix this so notes:

Create a Pacemaker that checks that the DiscordApi heart is beating via checking against #getLatestGatewayLatency and disconnect and reconnect the bot
- Require change to the DiscordBot interface to support a simple stop-then-start reconnect after a short delay
- Threshold it at 5 seconds
- Delay 5s between restarts
Need some sort of alerting setup