Telegram API retry logic

[rrrliu]

Add a generic wrapper function or class that handles retry logic for failed bot API requests. Perhaps for 429 rate limiting errors we sleep for 5 seconds before retrying.

[ichub]

why are you thinking about this? has it been an issue so far?

cc @robknight who implemented rate limiting + retry logic for pretix sync and may have thoughts about it.

[rrrliu]

why are you thinking about this? has it been an issue so far?

cc @robknight who implemented rate limiting + retry logic for pretix sync and may have thoughts about it.

We have seen 429 rate limiting errors on staging already, as well as one instance of downtime in the telegram api. We should only anticipate these types of errors to increase at Zu/Devconnect

[ichub]

you should also consider adding logic to restart the telegram bot if it crashes. be careful w/ that though, since this logic would also be triggered by a deploy, which you don't want to cause the bot to restart. unless the deploy fails. maybe you should add an authenticated endpoint that starts the bot if it has crashed, so we can have some way of doing that in production w/out restarting the entire server. in any case, adding monitoring is becoming increasingly important.

[cha0sg0d]

Gonna try https://grammy.dev/plugins/auto-retry first