Ratelimiting for feed refreshes and API requests generally

[sneakers-the-rat]

Currently I am writing this with a configurable crontab-syntax update period to refresh metadata from tracked journals, but it also makes sense to me to be able to refresh feeds on demand either by visiting the HTML page, or even whenever the feed is visited.

For that to work we would need to make a system for having the periodic update be the floor, and then some other interval to be the ceiling of an update frequency. To do that we would probably want to make some general handler for API requests instead of trying to do that as a one-off for feed refreshes: we will inevitably hit some ratelimiting/backoff signals from APIs and we should handle those gracefully.

This is probably en route to a more structured system for data sources - might be nice to have a metaclass that all data sources can inherit from that indicates what fields they are able to provide with mappings to methods to get them. Early thoughts, usually I try and implement a few iterations of something before trying to make a generic version

[sneakers-the-rat]

requests can just do this apparently: https://docs.python-requests.org/en/latest/user/advanced/#example-automatic-retries but that's at the level of an individual request, rather than rate-limiting all connections to a given domain. masto does this relatively well:

• inbox-level ratelimiting: https://github.com/NeuromatchAcademy/mastodon/blob/eb24c0ad07c4137517e6bd37ebcc99d6e2b86797/app/workers/activitypub/delivery_worker.rb#L61
• request-level retries and backoffs: https://github.com/NeuromatchAcademy/mastodon/blob/eb24c0ad07c4137517e6bd37ebcc99d6e2b86797/app/workers/activitypub/delivery_worker.rb#L11-L21