Cache robots.txt for each run of the script, to reduce load on the server

nanos / FediFetcher

FediFetcher is a tool for Mastodon that automatically fetches missing replies and posts from other fediverse instances, and adds them to your own Mastodon instance.

https://blog.thms.uk/fedifetcher?utm_source=github

MIT License

293 stars 213 forks source link

Cache robots.txt for each run of the script, to reduce load on the server #129

Closed nanos closed 1 week ago

nanos commented 1 week ago

This addresses #127