Cache robots.txt for 24 hours on disk to reduce load on servers

nanos / FediFetcher

FediFetcher is a tool for Mastodon that automatically fetches missing replies and posts from other fediverse instances, and adds them to your own Mastodon instance.

https://blog.thms.uk/fedifetcher?utm_source=github

MIT License

293 stars 213 forks source link

Cache robots.txt for 24 hours on disk to reduce load on servers #130

Closed nanos closed 5 days ago