search

nanos / FediFetcher

FediFetcher is a tool for Mastodon that automatically fetches missing replies and posts from other fediverse instances, and adds them to your own Mastodon instance.
https://blog.thms.uk/fedifetcher?utm_source=github
MIT License
293 stars 213 forks source link

Robots.txt is fetched for every request made. #127

Closed cooperaj closed 1 week ago

cooperaj commented 1 week ago

It'd probably be better to cache the results of a robots.txt check for any given run so that only a single call would be made.

Additionally, the user agent used to pull the robots.txt is not the same as the fedifetcher one.

"GET /robots.txt HTTP/1.1" 200 190 "-" "Python-urllib/3.11"

nanos commented 1 week ago

should both be addressed now.