search

nanos / FediFetcher

FediFetcher is a tool for Mastodon that automatically fetches missing replies and posts from other fediverse instances, and adds them to your own Mastodon instance.
https://blog.thms.uk/fedifetcher?utm_source=github
MIT License
291 stars 213 forks source link

Add custom extension to robots.txt cache #135

Open p37307 opened 2 days ago

p37307 commented 2 days ago

Because of the domain names extension, my OS is associating filetypes to the cached robots files in artifacts.

For instance, it associates domains ending in .au as audio files, .one as MS Office OneNote files, .cat as Windows security cat files, .com as MS_DOS applications, etc.

It may not be a big deal unless my search indexer tries to read its data, expecting to get data associated with those filetypes and mucks up.

Could you add a unique extension when it saves the robots file that is universally recognized as a text file.

image

image

image

nanos commented 2 days ago

Oops. I do realise that this is probably not the smartest anyway. I’m sure bad stuff will happen if someone has utf-8 glyphs in their domain name. I think I’ll just do a hash of the domain name and use that…