search

buttondown / roadmap

Buttondown's public roadmap
48 stars 0 forks source link

Allow authors to customize their `robots.txt` #2934

Open jmduke opened 3 weeks ago

jmduke commented 3 weeks ago

Discussed in https://github.com/buttondown/roadmap/discussions/2931

Originally posted by **krangor** June 24, 2024 Hi! Is there a way to include a robots.txt file on the web archive to tell AI crawlers to go away? (Or any other instructions for crawlers users might want to set up?) I'm aware that some crawlers ignore 'em but that's on them, not me...