Allow authors to customize their `robots.txt`

buttondown / roadmap

Buttondown's public roadmap

53 stars 0 forks source link

Allow authors to customize their `robots.txt` #2934

Open jmduke opened 5 months ago

jmduke commented 5 months ago

Discussed in https://github.com/buttondown/roadmap/discussions/2931

^{Originally posted by **krangor** June 24, 2024} Hi! Is there a way to include a robots.txt file on the web archive to tell AI crawlers to go away? (Or any other instructions for crawlers users might want to set up?) I'm aware that some crawlers ignore 'em but that's on them, not me...

docuguy1000 commented 4 months ago

Thank you for working on this!