search

Jaymon / bang

A static site generator with support for RSS feeds, sitemaps, and Google Amp
MIT License
13 stars 0 forks source link

Turn off AI crawlers in robots.txt #63

Open Jaymon opened 1 year ago

Jaymon commented 1 year ago 
User-Agent: AiBot
Disallow: /

User-Agent: Google-Extended
Disallow: /

This comes from a mediagazer post about Medium

Jaymon commented 10 months ago

https://medium.com/@nuaveu/how-to-block-ai-from-scraping-your-content-ba03098ba199