Add robots.txt to prevent scraping for AI training - Githubissues

cct-datascience / group-handbook-template

Template for creating a lab group handbook with Quarto

Creative Commons Attribution 4.0 International

4 stars 2 forks source link

Add robots.txt to prevent scraping for AI training #10

Open Aariq opened 1 month ago

Aariq commented 1 month ago

Add a robots.txt file to block web crawlers for AI training https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/

[ ] create robots.txt
[ ] add to _quarto.yml project: resources:
[ ] explain in README.md instructions

Aariq commented 1 month ago

Also take a look at this: https://darkvisitors.com/docs/robots-txt