Add robots.txt requesting popular AI bots to not scrape data

ubccsss / ubccsss.org

Website for the UBC CSSS

https://ubccsss.org/

MIT License

14 stars 46 forks source link

Closed RamitKataria closed 8 months ago

RamitKataria commented 8 months ago

Resolves #571

RamitKataria commented 8 months ago

Thanks! It's from Cloudflare haha

Isn't the noindex for Google search results which I think we do want?