Add robots.txt requesting popular AI bots to not scrape data

ubccsss / ubccsss.org

Website for the UBC CSSS

https://ubccsss.org/

MIT License

16 stars 47 forks source link

Closed RamitKataria closed 9 months ago

RamitKataria commented 9 months ago

Resolves #571

RamitKataria commented 9 months ago

Thanks! It's from Cloudflare haha

Isn't the noindex for Google search results which I think we do want?