Block AI scrapers - Githubissues

ubyssey / ubyssey.ca

The code that powers www.ubyssey.ca

GNU General Public License v2.0

35 stars 40 forks source link

Block AI scrapers #1450

Closed SamuelmdLow closed 1 month ago

SamuelmdLow commented 2 months ago

Scrapers from companies like Amazon, ByteDance, Anthropic and probably more hit our Archive page continuously all day everyday. We should use our Robots.txt file to block them