Consider adding a robots.txt

ooni / explorer

OONI Explorer: uncover evidence of internet censorship worldwide

https://explorer.ooni.org

BSD 3-Clause "New" or "Revised" License

71 stars 37 forks source link

Open ainghazal opened 2 years ago

ainghazal commented 2 years ago

search engines are indexing arbitrary measurements. Not important, but it's perhaps worthy to drop a robots.txt excluding measurement/*

hellais commented 1 year ago

We currently have it at an nginx level, but we should move this over into explorer so it's easier to maintain.