robots.txt for .dev dictionaries

UAlbertaALTLab / morphodict

The Language Independent Intelligent Dictionary

https://morphodict.readthedocs.io/

Apache License 2.0

23 stars 11 forks source link

robots.txt for .dev dictionaries #1196

Open fbanados opened 1 month ago

fbanados commented 1 month ago

We make several dictionaries web-accessible for testing and internal demos, which should not be considered as deploy-ready. For this, we should update the robots.txt files for those deployments to ensure that entries do not end up in search engines.

fbanados commented 3 weeks ago

Robots should close access to all LLM data collection.