Add robots.txt and exclude public lists from it

cc-archive / open-ledger

Prototype code and examples for work on the Creative Commons "CC Search" project

MIT License

48 stars 23 forks source link

Closed lizadaly closed 7 years ago

lizadaly commented 7 years ago

Currently's there's no robots.txt at all; we should have one to make it easy to update.

Secondarily, it should exclude public lists for now as that isn't content we control.

lizadaly commented 7 years ago

Done.