search

metadata-research / yamz

A crowd-sourced metadata dictionary
MIT License
9 stars 3 forks source link

Create robots.txt #79

Open cr625 opened 1 year ago

EMiller34 commented 1 year ago

This is a best practice to decide what we want robots to visit or not.

Discussion topic: Do we want a site map to assist robots in crawling?

machawk1 commented 2 months ago

Perhaps it should be a different Issue, but we should also consider a Sitemap to direct crawlers where to go in additional to what is allowed to crawl or not.