add a README, document data sources better

nrjones8 / robots-dot-txt-archive-bot

A project to collect, archive, and publish robots.txt files from across the internet - with a focus on government websites

https://robots-dot-txt-db.com/

6 stars 0 forks source link

Open nrjones8 opened 4 years ago