Nick/use better domain sources

nrjones8 / robots-dot-txt-archive-bot

A project to collect, archive, and publish robots.txt files from across the internet - with a focus on government websites

https://robots-dot-txt-db.com/

6 stars 0 forks source link

Closed nrjones8 closed 4 years ago