search

nrjones8 / robots-dot-txt-archive-bot

A project to collect, archive, and publish robots.txt files from across the internet - with a focus on government websites
https://robots-dot-txt-db.com/
6 stars 0 forks source link

make it easier to add new sources #7

Open nrjones8 opened 4 years ago

nrjones8 commented 4 years ago

right now every time we add a new source of hostnames, there are a handful of places that need to be updated manually, make a more central config