search

nrjones8 / robots-dot-txt-archive-bot

A project to collect, archive, and publish robots.txt files from across the internet - with a focus on government websites
https://robots-dot-txt-db.com/
6 stars 0 forks source link

pull in `title` or something similar from each hostname #4

Open nrjones8 opened 4 years ago

nrjones8 commented 4 years ago

not everybody knows that "uspto.gov" is "United States Patent and Trademark Office" - would be helpful to have that in the data as well. just use the title tag...?