ArchiveTeam / ArchiveBot

ArchiveBot, an IRC bot for archiving websites
http://www.archiveteam.org/index.php?title=ArchiveBot
MIT License
352 stars 72 forks source link

Add Googlebot, Bingbot, Python Requests et. al. #556

Open tomodachi94 opened 1 year ago

tomodachi94 commented 1 year ago

Added the above to the user-agents list.

googlecrawlers.json contains most of Google's spiders and crawlers.

bingbot.json and python-requests.json are self-explanatory (I hope).

bingbot.json and googlecrawlers.json contain Chrome version numbers, which should be updated alongside chrome.json.

Closes #392