search

monperrus / crawler-user-agents

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:
MIT License
1.13k stars 243 forks source link

Feature request virustotal #281

Closed akuankkan closed 2 years ago

akuankkan commented 2 years ago

Please add VirusTotal crawlers. They have several bots like next. You can check it by yourself on https://www.virustotal.com/gui/home/url:

  1. Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/49.0.2623.75 Safari\/537.36 Google Favicon
  2. Mozilla\/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http:\/\/code.google.com\/appengine; appid: s~virustotalcloud)
  3. AppEngine-Google; (+http:\/\/code.google.com\/appengine; appid: s~virustotalcloud)
  4. "Mozilla\/5.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident\/4.0; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)
  5. AppEngine-Google; (+http:\/\/code.google.com\/appengine; appid: s~virustotalcloud) As well https://redirectdetective.com/ and https://wheregoes.com/
monperrus commented 2 years ago

Great idea. Would you be able to make a pull-request?

akuankkan commented 2 years ago

I am not a dev, my team just using this list for our project. Would appreciate if you add it by yourself :)