search

monperrus / crawler-user-agents

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:
MIT License
1.17k stars 250 forks source link

Added Google+ crawler #261

Closed sylling closed 3 years ago

sylling commented 4 years ago

Added:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google (+https://developers.google.com/web/snippet/)

JayBizzle commented 4 years ago

This seems a bit generic to 100% confirm as a bot!?

floyd-jones commented 4 years ago

Seems like it but it isn't, here is another one from a google pixel Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)

here are links that confirm it: https://support.google.com/webmasters/thread/34525120?hl=en https://user-agents.net/string/mozilla-5-0-x11-linux-x86-64-applewebkit-537-36-khtml-like-gecko-chrome-56-0-2924-87-safari-537-36-google-https-developers-google-com-web-snippet

monperrus commented 4 years ago

FTR, https://developers.google.com/+/web/snippet/ is 404, it's probably too confidential to be merged for now.

monperrus commented 3 years ago

Closing stale PR, thanks for the proposal!