omrilotan / isbot

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string
https://isbot.js.org/
The Unlicense
876 stars 72 forks source link

"Google-Read-Aloud" was wrongly recognised as a bot #222

Closed TigersWay closed 8 months ago

TigersWay commented 8 months ago

User Agent String = "Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)"

Reproduce = test

If I believe Google, this is just an "helper" not really a bot.

Thanks!

omrilotan commented 8 months ago

Thank you for this issue.

Google marks "crawlers" and "fetchers" similarly. See https://support.google.com/webmasters/answer/1061943.

I will take the time to review if we can differentiate between the two based on user agent string patterns or rethink how we address all of Google's tools to use a more granular approach, which may have performance costs.

I will circle back here after I have made some initial testing.

TigersWay commented 8 months ago

I read that "answer" before, and honestly they are all - but one - bots! Not really bad bots (but we could argue on this 😄 ), but bots anyway... There is only one logical exception, that "Google-Read-Aloud" helper. I wish some user(s) could tell us more.

TigersWay commented 8 months ago

Just to keep the info: Google Read Aloud user agent

with this text: Google Read Aloud is not a web crawler: it is activated upon user request, and it doesn't follow links.

omrilotan commented 8 months ago

A user initiates it, but the request triggers a potential S2S request from Google's Web Proxy Server (google-proxy-<IPV4_ADDRESS>.google.com). This is why the server self-recognises itself as a bot. Due to this practice, this must still be considered a (very welcome) bot.

omrilotan commented 8 months ago

According to the patterns I've observed, I will deem Google's decision to mark this with the rest of the bots appropriate. I'm closing this ticket, but I am open to continuing the discussion if you feel differently.