googleweblight not detected as bot

JayBizzle / Crawler-Detect

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

https://crawlerdetect.io

MIT License

1.99k stars 258 forks source link

googleweblight not detected as bot #460

Closed JPMulder closed 3 years ago

JPMulder commented 3 years ago

Hi I've gone through this list https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers

Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko; googleweblight) Chrome/38.0.1025.166 Mobile Safari/535.19

Was not detected as bot

JayBizzle commented 3 years ago

weblight is not a bot (in the traditional sense).

See here - https://developers.google.com/search/docs/advanced/mobile/web-light#robots

Key info from that page...

The Web Light user agent is used only for explicit browse requests of a human visitor, and so it can ignore robots.txt rules, which are used to block automated crawling requests.

So it is a "bot", but it only visits when triggered by a human request, in which case, you wouldn't want to flag it as a bot.