search

alaz / legitbot

🤔 Is this Web request from a real search engine🕷 or from an impersonating agent 🕵️‍♀️?
Other
24 stars 11 forks source link

Add Chrome-Lighthouse to Google crawlers list #160

Closed aydinkazim closed 1 month ago

alaz commented 1 month ago

Thank you!

alaz commented 1 month ago

Hey @aydinkazim , I can't seem to find where this User Agent is documented. What I found is https://github.com/GoogleChrome/lighthouse/issues/14917 and it seems that the User Agent has been removed. I need a documentation reference.

aydinkazim commented 1 month ago

Hi again 👋

You're right, the official documentation shows that the Chrome-Lighthouse user agent has been removed, as mentioned in GoogleChrome/lighthouse#14917. However, based on the current logs from my bot, it's still utilizing the Chrome-Lighthouse user agent. Here's a snippet from my logs:

I, [2024-10-24T16:20:06.053244 #208883]  INFO -- : 
 REAL BOT IP: 74.125.208.7 AGENT: Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse

I, [2024-10-24T16:20:06.539276 #208883]  INFO -- : 
 REAL BOT IP: 74.125.208.8 AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Safari/537.36 Chrome-Lighthouse

I, [2024-10-24T16:20:06.704677 #208883]  INFO -- : 
 REAL BOT IP: 74.125.208.7 AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Safari/537.36 Chrome-Lighthouse

I, [2024-10-24T16:20:06.878626 #208883]  INFO -- : 
 REAL BOT IP: 74.125.208.7 AGENT: Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse

I, [2024-10-24T16:20:19.868516 #208883]  INFO -- : 
 REAL BOT IP: 74.125.208.7 AGENT: Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse

It seems that despite the removal in official references, the bot is still functioning with this user agent. I will continue monitoring this, but it may be a transitional state from Google Lighthouse.

alaz commented 1 month ago

Thank you for confirming. It does not look from the issue that they would consider reintroducing it back, but let's see. I do not see any harm in adding it to Legitbot meanwhile. I am going to publish a release by the weekend.

aydinkazim commented 1 month ago

Thank you for the update! I completely understand, and I agree that reintroducing it doesn't seem likely based on the current discussions. Looking forward to the upcoming release!

alaz commented 1 month ago

Published as 1.11.1