omrilotan / isbot

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string
https://isbot.js.org/
The Unlicense
876 stars 72 forks source link

The package does not work anymore because in Lighthouse aser agent does not include "Chrome Lighthouse"? #224

Closed Appolinars closed 7 months ago

Appolinars commented 7 months ago

Steps to reproduce

In Next JS application I tried to conditionaly load Third Part Scripts so they won't affect my Page Speed. image

Basically I just check if it's bot and return null: image

But it does not work, PageSpeed Insights still sees my scripts. I found this issue on GoogleChrome's Github: https://github.com/GoogleChrome/lighthouse/issues/14917

So do I understand correctly that this package does not work anymore with Lighthouse?

omrilotan commented 7 months ago

Thank you, @Appolinars . I will review and research on Monday so we can better decide what actions we can provide here.

omrilotan commented 7 months ago

I was not able to reproduce this issue.

My use of https://pagespeed.web.dev produces requests with User-Agent header: Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse

Since this package still uses User-Agent header for bot recognition, there's not much to be done within the perimeter of this package. I will refer you to clarifications section of the readme.

However, you can use reverse dns lookup to determine whether the source of the request is Google's servers:

import isbot from "isbot";
import { verify } from "reverse-dns-lookup";
import requestIP from "request-ip";

const clientIp = requestIP.getClientIp(request);
const isGoogleServer = await verify(clientIp, "google.com");

const knwonBot = isGoogleServer || isbot(request.headers["user-agent"]);
Appolinars commented 7 months ago

I was not able to reproduce this issue.

My use of https://pagespeed.web.dev produces requests with User-Agent header: Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4590.2 Mobile Safari/537.36 Chrome-Lighthouse

Since this package still uses User-Agent header for bot recognition, there's not much to be done within the perimeter of this package. I will refer you to clarifications section of the readme.

However, you can use reverse dns lookup to determine whether the source of the request is Google's servers:

import isbot from "isbot";
import { verify } from "reverse-dns-lookup";
import requestIP from "request-ip";

const clientIp = requestIP.getClientIp(request);
const isGoogleServer = await verify(clientIp, "google.com");

const knwonBot = isGoogleServer || isbot(request.headers["user-agent"]);

Thanks for your reply