search

catchpoint / WebPageTest.agent

Cross-platform WebPageTest agent
Other
213 stars 138 forks source link

UAModifier not taken into account in lighthouse #358

Open abarre opened 4 years ago

abarre commented 4 years ago

The UAModifier option is not applied on lighthouse.

I suspect that this PR introduces an issue.

For a bit of context, we use the UAModifier setting to configure bot detector such as Datadome. Currently, the bot detector blocks the lighthouse run. The bot detector doesn't receive the UAModifier string. I suspect that it receive Chrome-Lighthouse instead that is flagged as Bad IP Reputation, DDOS Attack et Fake Chrome Browser.

I think that --chrome-flags user-agent should always be set when the UAModifier is specified regardless of the emulatedFormFactor.

Also, I can see that this commit introduces an option lighthouseConfig. Can you clarify how we can use this option ? It is available in the API ?

After reading https://github.com/GoogleChrome/lighthouse/issues/10910, I am not sure that this can be solved without lighthouse update.

CC @wildlyinaccurate

wildlyinaccurate commented 4 years ago

Did the UA modifier ever work in Lighthouse? In all of my testing Lighthouse would ignore it. The Lighthouse team have always maintained that this is the expected behaviour (https://github.com/GoogleChrome/lighthouse/issues/9927)

wildlyinaccurate commented 4 years ago

FWIW the Chrome-Lighthouse modifier was already present in mobile tests. #297 made sure it was present in desktop tests too. You should be able to whitelist this string in your bot detector.

wildlyinaccurate commented 4 years ago

Sorry @abarre, I forgot to mention how the lighthouseConfig option works. It's a JSON string that is stored in a local file on the test agent and passed to the --config-path option of Lighthouse.

You can generate JSON for the built-in Lighthouse configs relatively easily. For example the PageSpeed mobile config:

> JSON.stringify(require("lighthouse/lighthouse-core/config/lr-mobile-config"))
'{"extends":"lighthouse:default","settings":{"maxWaitForFcp":15000,"maxWaitForLoad":35000,"skipAudits":["uses-http2"]},"audits":["metrics/first-contentful-paint-3g"],"categories":{"performance":{"auditRefs":[{"id":"first-contentful-paint-3g","weight":0}]}}}'