PageSpeed Insights provide huge LCP values compared to local LH test runs

varna commented 3 years ago

Provide the steps to reproduce

Run Chrome Audit (LH 6.4.0) on https://agrobaseapp.com/lithuania/pesticide/alister-grande-od
Run PSI (LH 6.3.0) on https://agrobaseapp.com/lithuania/pesticide/alister-grande-od
Compare LCP

What is the current behavior?

Chrome LH Audit has LCP of 1-2 seconds on average:

PSI or web.dev has an average LCP of 4-5 seconds:

What is the expected behavior?

Better LCP score on PSI which would remove "errors" from Google Search Console.

Environment Information

Affected Channels: DevTools, PSI, web.dev
Lighthouse version: 6.4.0 and 6.3.0
Chrome version: 88.0.4324.150 (Official Build) (x86_64)
Node.js version:
Operating System: macOS 11.1

Related issues It would be a lot easier to work with LCP and debug it if timeline was provided over the screenshots with LCP event trigger and LCP target element marked visually

adamraine commented 3 years ago

There are a number of reasons a page might return different results, see https://github.com/GoogleChrome/lighthouse/blob/master/docs/variability.md. One possible reason for variance could be the difference between location you are testing the page from and the location of the server PSI is running on.

That being said, I am consistently getting 1-2s in LH and 4-5s in PSI as well, so we should inspect a trace from PSI to debug this.

jorgellose commented 3 years ago

I experience the same, we have servers in Europe and our clients are as well from europe, it could be related to the place where pagespeed executes the test https://www.sklum.com/es/

varna commented 3 years ago

My site should be using global CDN by Vercel. Tried running LH from Europe and Canada, didn't seem to have much difference. I don't know how to trace PSI, so I can't provide solid proof.

varna commented 3 years ago

Something changed in PSI. Average LCP dropped to 2.2s I check PSI almost daily. I did publish an update for the website a day ago, which improves desktop CLS (set some media queries to avoid layout shift on desktop). I don't think this should have anything to do with the LCP score change for mobile, right?

adamraine commented 3 years ago

@jorgellose the difference in you scores is more likely a result of normal variability

@varna Looks like the chrome version was bumped recently (M85 -> M88). After further investigation, it looks like M85 is giving me a higher (~4s) LCP in LH. Recent changelog for LCP is here. One of those changes probably explains how LCP was different in PSI until now.

midudev commented 3 years ago

Hi there,

from February 19th (1AM CET) we're seeing on our Performance Dashboards as well for almost all our pages a big change on LCP. We're using PageSpeed Insights API.

The LCP time is doubled in almost every site we're checking.

The url we're checking is: https://www.infojobs.net/jobsearch/search-results/list.xhtml?disable-tealium&disable-cmp&mpi_adit_display=enabled:false

If we try PageSpeed Insights website, values seems to be "normal". I though PageSpeed Insights website was using internal API.

PageSpeed Insights is giving us normal values instead the PageSpeed Insights API that is giving us the huge LCP.

I checked the @adamraine explanation about normal variability and changes on LCP for Chrome but still doesn't seem to be enough to explain this bump.

Maybe now a different location is used by default on PageSpeed Insights now?
Maybe some bug has been introduced on LCP?
Definitely something that I'm not catching that is giving us now a huge LCP in all our websites?

PS: No changes on our code on our end for these dates. Latest changes were made two weeks ago (and it's only doing a fetch of the PageSpeed Insights API to send to DataDog).

adamraine commented 3 years ago

It looks like the spike started when LH 7.1.0 was deployed to PSI and the Chrome version was bumped to M88.

I was able to reproduce the spike in LCP from the PSI API one time.

On runs where LCP was ~2s, there was no LCP element listed in largest-contentful-paint-element. On the run where LCP was ~10s the LCP element was this cookie message:

"node": {
    "snippet": "\u003cp\u003e",
    "nodeLabel": "Utilizamos cookies e identificadores propios y de terceros con tu consentimient…",
    "type": "node",
    "boundingRect": {
      "top": 373,
      "width": 552,
      "height": 70,
      "bottom": 443,
      "left": 399,
      "right": 951
    },
    "path": "1,HTML,1,BODY,7,DIV,0,DIV,0,DIV,1,DIV,0,DIV,0,DIV,1,P",
    "selector": "div.sui-MoleculeModal-content \u003e div.sui-TcfFirstLayer-body \u003e div.sui-TcfFirstLayer-info \u003e p",
    "lhId": "page-17-P"
}

Perhaps the changes to handling removed LCP elements in M88 is causing these differences?

connorjclark commented 3 years ago

PageSpeed Insights is giving us normal values instead the PageSpeed Insights API

so this shouldn't be the case, are you comparing the same form factor (mobile vs mobile / desktop vs desktop)? Besides that, any difference between the two can be attributed to expected variance.

EDIT: one caveat: the webapp just uses the PSI API. the API does, however, pick a server based on the IP of the caller. I imagine you are manually using the webapp, but maybe calling the PSI API from some server located elsewhere? if the server is a in a significantly different place you might see extra variance from that, but shouldn't be too extreme.

rishabtewari commented 3 years ago

So what is happening in my case we are getting our LCP in 2s maximum but when we are running Lighthouse report we are getting LCP 13s so it is because in Server side rendered page our LCP comes in 2s but after the hydration when Client Side rendering is there LCP is again switched and we are getting same LCP timing as 13s . So can anybody tell me what we do in this case.

suraj-sundariya commented 9 months ago

Hi @rishabtewari Did you get any solution ?. we are also having similar issue.

@Team in chrome lighthouse dev tool we are getting performance score of 92, everything is in green but PSI is returning 54.

I tried exactly same config in lighthouse dev tool like device : Moto G power network : slow 3G

But still in dev tool we are getting excellent score and not in PSI

JeBuSBrian commented 8 months ago

After checking server logs, I have discovered that PSI does not actually use "Moto G Power" emulation profile. It states that it uses Moto G Power on the PSI page, but that is not true. It is actually using the Moto G (4) emulation profile.

So, Lighthouse using Moto G Power emulation is actually significantly more powerful than what PSI currently does.

I don't know if someone on the PSI-side screwed up a config or what, but the problem is on that end.

goranhorvathr commented 7 months ago

We are currently conducting an investigation into some inconsistencies observed with Total Blockin Time (TBT) and Largest Contentful Paint (LCP) metrics, particularly concerning surprisingly long load times reported for relatively small images (e.g., a 41kb image flagging a 6-second load time) as identified through PageSpeed Insights (PSI). Interestingly, these issues do not seem to arise during Chrome Audits, where performance metrics appear significantly better, and this discrepancy is specifically noted in mobile environments.

In an effort to identify the root cause, we have conducted a thorough review of main thread activity, eliminating any potential bottlenecks we suspected might be contributing to the issue. Despite these efforts, the problem persists. It is worth noting that when analyzing performance through Chrome's performance tab, none of these issues are highlighted, and the website's loading performance—including metrics such as Largest Contentful Paint (LCP), First Contentful Paint (FCP), and First Input Delay (FID)—is exemplary. This high level of performance is consistent with what we observe on real devices.

A peculiar finding in our investigation is that the LCP resource often exhibits a significant "Load Delay." Upon removing the element identified by PSI as problematic, the tool immediately flags the next element, regardless of its nature—be it a single line of text, a paragraph, or a section. This results in PSI attributing unrealistic loading times, sometimes exceeding 13 seconds, and in some cases, reaching up to 50 seconds. It is important to mention that our tests are conducted on Shopify stores.

Initially, we hypothesized that the issue might be due to some form of blockage, but this theory has not led to any conclusive insights, leaving us back at square one. It is also critical to note that other tools, including Chrome Audit, the Google Core Web Vitals (CWV) Library, Performance Observer, and the Google Chrome performance tab, do not flag any of the issues detected by PSI. This discrepancy is puzzling, leading us to question the reliability of these tools and to suspect that there might be a fundamental difference in how PSI evaluates performance metrics.

Given these observations, we are continuing our investigation to pinpoint the cause of these discrepancies and to understand why PSI consistently flags issues not detected by other performance evaluation tools.

JoelLisenby commented 1 week ago

I wanted to confirm that this does appear to be the case, see evidence below:

Pagespeed insights CPU/Memory Power = 274 Lighthouse (Linux CLI) CPU/Memory Power = 3670 Lighthouse (Google Chrome Dev Tools) CPU/Memory Power = 3509

Pagespeed Insights at https://pagespeed.web.dev/:

Lighthouse scores in headless lighthouse on ubuntu server (cli, WSL2 in Windows 11):

Lighthouse scores in Google Chrome dev tools:

GoogleChrome / lighthouse