GoogleChrome / web-vitals

Essential metrics for a healthy site.
https://web.dev/vitals
Apache License 2.0
7.46k stars 409 forks source link

Firefox e2e tests flakey #472

Open tunetheweb opened 4 months ago

tunetheweb commented 4 months ago

Looks in Firefox Nightly FCP is often quite slow in our e2e tests.

For example this run has this when I turn on debugging:

[0-1] reports the correct value after the first paint {
[0-1]   name: 'FCP',
[0-1]   value: 53[15](https://github.com/GoogleChrome/web-vitals/actions/runs/8923549678/job/24508034241#step:6:16),
[0-1]   rating: 'poor',
[0-1]   delta: 5315,
[0-1]   entries: [
[0-1]     {
[0-1]       name: 'first-contentful-paint',
[0-1]       entryType: 'paint',
[0-1]       startTime: 5315,
[0-1]       duration: 0
[0-1]     }
[0-1]   ],
[0-1]   id: 'v4-[17](https://github.com/GoogleChrome/web-vitals/actions/runs/8923549678/job/24508034241#step:6:18)14651853858-9340183650873',
[0-1]   navigationType: 'navigate'
[0-1] }

A rerun often solves it as you can see later in that log:

[0-1] reports the correct value after the first paint { [0-1] name: 'FCP', [0-1] value: 72, [0-1] rating: 'good', [0-1] delta: 72, [0-1] entries: [ [0-1] { [0-1] name: 'first-contentful-paint', [0-1] entryType: 'paint', [0-1] startTime: 72, [0-1] duration: 0 [0-1] } [0-1] ], [0-1] id: 'v4-1714651859617-3386553824299', [0-1] navigationType: 'navigate' [0-1] }



But it doesn't seem to solve it for these tests which have been continually failing:
- FCP - 'reports if the page is restored from bfcache'
- LCP - 'reports the correct value on hidden (reportAllChanges === false)'

I hoped it was just some nightly flakiness but doesn't seem to be getting resolved so we might want to comment these out for now for Firefox.

I can't reproduce this on MacOS, nor on linux (Chromebox) so it only seems to happen in GitHub Actions. Possibly related to there not being a display there?
tunetheweb commented 4 months ago

Raised a bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1894695

tunetheweb commented 4 months ago

So #476 resolved a lot of the cases for now but this one is still susceptible to failing intermittently due to the Firefox slowness in GitHub Actions:

https://github.com/GoogleChrome/web-vitals/blob/fd8b0c3a59089c080fdb3ed9297a9e73867f4963/test/e2e/onLCP-test.js#L715-L718

A rerun often fixes it but might take a few attempts. If it gets really annoying we can consider skipping this last line out for Firefox too.

tunetheweb commented 4 months ago

Either way leaving this open to remind us to remove those workarounds once the real fix is released.