search

GoogleChrome / web-vitals

Essential metrics for a healthy site.
https://web.dev/vitals
Apache License 2.0
7.55k stars 414 forks source link

Turn off flakey Firefox tests #473

Closed tunetheweb closed 5 months ago

tunetheweb commented 5 months ago

Makes progress on #472 by commenting out the frequently failing tests.

tunetheweb commented 5 months ago

Do you think it's better to comment out particular failures or just disable FF tests until this is fixed? My concern is it will be very easy to forget they've been commented out.

I did just comment out the asserts that failed (the rating checks). Annoyingly the LCP one that's failing is in the assertStandardReportsAreCorrect function used by a lot of the tests so it lost that assert for all of them but I preferred that to pulling that code out for that one test, or disabling the whole test.

I still think that's better to keep most of the Firefox tests working. I plan on leaving #472 open until this is fixed by Firefox as a reminder (hence why it's "makes progress on" rather than "fixes" in the initial comment which would have closed it).

Ultimately I'm hoping this is a short term issue and will be fixed on next version, but getting bored of seeing the failures and worried we will miss a REAL failure because of this.

philipwalton commented 5 months ago

Gotcha. Keeping the issue open as a reminder to address it eventually SGTM.