search

mozilla-mobile / perf-frontend-issues

A repository to hold issues related to front-end mobile application performance.
4 stars 0 forks source link

Discrepancy in FNPRMS: large fennec-nightly improvement on 6-19 does not appear on fenix-nightly #145

Closed mcomella closed 4 years ago

mcomella commented 4 years ago

Screenshot of Samsung S10 results:

image

The discrepancy is worrying: is our methodology correct?

The old Pixel 2 results of 0.55 ms better match my local P2 than the current dashboard times of 0.43 ms.

Then again, we can just switch to mach perf-test and maybe this won't continue to be an issue.

mcomella commented 4 years ago

We recently updated the parsing code assuming fenix-nightly is the only variant that's working (b/c of https://github.com/mozilla-mobile/perf-frontend-issues/issues/137) – I wonder if old results were reinterpreted with this change.

mcomella commented 4 years ago

Triage: there's too much changing with how we measure things right now so we don't care too much: closing.