Gather baseline perf tests for Focus

mozilla-mobile / perf-frontend-issues

A repository to hold issues related to front-end mobile application performance.

4 stars 0 forks source link

Gather baseline perf tests for Focus #211

Closed mcomella closed 2 years ago

mcomella commented 3 years ago

For Focus v8.16.0, run baseline performance tests:

page load (ask acreskey or bdekoz how to run page load tests or to run it on our behalf. maybe the latter is preferable)
startup (modify measure_start_up.py)
app size (this should probably come from the GPlay dashboard b/c that's how we measure fenix)

mcomella commented 3 years ago

I took some preliminary start up measurements with a WIP progress version of the measure_start_up.py script. Here's what I noticed:

From v8.16.0 to v8.17.1, which added the major architecture change, there was no major change in time to first frame for MAIN or VIEW. I was unable to measure nav start because the logcat was removed in Oct 2020.
From v8.8.1 (Oct 2020) to v8.16.0, which was an arbitrary version that still included PageStart markers:
- MAIN regressed 22ms (could be noise)
- VIEW regressed 169ms

Next action items are to get this reviewed and add the logcat markers we use for nav start back into the new versions of focus for a proper comparison of nav start.

mcomella commented 3 years ago

By comparison to v8.17.1 of focus to Nightly 2021-07-15 91.0a1 of fenix (only 2 iterations on fenix) on my Moto G5:

MAIN first frame is 762ms slower in fenix
VIEW first frame is 708ms slower in fenix
VIEW nav start is 223.5ms faster in fenix than v8.8.1 of focus

mcomella commented 3 years ago

The script was accepted as is and merged in https://github.com/mozilla-mobile/fenix/pull/20463#event-5071287291.

What's remaining for this PR is: wait for my Moto G5 replacement battery to arrive and take the measurements.

mcomella commented 3 years ago

To measure NavStart on subsequent focus builds, I filed https://github.com/mozilla-mobile/focus-android/issues/5021

mcomella commented 3 years ago

I collected preliminary start up and app size data in this spreadsheet. wrt app size, it seems the GH releases numbers roughly align with the GPlay dashboard download sizes (GPlay is a MB or two smaller). I additionally discovered app updates are very small in focus so I filed https://github.com/mozilla-mobile/focus-android/issues/5031.

mcomella commented 3 years ago

I asked bdekoz to take page load measurements for:

Before changes: Focus v8.16.0
After changes: v8.18.0-rc.1 (or later)

mcomella commented 3 years ago

I have the final measurements for page load and app size and included it in my final report. I want to get a quick sanity check before sharing it out though.

mcomella commented 3 years ago

After feedback, I created a simpler report: https://docs.google.com/document/d/1PciDeCl98C0dhZ8t67h9TAH3xlqaESuUnDcf8729f9E/edit# We still need to wait on page load numbers though.

mcomella commented 3 years ago

We backported the onboarding override in https://github.com/mozilla-mobile/focus-android/issues/5364. We built the release builds and are just waiting to run the results.

mcomella commented 2 years ago

I think there are consistent conversations between the fenix team and perf so if this was needed, it would have come up and been prioritized. Since it doesn't seem to have been, I think it's probably not important right now. Closing.