search

mozilla-mobile / focus-android

⚠️ Firefox Focus (Android) moved to a new repository. It is now developed and maintained as part of: https://github.com/mozilla-mobile/firefox-android
https://github.com/mozilla-mobile/firefox-android
Mozilla Public License 2.0
2.11k stars 711 forks source link

33ms regression to cold main first frame (10/1) #7925

Closed mcomella closed 1 year ago

mcomella commented 1 year ago

image

iorgamgabriel commented 1 year ago

I tested this on a Samsung A51 with Android 12.0 with Power saving mode activated and some of the tests with Pixel 3 XL . I didn't find a very big difference in the results. The tests were done using at least 100 runs because there was a lot of variation between results and 25 or 50 runs would not give consistent/reliable mean/median values.

Following the steps from https://github.com/mozilla-mobile/perf-tools I tested with using

python3 measure_start_up.py -c=100 --product=focus nightly cold_main_first_frame

Based on the reported regression(https://earthangel-b40313e5.influxcloud.net/d/7u0o-gdnz/focus-startup-nightly-via-backfill-py?orgId=1&from=now-90d&to=now) time range I focused my tests on Focus on the following PRs merged between September 29-October 03. But I tested with a larger range like (https://github.com/mozilla-mobile/focus-android/pulls?q=is%3Apr+is%3Aclosed+merged%3A2022-09-27..2022-10-06+base%3Amain+sort%3Aupdated-asc+) only the first and the last pull requests.

Results on Focus for 2022-09-29 to 2022-10-03:

Merge day Revision # Title # runs Max Mean Median Min Good Graph
29 Sep ff3c1bb For 6531 new shortcutsDoNotOpenInNewTabTest UI smoke test 100 843.0 735.35 732.0 699.0 ? link
29 Sep 16c1e96 Update to Android-Components 107.0.20220929143100 100 802.0 739.94 737.0 707.0 ? link
29 Sep 36bbe74 Update to Android-Components 107.0.20220929190135 100 833.0 741.22 737.0 708.0 ? link
1 Oct 80163c3 Update to Android-Components 107.0.20220930190122 100 860.0 749.14 746.0 719.0 ? link
2 Oct 30fc0b5 Update to Android-Components 107.0.20221002190240 100 799.0 746.46 746.0 706.0 ? link
3 Oct cfb2c10 For #7580 re-enable Open custom tab in focus UI test 100 819.0 743.94 739.5 704.0 ? link

Results on Focus for 2022-09-27 to 2022-10-06:

Merge day Revision # Title # runs Max Mean Median Min Good Graph
27 Sep d08ddf9 Import strings from android-l10n. 100 786.0 731.98 730.0 693.0 ? link
6 Oct 4a2c621 Remove extraneous coroutines_version 100 807.0 734.74 732.0 701.0 ? link

As you can see all the values for Median are around 740 . @mcomella Seems like the results are stable and no clear regression is shown. Do you suggest any other ways to catch the issue?

cpeterso commented 1 year ago

Moved to bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1802598

Change performed by the Move to Bugzilla add-on.