search

mozilla-mobile / firefox-ios

Firefox for iOS
Mozilla Public License 2.0
12.14k stars 2.91k forks source link

Investigate and compile a list of flaky tests #21368

Open data-sync-user opened 1 month ago

data-sync-user commented 1 month ago

To investigate and create list of tickets for flaky tests. As well as run locally and capture some initial investigation notes. For now, focusing on the top 10 tests.

Based on the Bitrise:

[+https://app.bitrise.io/insights/095364a9afb324b2/explore/tests?interval=daily&durati[…]r_by=failure_count&failure_rate_order_by_direction=descending+|https://app.bitrise.io/insights/095364a9afb324b2/explore/tests?interval=daily&duration=P30D&app_slug=6c06d3a40422d10f&workflow=firefox_configure_build&pipeline=pipeline_build_and_test&tab=failure_rate&secondary_tab=test_cases&failure_rate_order_by=failure_count&failure_rate_order_by_direction=descending]

|Name|Class|Pass Locally?|Tech Notes|

|testCompileListsNotInStore_callsCompletionHandlerSuccessfully()|ContentBlockerTests|Yes|This is testing the closure, not sure why its failing but maybe mocking out the ruleStore. Need to look into best way to test GCD methods. Test seems to be crashing/timing out since performance measurements added in https://mozilla-hub.atlassian.net/browse/FXIOS-9270 aren’t recorded on fail| |testQueryLogins()|PasswordManagerViewModelTests|No !image-20240801-195314.png|width=645,height=81,alt="image-20240801-195314.png"!|https://mozilla-hub.atlassian.net/browse/FXIOS-9762 Fix: Separate tests into its own individual methods. Unable to reproduce flakey tests with this change. Ran 1000 times.

PR Merged, will keep monitoring.| |testPocketStoriesCaching()|PocketStoriesTests|Yes|This seems to always fail with the testQueryLogins test, hoping fixing that will cause this to not error for some reason. Not sure why one error bleeds into another though. Can follow up with automation team on this. https://app.bitrise.io/build/0277307c-c3a1-4deb-a659-a8053b2126f3| |test_syncLogin_NimbusIsCalled() test_loginsSaved_GleanIsCalled()|TelemetryWrapperTests|: - Crash: Client (18324) AppContainer.resolve<A>()|Crash with AppContainer https://app.bitrise.io/build/3dca0a33-4dee-416c-a74e-5b12264cca31| |testTapHome() testSetsInitialImages() testSearchStateImages() test_tabToolBarHelper_basicCreation_doesntLeak()|TabToolbarHelperTests|No https://app.bitrise.io/insights/095364a9afb324b2/explore/tests?interval=daily&duration=P30D&app_slug=6c06d3a40422d10f&workflow=firefox_configure_build&pipeline=pipeline_build_and_test&tab=failure_rate&secondary_tab=related_test_case_runs&failure_rate_order_by=failure_count&failure_rate_order_by_direction=descending&test_case_name=testTapHome()&test_suite_name=ClientTests&module=TabToolbarHelperTests|MockTabToolbar is not using MockTabManager, investigate why not and may resolve the TabToolbarHelperTests related tests.

See crash with AppContainer here too.| |-test_removeAllTabs_savesCorrectBackupCloseTab()- test_removeAllTabs_hasOneHomeTabAfter()|-TabManagerTests-| |-Not existent in main-| |testTabDidGainFocus_highThrottleTime_doesntSync testTabDidGainFocus_multipleThrottle_withoutWaitdoesntSync|AccountSyncHandlerTests|!image-20240802-204958.png|width=972,height=200,alt="image-20240802-204958.png"!|This was due to throttle code changing, but the tests not being updated. Updated the tests so that it follows more of the expectations from Throttle code. https://app.bitrise.io/insights/095364a9afb324b2/explore/tests?interval=daily&duration=P30D&tab=failure_rate&app_slug=6c06d3a40422d10f&pipeline=pipeline_build_and_test&workflow=firefox_configure_build&module=AccountSyncHandlerTests&breakdown=ClientTests&display_option=top10&secondary_tab=test_cases https://mozilla-hub.atlassian.net/browse/FXIOS-9772 Fix: Update tests to match our throttler code.|

┆Issue is synchronized with this Jira Spike

data-sync-user commented 1 month ago

➤ Isabel Rios Escobar commented:

Let me add a link to one issue already open for one test: https://github.com/mozilla-mobile/firefox-ios/issues/20525 ( https://github.com/mozilla-mobile/firefox-ios/issues/20525|smart-link )

data-sync-user commented 1 month ago

➤ Cyndi Chin commented:

Weekly Statues (8/5)

data-sync-user commented 1 month ago

➤ Cyndi Chin commented:

New observed flaky test: testClickingDiscoverCell_callsTapTileAction https://app.bitrise.io/build/5e697aaf-46d7-468c-8d96-45fb0fb43f05 ( https://app.bitrise.io/build/5e697aaf-46d7-468c-8d96-45fb0fb43f05|smart-link )

data-sync-user commented 3 weeks ago

➤ Cyndi Chin commented:

Updates and new observations will be documented in this spreadsheet

https://docs.google.com/spreadsheets/d/1ZUlPdZem-JQV6t4u0lnoETkmqg5aKOn6_s2vdd2-Oxg/edit?gid=1728646075#gid=1728646075 ( https://docs.google.com/spreadsheets/d/1ZUlPdZem-JQV6t4u0lnoETkmqg5aKOn6_s2vdd2-Oxg/edit?gid=1728646075#gid=1728646075|smart-link )

data-sync-user commented 3 weeks ago

➤ Cyndi Chin commented:

Closing this ticket as done as majority of the flaky tests are addressed based on the ones that were occurring majority of the time. Future flaky tests will be added the spreadsheet instead.