fluttergithubbot commented 3 weeks ago

The post-submit test builder Mac_mokey microbenchmarks had a flaky ratio 13.13% for the past (up to) 100 commits, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_mokey%20microbenchmarks/130 Commit: https://github.com/flutter/flutter/commit/5c5ae033e12a3c19678c3d2b22e3e5b021b90f65

Recent test runs: https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_mokey%20microbenchmarks

Please follow https://github.com/flutter/flutter/blob/master/docs/infra/Reducing-Test-Flakiness.md#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).

jonahwilliams commented 2 weeks ago

xref: https://github.com/flutter/flutter/issues/153367

jonahwilliams commented 2 weeks ago

From investigation, and the bug above these are cases where the apps are failing to install. No additional insight from the logs, though it looks like the devices might be dying during install?

jtmcdole commented 2 weeks ago

Mac_mokey microbenchmarks

INSTALL_FAILED_UPDATE_INCOMPATIBLE

Log:

[2024-08-12 18:03:20.885751] [STDOUT] [STDOUT] [  +67 ms] Installing APK.
[2024-08-12 18:03:20.886797] [STDOUT] [STDOUT] [   +1 ms] Installing build/app/outputs/flutter-apk/app-profile.apk...
[2024-08-12 18:03:20.886839] [STDOUT] [STDOUT] [        ] executing: /opt/s/w/ir/cache/android/sdk/platform-tools/adb -s OZDIUKAAA6RSBUEY install -t -r /opt/s/w/ir/x/w/recipe_cleanup/tmp7yl9bhw9/flutter sdk/dev/benchmarks/microbenchmarks/build/app/outputs/flutter-apk/app-profile.apk
[2024-08-12 18:03:22.944204] [STDOUT] [STDOUT] [+2057 ms] Performing Streamed Install
[2024-08-12 18:03:22.944238] [STDOUT] [STDOUT] 
[2024-08-12 18:03:22.944247] [STDOUT] [STDOUT]            adb: failed to install /opt/s/w/ir/x/w/recipe_cleanup/tmp7yl9bhw9/flutter sdk/dev/benchmarks/microbenchmarks/build/app/outputs/flutter-apk/app-profile.apk: Failure [INSTALL_FAILED_UPDATE_INCOMPATIBLE: Existing package com.yourcompany.microbenchmarks signatures do not match newer version; ignoring!]
[2024-08-12 18:03:22.944396] [STDOUT] [STDOUT] [        ] Installing build/app/outputs/flutter-apk/app-profile.apk... (completed in 2,057ms)
[2024-08-12 18:03:22.946686] [STDERR] [STDERR] [        ] Error: ADB exited with exit code 1
[2024-08-12 18:03:22.946798] [STDERR] [STDERR] [   +1 ms] Performing Streamed Install
[2024-08-12 18:03:22.946821] [STDERR] [STDERR] 
[2024-08-12 18:03:22.946830] [STDERR] [STDERR]            adb: failed to install /opt/s/w/ir/x/w/recipe_cleanup/tmp7yl9bhw9/flutter sdk/dev/benchmarks/microbenchmarks/build/app/outputs/flutter-apk/app-profile.apk: Failure [INSTALL_FAILED_UPDATE_INCOMPATIBLE: Existing package com.yourcompany.microbenchmarks signatures do not match newer version; ignoring!]
[2024-08-12 18:03:22.946861] [STDOUT] [STDOUT] [        ] Warning: Failed to install APK.
[2024-08-12 18:03:22.947438] [STDERR] [STDERR] [        ] Error running application on mokey.

These don't seem to wait for the following to complete, at least according to the log files.

[2024-08-13 05:12:30.732368] [STDOUT] [STDOUT] [        ] executing: /opt/s/w/ir/cache/android/sdk/platform-tools/adb -s OZDIUKAAA6RSBUEY shell am force-stop com.yourcompany.microbenchmarks
[2024-08-13 05:12:30.797037] [STDOUT] [STDOUT] [  +64 ms] executing: /opt/s/w/ir/cache/android/sdk/platform-tools/adb -s OZDIUKAAA6RSBUEY shell pm list packages com.yourcompany.microbenchmarks

69
58
50
46
130
128
125
121
114
108
104
102

Missing device

The follow are just transient devices failures

52: No Android devices detected

Looks like a devicelab specific problem:

flutter-devicelab-mac-18 https://github.com/flutter/flutter/issues/154114

https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_mokey%20microbenchmarks/259/overview https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_mokey%20microbenchmarks/250/overview https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_mokey%20microbenchmarks/249/infra https://luci-milo.appspot.com/ui/p/flutter/builders/prod/Mac_mokey%20run_debug_test_android/248/overview

jtmcdole commented 2 weeks ago

I think we ought to specifically call adb uninstall rather than adb install -r because the latter keeps the data around and doesn't work if the app was signed differently. Will investigate.

zanderso commented 2 weeks ago

@kentnguyen99 reports that the device attached to mac-18 is sporadically rebooting. He is removing it from the lab and will replace it with an underutilized device from the staging pool.

jtmcdole commented 2 weeks ago

Updates from digging a bit more:

It is waiting on the package list; but the list either is empty or doesn't have the microbenchmark package name
The double output is not "we tried to install twice"; its stdout and stderr being dumped by RunResult (blah).

In the case of this error we don't think the app is installed from pm list packages, but we very clearly know its true from the logs of adb saying INSTALL_FAILED_UPDATE_INCOMPATIBLE.

So it would be nice:

Look at the ADB error and maybe devine the exit reason and pass it back so we could action on this (by uninstalling).
Output some more debug logs from some of the adb commands to divine if they are having problems (pm package list)

Will look at it some more tomorrow.

zanderso commented 2 weeks ago

The bad device on mac-18 has been swapped out for a different one.

fluttergithubbot commented 2 weeks ago

Recent test runs: https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_mokey%20microbenchmarks

jtmcdole commented 2 weeks ago

Mac_mokey microbenchmarks can (maybe) be solved by adding an uninstall outside of the loop. Mac_ios microbenchmarks_ios started timing out because installing or uninstalling takes ~13 seconds per test. There are 21 tests. The fix that was submitted (and then reverted) had the uninstall before each test, pushing this over the limit.

Moving to just one uninstall outside of the loop gets really dang close to the limit of a test suite that is already on the edge of failing (14:53). So instead I'm going to propose making the "micro benchmarks" compile together and run one after the other. This should reduce the testing time on mac by at least 5 minutes.

goderbauer commented 1 week ago

(triage): We are waiting for another bot report, expecting it to go down.

jtmcdole commented 1 week ago

The last 100 builds are clean. Just waiting now.

fluttergithubbot commented 1 week ago

[prod pool] flaky ratio for the past (up to) 100 commits between 2024-08-24 and 2024-08-31 is 2.00%. Flaky number: 2; total number: 100. One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_mokey%20microbenchmarks/259 Commit: https://github.com/flutter/flutter/commit/9308a799c6e3d1aaca4d473d4828990e6369f77e Flaky builds: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_mokey%20microbenchmarks/259 https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_mokey%20microbenchmarks/250

Recent test runs: https://flutter-dashboard.appspot.com/#/build?taskFilter=Mac_mokey%20microbenchmarks

fluttergithubbot commented 3 days ago

[prod pool] flaky ratio for the past (up to) 100 commits between 2024-09-03 and 2024-09-10 is 0.00%. Flaky number: 0; total number: 100.

flutter / flutter