Linux_mac_unopt build has a recent history of a number of infra failures

flar commented 6 months ago

Type of Request

infra task

Infrastructure Environment

LUCI, Github

What is happening?

https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20mac_unopt?limit=100

Recent tries have had a number of failures due to either ios_debug_sim or ios_debug_unopt_sim both failing with an error Cannot test target “ScenariosUITests” on “Any iOS Device”: Tests must be run on a concrete device

Steps to reproduce

No response

Expected results

No response

vashworth commented 6 months ago

So this happens when there are multiple simulators that match the destination criteria.

So for example, https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20Engine%20Drone/724612/overview

The criteria is -destination "platform=iOS Simulator,OS=17.0,name=iPhone SE (3rd generation)"

However, there are 2 simulators matching those criteria (see bottom 2 in list).

--- xcodebuild: WARNING: Using the first of multiple matching destinations:
 { platform:iOS, id:dvtdevice-DVTiPhonePlaceholder-iphoneos:placeholder, name:Any iOS Device }
 { platform:iOS Simulator, id:dvtdevice-DVTiOSDeviceSimulatorPlaceholder-iphonesimulator:placeholder, name:Any iOS Simulator Device }
 { platform:iOS Simulator, id:04D0CF83-37E7-4A9F-AFD6-95810D211DDA, OS:17.0, name:TestAdd2AppSim }
 { platform:iOS Simulator, id:1CD40700-B3BE-4C2E-A2A3-19D2A38A4FC7, OS:17.0, name:iPad (10th generation) }
 { platform:iOS Simulator, id:92ACF8F8-B569-4E98-8A70-BAF00AD4F085, OS:17.0, name:iPad (10th generation) }
 { platform:iOS Simulator, id:DEC101C4-C4D1-41BD-9387-6423F4ACC422, OS:17.0, name:iPad Air (5th generation) }
 { platform:iOS Simulator, id:5BD9BD20-7408-45B8-8D5C-4B5462B4B832, OS:17.0, name:iPad Pro (11-inch) (4th generation) }
 { platform:iOS Simulator, id:8CD15AE9-61B4-4325-9EC8-0C5452326548, OS:17.0, name:iPad Pro (11-inch) (4th generation) }
 { platform:iOS Simulator, id:3AF13222-E25F-4AAD-9B25-313B546160A5, OS:17.0, name:iPad Pro (12.9-inch) (6th generation) }
 { platform:iOS Simulator, id:A29340A6-7253-4BC6-B802-82911F4A206C, OS:17.0, name:iPad Pro (12.9-inch) (6th generation) }
 { platform:iOS Simulator, id:26DCF2DC-28E2-4391-8692-2A095B834051, OS:17.0, name:iPad mini (6th generation) }
 { platform:iOS Simulator, id:B58F5198-FE10-444F-9426-8BABA4FD625B, OS:17.0, name:iPad mini (6th generation) }
 { platform:iOS Simulator, id:D52F1164-907E-495B-837C-D42545DCFE22, OS:17.0, name:iPhone 14 }
 { platform:iOS Simulator, id:4DAFDB51-3728-4A43-B99B-64A3D85E059A, OS:17.0, name:iPhone 14 Plus }
 { platform:iOS Simulator, id:AAA38124-9954-4566-9500-917734C9C0B9, OS:17.0, name:iPhone 14 Pro }
 { platform:iOS Simulator, id:3FD1A3D9-B78B-42F7-8066-75C1EF6102F9, OS:17.0, name:iPhone 14 Pro Max }
 { platform:iOS Simulator, id:051BC11B-6054-41D7-A0AB-01696DF133A8, OS:17.0, name:iPhone 15 }
 { platform:iOS Simulator, id:CCFE466C-6647-4D96-9A23-A2E518BFBBFC, OS:17.0, name:iPhone 15 }
 { platform:iOS Simulator, id:4744DE1E-8172-4D57-8625-7B2F289E0981, OS:17.0, name:iPhone 15 Plus }
 { platform:iOS Simulator, id:7C56DD58-96A0-4F64-8B83-0055C313209D, OS:17.0, name:iPhone 15 Plus }
 { platform:iOS Simulator, id:79B8084D-1D9E-4CA7-B113-5D42A648530D, OS:17.0, name:iPhone 15 Pro }
 { platform:iOS Simulator, id:7A142D05-9F1D-401B-B01E-B141E8CF21E0, OS:17.0, name:iPhone 15 Pro }
 { platform:iOS Simulator, id:8BA38FEC-DBFB-4975-ADC0-98951ACDDA9F, OS:17.0, name:iPhone 15 Pro Max }
 { platform:iOS Simulator, id:E458655E-3E57-4C1B-AFA8-BA866C7737A4, OS:17.0, name:iPhone 15 Pro Max }
 { platform:iOS Simulator, id:89E04F53-A08E-42DF-8B77-C41C9CA3F315, OS:17.0, name:iPhone SE (3rd generation) }
 { platform:iOS Simulator, id:C272E0A6-884D-412F-82C8-74797D994E00, OS:17.0, name:iPhone SE (3rd generation) }

You'd think it'd be smart enough to use the first that matches all the criteria, but it seems to ignore the name when it selects from multiple.

The fix is to delete any simulators with the name first, then create a simulator with the name, then run the test.

Needs to be done before here: https://github.com/flutter/engine/blob/main/testing/scenario_app/run_ios_tests.sh#L64-L70

We do this in packages for this exact problem: https://github.com/flutter/packages/blob/60a1ffc2f77fade3e375a98177729b2f3e90e06e/.ci/scripts/create_simulator.sh#L14-L27

keyonghan commented 6 months ago

Needs to be done before here: https://github.com/flutter/engine/blob/main/testing/scenario_app/run_ios_tests.sh#L64-L70

Seems this is a test by test fix. Will other tests from framework/engine potentially experience similar issues?

vashworth commented 6 months ago

Needs to be done before here: https://github.com/flutter/engine/blob/main/testing/scenario_app/run_ios_tests.sh#L64-L70

Seems this is a test by test fix. Will other tests from framework/engine potentially experience similar issues?

Yes, they could. From my knowledge, though, most other tests are already setup to do this.

Some examples:

I'm not sure if there's like a catch-all solution we could do since different tests use different simulator types, names, etc - but open to ideas.

vashworth commented 6 months ago

Perhaps we could delete all simulator devices in recipes before every build and rely on the test to create its needed simulator device.

keyonghan commented 6 months ago

Perhaps we could delete all simulator devices in recipes before every build and rely on the test to create its needed simulator device.

What would be the time cost to delete all simulator devices and re-create? Or delete only when >1 devices exist.

flar commented 6 months ago

I think the tests should specify a simulator and manually create it if it doesn't exist. Running with "whatever the bot has installed" could lead to inconsistent test results.

vashworth commented 6 months ago

What would be the time cost to delete all simulator devices and re-create? Or delete only when >1 devices exist.

It's hard to say for sure without testing it, but probably less than 5 seconds for the most part. The initial delete all devices may take longer, but after that there shouldn't be a ton of devices to delete (probably only one) since it would delete on each build.

Example: See step 12.15 (which attempts to delete and then creates simulator) - takes 3.9 seconds: https://ci.chromium.org/ui/p/flutter/builders/try/Mac_arm64%20custom_package_tests%20master/10823/overview

keyonghan commented 6 months ago

I think the tests should specify a simulator and manually create it if it doesn't exist. Running with "whatever the bot has installed" could lead to inconsistent test results.

It makes sense to me. What I mentioned earlier (as @vashworth said) is we clean up the devices and install all. From test side, you still use the defined simulator version and nothing should be affected. The difference is test side will not see multiple entries.

keyonghan commented 6 months ago

Example: See step 12.15 (which attempts to delete and then creates simulator) - takes 3.9 seconds: https://ci.chromium.org/ui/p/flutter/builders/try/Mac_arm64%20custom_package_tests%20master/10823/overview

3.9 is okay. Let's see if https://github.com/flutter/engine/pull/50835 covers the last corner case. If not, we may consider implementing from recipes side to cover all.

vashworth commented 6 months ago

It makes sense to me. What I mentioned earlier (as @vashworth said) is we clean up the devices and install all. From test side, you still use the defined simulator version and nothing should be affected. The difference is test side will not see multiple entries.

We actually wouldn't "install all" (I don't think there is a way to do that), we'd only delete all. So for a test to use a simulator, it will need to create it. If a test tries to use a simulator without first creating it, it would fail.

https://github.com/flutter/engine/pull/50835 should definitely handle this particular case, but if we want a more consistent way of handling this across all tests (engine/framework), then we may want to do the recipes side.

github-actions[bot] commented 6 months ago

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

flutter / flutter