Closed flar closed 6 months ago
So this happens when there are multiple simulators that match the destination criteria.
So for example, https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20Engine%20Drone/724612/overview
The criteria is -destination "platform=iOS Simulator,OS=17.0,name=iPhone SE (3rd generation)"
However, there are 2 simulators matching those criteria (see bottom 2 in list).
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:iOS, id:dvtdevice-DVTiPhonePlaceholder-iphoneos:placeholder, name:Any iOS Device }
{ platform:iOS Simulator, id:dvtdevice-DVTiOSDeviceSimulatorPlaceholder-iphonesimulator:placeholder, name:Any iOS Simulator Device }
{ platform:iOS Simulator, id:04D0CF83-37E7-4A9F-AFD6-95810D211DDA, OS:17.0, name:TestAdd2AppSim }
{ platform:iOS Simulator, id:1CD40700-B3BE-4C2E-A2A3-19D2A38A4FC7, OS:17.0, name:iPad (10th generation) }
{ platform:iOS Simulator, id:92ACF8F8-B569-4E98-8A70-BAF00AD4F085, OS:17.0, name:iPad (10th generation) }
{ platform:iOS Simulator, id:DEC101C4-C4D1-41BD-9387-6423F4ACC422, OS:17.0, name:iPad Air (5th generation) }
{ platform:iOS Simulator, id:5BD9BD20-7408-45B8-8D5C-4B5462B4B832, OS:17.0, name:iPad Pro (11-inch) (4th generation) }
{ platform:iOS Simulator, id:8CD15AE9-61B4-4325-9EC8-0C5452326548, OS:17.0, name:iPad Pro (11-inch) (4th generation) }
{ platform:iOS Simulator, id:3AF13222-E25F-4AAD-9B25-313B546160A5, OS:17.0, name:iPad Pro (12.9-inch) (6th generation) }
{ platform:iOS Simulator, id:A29340A6-7253-4BC6-B802-82911F4A206C, OS:17.0, name:iPad Pro (12.9-inch) (6th generation) }
{ platform:iOS Simulator, id:26DCF2DC-28E2-4391-8692-2A095B834051, OS:17.0, name:iPad mini (6th generation) }
{ platform:iOS Simulator, id:B58F5198-FE10-444F-9426-8BABA4FD625B, OS:17.0, name:iPad mini (6th generation) }
{ platform:iOS Simulator, id:D52F1164-907E-495B-837C-D42545DCFE22, OS:17.0, name:iPhone 14 }
{ platform:iOS Simulator, id:4DAFDB51-3728-4A43-B99B-64A3D85E059A, OS:17.0, name:iPhone 14 Plus }
{ platform:iOS Simulator, id:AAA38124-9954-4566-9500-917734C9C0B9, OS:17.0, name:iPhone 14 Pro }
{ platform:iOS Simulator, id:3FD1A3D9-B78B-42F7-8066-75C1EF6102F9, OS:17.0, name:iPhone 14 Pro Max }
{ platform:iOS Simulator, id:051BC11B-6054-41D7-A0AB-01696DF133A8, OS:17.0, name:iPhone 15 }
{ platform:iOS Simulator, id:CCFE466C-6647-4D96-9A23-A2E518BFBBFC, OS:17.0, name:iPhone 15 }
{ platform:iOS Simulator, id:4744DE1E-8172-4D57-8625-7B2F289E0981, OS:17.0, name:iPhone 15 Plus }
{ platform:iOS Simulator, id:7C56DD58-96A0-4F64-8B83-0055C313209D, OS:17.0, name:iPhone 15 Plus }
{ platform:iOS Simulator, id:79B8084D-1D9E-4CA7-B113-5D42A648530D, OS:17.0, name:iPhone 15 Pro }
{ platform:iOS Simulator, id:7A142D05-9F1D-401B-B01E-B141E8CF21E0, OS:17.0, name:iPhone 15 Pro }
{ platform:iOS Simulator, id:8BA38FEC-DBFB-4975-ADC0-98951ACDDA9F, OS:17.0, name:iPhone 15 Pro Max }
{ platform:iOS Simulator, id:E458655E-3E57-4C1B-AFA8-BA866C7737A4, OS:17.0, name:iPhone 15 Pro Max }
{ platform:iOS Simulator, id:89E04F53-A08E-42DF-8B77-C41C9CA3F315, OS:17.0, name:iPhone SE (3rd generation) }
{ platform:iOS Simulator, id:C272E0A6-884D-412F-82C8-74797D994E00, OS:17.0, name:iPhone SE (3rd generation) }
You'd think it'd be smart enough to use the first that matches all the criteria, but it seems to ignore the name when it selects from multiple.
The fix is to delete any simulators with the name first, then create a simulator with the name, then run the test.
Needs to be done before here: https://github.com/flutter/engine/blob/main/testing/scenario_app/run_ios_tests.sh#L64-L70
We do this in packages for this exact problem: https://github.com/flutter/packages/blob/60a1ffc2f77fade3e375a98177729b2f3e90e06e/.ci/scripts/create_simulator.sh#L14-L27
Needs to be done before here: https://github.com/flutter/engine/blob/main/testing/scenario_app/run_ios_tests.sh#L64-L70
Seems this is a test by test fix. Will other tests from framework/engine potentially experience similar issues?
Needs to be done before here: https://github.com/flutter/engine/blob/main/testing/scenario_app/run_ios_tests.sh#L64-L70
Seems this is a test by test fix. Will other tests from framework/engine potentially experience similar issues?
Yes, they could. From my knowledge, though, most other tests are already setup to do this.
Some examples:
I'm not sure if there's like a catch-all solution we could do since different tests use different simulator types, names, etc - but open to ideas.
Perhaps we could delete all simulator devices in recipes before every build and rely on the test to create its needed simulator device.
Perhaps we could delete all simulator devices in recipes before every build and rely on the test to create its needed simulator device.
What would be the time cost to delete all simulator devices and re-create? Or delete only when >1 devices exist.
I think the tests should specify a simulator and manually create it if it doesn't exist. Running with "whatever the bot has installed" could lead to inconsistent test results.
What would be the time cost to delete all simulator devices and re-create? Or delete only when >1 devices exist.
It's hard to say for sure without testing it, but probably less than 5 seconds for the most part. The initial delete all devices may take longer, but after that there shouldn't be a ton of devices to delete (probably only one) since it would delete on each build.
Example: See step 12.15 (which attempts to delete and then creates simulator) - takes 3.9 seconds: https://ci.chromium.org/ui/p/flutter/builders/try/Mac_arm64%20custom_package_tests%20master/10823/overview
I think the tests should specify a simulator and manually create it if it doesn't exist. Running with "whatever the bot has installed" could lead to inconsistent test results.
It makes sense to me. What I mentioned earlier (as @vashworth said) is we clean up the devices and install all. From test side, you still use the defined simulator version and nothing should be affected. The difference is test side will not see multiple entries.
Example: See step 12.15 (which attempts to delete and then creates simulator) - takes 3.9 seconds: https://ci.chromium.org/ui/p/flutter/builders/try/Mac_arm64%20custom_package_tests%20master/10823/overview
3.9 is okay. Let's see if https://github.com/flutter/engine/pull/50835 covers the last corner case. If not, we may consider implementing from recipes side to cover all.
It makes sense to me. What I mentioned earlier (as @vashworth said) is we clean up the devices and install all. From test side, you still use the defined simulator version and nothing should be affected. The difference is test side will not see multiple entries.
We actually wouldn't "install all" (I don't think there is a way to do that), we'd only delete all. So for a test to use a simulator, it will need to create it. If a test tries to use a simulator without first creating it, it would fail.
https://github.com/flutter/engine/pull/50835 should definitely handle this particular case, but if we want a more consistent way of handling this across all tests (engine/framework), then we may want to do the recipes side.
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v
and a minimal reproduction of the issue.
Type of Request
infra task
Infrastructure Environment
LUCI, Github
What is happening?
https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20mac_unopt?limit=100
Recent tries have had a number of failures due to either
ios_debug_sim
orios_debug_unopt_sim
both failing with an errorCannot test target “ScenariosUITests” on “Any iOS Device”: Tests must be run on a concrete device
Steps to reproduce
No response
Expected results
No response