Mac builders are failing (timing out) on `find_sdk`

matanlurey commented 1 month ago

Using prebuilt Dart SDK binary. If you are editing Dart sources and wish to compile the Dart SDK, set `--no-prebuilt-dart-sdk`.
Generating GN files in: out/ci/host_debug_framework
ERROR at //build/config/mac/mac_sdk.gni:41:7: Script returned non-zero exit code.
      exec_script("//build/mac/find_sdk.py", find_sdk_args, "list lines")
      ^----------
Current dir: /Volumes/Work/s/w/ir/cache/builder/src/out/ci/host_debug_framework/
Command: vpython3 /Volumes/Work/s/w/ir/cache/builder/src/build/mac/find_sdk.py --print_sdk_path 10.14
Returned 1.
stderr:

...

subprocess.TimeoutExpired: Command '['xcodebuild', '-showsdks', '-json']' timed out after 300 seconds

This is not a 1-off:

gaaclarke commented 1 month ago

More recent example: https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20Engine%20Drone/1164527/overview

chinmaygarde commented 1 month ago

And another one today https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8736442251949310241/+/u/gn_--target-dir_ci_android_profile_arm64_--runtime-mode_profile_--android_--android-cpu_arm64_--no-rbe_--no-goma/stdout

zanderso commented 3 weeks ago

I have not seen any instances of this since https://github.com/flutter/buildroot/pull/912, so I'll close this.

zanderso commented 3 weeks ago

Still happening =(

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8733890752054376529/+/u/gn_--ios_--runtime-mode_debug_--simulator_--no-lto_--target-dir_ci_ios_debug_sim_clang_tidy_--rbe_--no-goma_--xcode-symlinks_--rbe-server-address_unix:___Volumes_Work_s_w_ir_x_w_rc_rbe93mh56l6_reproxy.sock/stdout

zanderso commented 3 weeks ago

fyi @jmagman

jmagman commented 2 weeks ago

The steps before it show Xcode is installed and selected. The TimeoutExpired is coming from the subprocess check, so I doubt it's something weird like it's writing to stderr and then hanging (https://github.com/flutter/buildroot/blob/e256f5c0153dbde76c210a2715015c2ea5fe4cd2/build/mac/find_sdk.py#L83-L87 looks suspicious from the initial import of the script but I don't know what's going on there.).

Maybe we have to go back to the sdk regex parsing, and hope issues like https://github.com/flutter/flutter/issues/94513 don't happen anymore now that we are using cipd packages.

Let's make sure we don't regress the stable order in case there are multiples being discovered in a nondeterministic order, however it's done. https://github.com/flutter/buildroot/pull/652

flutter / flutter

Mac builders are failing (timing out) on `find_sdk` #155106