Bazel Windows Cross Compile Broken

jxy-s commented 2 months ago

Brief description of your issue

Compiling non-x64 for Windows through Bazel is broken. This is necessary to do testing for other architectures on Windows. This probably changed when we removed the ability to compile with the EWDK. The EWDK toolchain previously supported cross compilation. The cross compilation platform defintions here are broken somehow with the "normal" Bazel tool-chain resolution: https://github.com/archonitelabs/radiant-cpp/blob/e32b6839c9a9be11a34fdb4c9643319a3f27a87b/BUILD.bazel#L45-L79

Steps to reproduce (optional)

On an x64 or ARM64 Windows host:

➜ rad build --win_arm64
ERROR: C:/users/jxy/_bazel_jxy/ytddry7e/external/bazel_tools/tools/cpp/BUILD:58:19: in cc_toolchain_alias rule @@bazel_tools//tools/cpp:current_cc_toolchain:
Traceback (most recent call last):
        File "/virtual_builtins_bzl/common/cc/cc_toolchain_alias.bzl", line 26, column 48, in _impl
        File "/virtual_builtins_bzl/common/cc/cc_helper.bzl", line 219, column 17, in _find_cpp_toolchain
Error in fail: Unable to find a CC toolchain using toolchain resolution. Target: @@bazel_tools//tools/cpp:current_cc_toolchain, Platform: @@//:windows_arm64, Exec platform: @@local_config_platform//:host
ERROR: C:/users/jxy/_bazel_jxy/ytddry7e/external/bazel_tools/tools/cpp/BUILD:58:19: Analysis of target '@@bazel_tools//tools/cpp:current_cc_toolchain' failed
ERROR: C:/users/jxy/source/repos/archonitelabs/radiant-cpp/test/BUILD.bazel:20:8: errors encountered resolving select() keys for //test:all_test14
ERROR: Analysis of target '//test:all_test14' failed; build aborted
INFO: Elapsed time: 0.479s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
2024-04-26 11:02:03 ERROR    build failed!

Interestingly an ARM64 host resolves the default host tool-chain to x64 and outputs x64 binaries instead of ARM64. This seems like a bug in the MSVC Bazel tool-chain?

On my x64 host:

➜ bazel build --toolchain_resolution_debug=.* --platforms=:windows_arm64 //...
INFO: ToolchainResolution: Target platform //:windows_arm64: Selected execution platform @@local_config_platform//:host,
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/cpp:toolchain_type for target platform //:windows_arm64
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-x64_windows; mismatching values: x86_64
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
      ToolchainResolution:   Rejected toolchain @@bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-x64_windows; mismatching values: x86_64
      ToolchainResolution:   Rejected toolchain @@local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
      ToolchainResolution:   Rejected toolchain @@local_config_cc//:cc-compiler-x64_windows; mismatching values: x86_64
      ToolchainResolution: No @@bazel_tools//tools/cpp:toolchain_type toolchain found for target platform //:windows_arm64.
INFO: ToolchainResolution: Target platform //:windows_arm64: Selected execution platform @@local_config_platform//:host,
ERROR: C:/users/jxy/_bazel_jxy/ytddry7e/external/bazel_tools/tools/cpp/BUILD:58:19: in cc_toolchain_alias rule @@bazel_tools//tools/cpp:current_cc_toolchain:
Traceback (most recent call last):
        File "/virtual_builtins_bzl/common/cc/cc_toolchain_alias.bzl", line 26, column 48, in _impl
        File "/virtual_builtins_bzl/common/cc/cc_helper.bzl", line 219, column 17, in _find_cpp_toolchain
Error in fail: Unable to find a CC toolchain using toolchain resolution. Target: @@bazel_tools//tools/cpp:current_cc_toolchain, Platform: @@//:windows_arm64, Exec platform: @@local_config_platform//:host
ERROR: C:/users/jxy/_bazel_jxy/ytddry7e/external/bazel_tools/tools/cpp/BUILD:58:19: Analysis of target '@@bazel_tools//tools/cpp:current_cc_toolchain' failed
ERROR: C:/users/jxy/source/repos/archonitelabs/radiant-cpp/test/BUILD.bazel:28:8: errors encountered resolving select() keys for //test:all_test17
ERROR: Analysis of target '//test:all_test17' failed; build aborted
INFO: Elapsed time: 0.703s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully

➜ Get-ChildItem -Include cl.exe -Recurse "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools" | ForEach-Object{echo $_.FullName}
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\arm\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx86\arm\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx86\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx86\x86\cl.exe

Seems like a bug in Bazel tool-chain resolution for a standard MSVC (Visual Studio) install. A workaround might be manually activating the build environment before triggering Bazel tool-chain resolution:

➜ Get-ChildItem -Include vcvar*.bat -Recurse "C:\Program Files\Microsoft Visual Studio\2022\Community\VC" | ForEach-Object{echo $_.FullName}
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars32.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsamd64_arm.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsamd64_arm64.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsamd64_x86.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsx86_amd64.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsx86_arm.bat
C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsx86_arm64.bat

Expected behavior (optional)

Specifying a platform to build should cross-compile for that architecture for Windows. Not specifying an architecture should build using the native host tool-chain.

Actual behavior (optional)

Compilation fails if you try to cross compile. On an ARM64 host the located tool-chain is x64 and chooses the x64 target, so you end up with a x64 binary instead of the intended native ARM64 one.

Environment (optional)

Windows x64 and ARM64
Bazel MSVC

jxy-s commented 2 months ago

This is at least a regression in Bazel. And at worst the platforms constraints never worked correctly - I have submitted a bug to Bazel: https://github.com/bazelbuild/bazel/issues/22164

In the meantime, I have implemented bzlmod support in the EWDK toolchain repo. That will allow us to re-integrate with EWDK toolchain resolution as a workaround: https://github.com/0xf005ba11/bazel_ewdk_cc/pull/1

I will be putting together a PR on this repo to provide a workaround. Hopefully Bazel can address the problem or advise on another solution.

jxy-s commented 2 months ago

I've tracked down another problem in bazelisk that explains the problem with trying to compile natively on ARM64: https://github.com/bazelbuild/bazelisk/issues/572 https://github.com/bazelbuild/bazel/issues/22164#issuecomment-2081383179

jxy-s commented 2 months ago

I've implemented the workaround for cross compiling here: https://github.com/archonitelabs/radiant-cpp/pull/9

Keeping this open for tracking the issues brought to Bazel and for visibility.

archonitelabs / radiant-cpp