--incompatible_enable_cc_toolchain_resolution still tries to access local_config_cc

[AustinSchuh]

Splitting off of https://github.com/bazelbuild/bazel/issues/7260,

I flipped --incompatible_enable_cc_toolchain_resolution today with 4.0.0-rc7 (essentially), provided my own toolchains, removed the --crosstool_top flag, and promptly got the following:

$ BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 bazel build -c opt --platforms=@aos//tools/platforms:linux_aarch64 //...
ERROR: /home/austin/.cache/bazel/_bazel_austin/71ec74b5bb538661db62db908afb04ce/external/local_config_cc/BUILD:29:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'aarch64'
ERROR: Analysis of target '//build:docker_rootfs' failed; build aborted: Analysis of target '@local_config_cc//:toolchain' failed
INFO: Elapsed time: 0.836s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

We don't have an android or IOS build, so I would have thought I could flip it without waiting.

INFO: ToolchainResolution: Target platform @aos//tools/platforms:linux_x86: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @llvm_toolchain//:cc-clang-linux-k8
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: execution @local_config_platform//:host: Selected toolchain @llvm_toolchain//:cc-clang-linux-k8
INFO: ToolchainResolution:     Type @bazel_tools//tools/python:toolchain_type: execution platform @local_config_platform//:host: Rejected toolchain @io_bazel_rules_docker//toolchains:default_container_py_runtime_pair; mismatching values: run_in_container
INFO: ToolchainResolution: Target platform @aos//tools/platforms:linux_aarch64: Selected execution platform @io_bazel_rules_docker//platforms:local_container_platform, type @bazel_tools//tools/python:toolchain_type -> toolchain @io_bazel_rules_docker//toolchains:default_container_py_runtime_pair, type @bazel_tools//tools/cpp:toolchain_type -> toolcha    in @llvm_toolchain//:cc-clang-linux-aarch64
INFO: ToolchainResolution: Target platform @aos//tools/platforms:linux_aarch64: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/python:toolchain_type -> toolchain @bazel_tools//tools/python:_autodetecting_py_runtime_pair, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @llvm_toolchain//:cc-clang-linux-aarc    h64
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform @aos//tools/platforms:linux_aarch64: execution @io_bazel_rules_docker//platforms:local_container_platform: Selected toolchain @llvm_toolchain//:cc-clang-linux-aarch64
ERROR: /home/austin/.cache/bazel/_bazel_austin/71ec74b5bb538661db62db908afb04ce/external/local_config_cc/BUILD:29:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'aarch64'

suggests that toolchain resolution succeeded, and the correct toolchains were selected. But, for some reason, the default local_config_cc is still being looked at, maybe because it is the default for --crosstool_top or something like that. This ends up being a problem because I'm cross-compiling for aarch64, and local_config_cc isn't going to be able to find a compiler for that.

[AustinSchuhBRT]

FYI, @katre @gregestren, split from the other ticket.

[eagleonhill]

We encounters the same problem: whenever we want to cross-compile anything, with a platform mapping that doesn't map to host cpu(k8), I get the same error.

Looks like the reference is from CppHelper.getToolchainUsingDefaultCcToolchainAttribute calls, which always read crosstool_top.

[katre]

As a workaround, can you create a cc_toolchain_suite and set --crosstool_top to keep the legacy CC machinery happy?

@oquenchil, can we make this check ignorable when toolchain resolution is enabled, or is it going to be necessary until we can flip the flag and remove the legacy code?

[AustinSchuhBRT]

That's what I ended up doing, but it didn't feel right. Hence the bug.

[katre]

Definitely a bug, I'm just glad to know there's a workaround if anyone hits this in the future.

[carpenterjc]

I have found you can delay setting the crosstool_top to the platform_mappings file which avoids having it globally.

This is with Bazel 5.1

For example:

platforms:
  //:my_platform
    --cpu=x64_windows
    --crosstool_top=//:my_toolchains 

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[limdor]

@bazelbuild/triage please keep, this is still something that does not make sense

[comius]

This was possibly resolve with https://github.com/bazelbuild/bazel/commit/0013115fb495380d262a2cc30bcffef248ecb105 After the commit crosstool_top is ignored with C++ toolchain resolution enabled (before it was still a dependency of C++ rules).

@AustinSchuh could you verify and close?

[AustinSchuh]

@comius , I can't tell for certain, but I think I'm getting worse errors now on bazel 7. They may be different, I can't tell easily. With both --incompatible_enable_cc_toolchain_resolution and BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1, I now unconditionally get local_config_cc as the result of toolchain resolution.

INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //tools/platforms:linux_x86: Rejected toolchain @bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform //tools/platforms:linux_x86: execution //tools/platforms:linux_x86: Selected toolchain @bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-k8
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //tools/platforms:linux_x86: Rejected toolchain @bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //tools/platforms:linux_x86: execution platform //tools/platforms:linux_x86: Skipping toolchain @bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-k8; execution platform already has selected toolchain
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //tools/platforms:linux_x86: execution platform //tools/platforms:linux_x86: Skipping toolchain @llvm_toolchain//:cc-clang-x86_64-linux; execution platform already has selected toolchain
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //tools/platforms:linux_x86: Rejected toolchain @llvm_toolchain//:cc-clang-aarch64-linux; mismatching values: aarch64
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //tools/platforms:linux_x86: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: armv7, android
INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform //tools/platforms:linux_x86: execution platform //tools/platforms:linux_x86: Skipping toolchain @local_config_cc//:cc-compiler-k8; execution platform already has selected toolchain
INFO: ToolchainResolution: Target platform //tools/platforms:linux_x86: Selected execution platform //tools/platforms:linux_x86, type @bazel_tools//tools/python:toolchain_type -> toolchain @bazel_tools//tools/python:_autodetecting_py_runtime_pair, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-k8

You can see here that Bazel picks @bazel_tools~cc_configure_extension~local_config_cc//:cc-compiler-k8 which I don't want it to do. I don't want that toolchain to exist at all, and want it to pick @llvm_toolchain//:cc-clang-x86_64-linux instead.

(I can probably persuade it to work with `--extra_toolchains or one of the other flags, but that seems like it leaves me at risk of accidentally getting the wrong toolchain somewhere else.)

Any ideas?

[fmeum]

@AustinSchuh Do you have a reproducer you could share? On an empty project with just a single cc_library target, I get:

$ USE_BAZEL_VERSION=7.0.0rc4 bazel build //:lib --repo_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 --incompatible_enable_cc_toolchain_resolution --toolchain_resolution_debug=.*
INFO: Invocation ID: 5f1cf9a1-29a6-496e-bcf0-0e5a764647b9
WARNING: Build option --toolchain_resolution_debug has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: ToolchainResolution: Target platform @@local_config_platform//:host: Selected execution platform @@local_config_platform//:host, 
INFO: ToolchainResolution: Performing resolution of @@bazel_tools//tools/cpp:toolchain_type for target platform @@local_config_platform//:host
      ToolchainResolution: No @@bazel_tools//tools/cpp:toolchain_type toolchain found for target platform @@local_config_platform//:host.
INFO: ToolchainResolution: Target platform @@local_config_platform//:host: Selected execution platform @@local_config_platform//:host, 
ERROR: /home/fhenneke/.cache/bazel/_bazel_fhenneke/e9921a0e334967d7d44291dbeea196de/external/bazel_tools/tools/cpp/BUILD:58:19: in cc_toolchain_alias rule @@bazel_tools//tools/cpp:current_cc_toolchain: 
Traceback (most recent call last):
    File "/virtual_builtins_bzl/common/cc/cc_toolchain_alias.bzl", line 26, column 48, in _impl
    File "/virtual_builtins_bzl/common/cc/cc_helper.bzl", line 219, column 17, in _find_cpp_toolchain
Error in fail: Unable to find a CC toolchain using toolchain resolution. Target: @@bazel_tools//tools/cpp:current_cc_toolchain, Platform: @@local_config_platform//:host, Exec platform: @@local_config_platform//:host

which looks right to me: no toolchain from @bazel_tools is registered.

[carpenterjc]

@AustinSchuh @comius Speculation here, but could this be because some rules have switched to building some tools rather than using pre-build binaries? For example they had to switch a python launcher to a pre-built one for windows with this defect? https://github.com/bazelbuild/bazel/issues/19977

[AustinSchuh]

I feel a bit embarrassed, but I'm having trouble reproducing the failure too. I'm getting this feeling like I somehow didn't add BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN like I thought I did... Thanks for digging in.