refreshAll failed to find C++ standard library if I specify '--sysroot'

[dantesun]

I am working on istio wasm-cpp extensions project. And I am using the following bazel configuration:

# Hedron's Compile Commands Extractor for Bazel
# https://github.com/hedronvision/bazel-compile-commands-extractor
http_archive(
    name = "hedron_compile_commands",

    # Replace the commit hash (0e990032f3c5a866e72615cf67e5ce22186dcb97) in both places (below) with the latest (https://github.com/hedronvision/bazel-compile-commands-extractor/commits/main), rather than using the stale one here.
    # Even better, set up Renovate and let it do the work for you (see "Suggestion: Updates" in the README).
    url = "https://github.com/hedronvision/bazel-compile-commands-extractor/archive/204aa593e002cbd177d30f11f54cff3559110bb9.tar.gz",
    strip_prefix = "bazel-compile-commands-extractor-204aa593e002cbd177d30f11f54cff3559110bb9",
    # When you first run this tool, it'll recommend a sha256 hash to put here with a message like: "DEBUG: Rule 'hedron_compile_commands' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = ..."
)
load("@hedron_compile_commands//:workspace_setup.bzl", "hedron_compile_commands_setup")
hedron_compile_commands_setup()
load("@hedron_compile_commands//:workspace_setup_transitive.bzl", "hedron_compile_commands_setup_transitive")
hedron_compile_commands_setup_transitive()
load("@hedron_compile_commands//:workspace_setup_transitive_transitive.bzl", "hedron_compile_commands_setup_transitive_transitive")
hedron_compile_commands_setup_transitive_transitive()
load("@hedron_compile_commands//:workspace_setup_transitive_transitive_transitive.bzl", "hedron_compile_commands_setup_transitive_transitive_transitive")
hedron_compile_commands_setup_transitive_transitive_transitive()

When I run bazel run @hedron_compile_commands//:refresh_all, I got the error:

external/hedron_compile_commands/refresh_all.py", line 818, in _emscripten_platform_patch                                                            assert sysroot, f'Emscripten sysroot not detected in CMD: {compile_action.arguments}'                                                        AssertionError: Emscripten sysroot not detected in CMD: ['external/proxy_wasm_cpp_sdk/toolchain/emcc.sh', '-MD', '-MF', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/extensions/sni_misdirect/_objs/_wasm_sni_misdirect.wasm/plugin.d', '-frandom-seed=bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/extensions/sni_misdirect/_objs/_wasm_sni_misdirect.wasm/plugin.o', '-iquote', '.', '-iquote', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin', '-iquote', 'external/com_google_absl', '-iquote', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/external/com_google_absl', '-iquote', 'external/proxy_wasm_cpp_sdk', '-iquote', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/external/proxy_wasm_cpp_sdk', '-iquote', 'external/bazel_tools', '-iquote', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/external/bazel_tools', '-std=c++17', '-no-canonical-prefixes', '-O3', '-ffunction-sections', '-fdata-sections', '-flto', '-c', 'extensions/sni_misdirect/plugin.cc', '-o', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/extensions/sni_misdirect/_objs/_wasm_sni_misdirect.wasm/plugin.o']

I found that I have to specify the 'sysroot' to Bazel C++ compiler options, so I downloaed the Clang SDK from https://github.com/llvm/llvm-project/releases/download/llvmorg-17.0.5/clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz, unpacked into /home/dsun/apps/clang. Then I specify the 'sysroot' in bazel.rc

build --config=clang
build:gcc --cxxopt=-std=c++17

build:clang --action_env=CC=clang --action_env=CXX=clang++
build:clang --action_env=BAZEL_COMPILER=clang
build:clang --linkopt=-fuse-ld=lld
build:clang --cxxopt=-std=c++17
build:clang --cxxopt="--sysroot=/home/dsun/apps/clang"

Then I ran bazel run @hedron_compile_commands//:refresh_all again and I got the following error:

external/hedron_compile_commands/print_args.cpp:3:10: fatal error: 'iostream' file not found
    3 | #include <iostream>
      |          ^~~~~~~~~~
1 error generated.                                                                                                                               1710429164.770253786: src/main/tools/linux-sandbox-pid1.cc:507: wait returned pid=2, status=0x100
1710429164.770277908: src/main/tools/linux-sandbox-pid1.cc:525: child exited normally with code 1
1710429164.770652471: src/main/tools/linux-sandbox.cc:233: child exited normally with code 1
Target @hedron_compile_commands//:refresh_all failed to build
INFO: Elapsed time: 0.642s, Critical Path: 0.12s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
[INFO 23:12:44.804 src/main/cpp/blaze.cc:2099] failure_detail: message: "linux-sandbox failed: error executing command \n  (cd /home/dsun/.cache/bazel/_bazel_dsun/c4e768d90cc77b1af937ef4d90fbb834/sandbox/linux-sandbox/161/execroot/istio_ecosystem_wasm_extensions && \\\n  exec env - \\\n    BAZEL_COMPILER=clang \\\n    CC=clang \\\n    CXX=clang++ \\\n    PATH=/home/dsun/.cache/bazelisk/downloads/sha256/4f0e14764a60375855e59bafb1de26f5189f6d34a65587c8d6dc499e57a8a8f0/bin:/home/dsun/node_modules/.bin:/home/dsun/.krew/bin:/home/dsun/bin:/home/dsun/go/bin:/home/dsun/node_modules/.bin:/home/dsun/.krew/bin:/home/dsun/bin:/home/dsun/go/bin:/home/dsun/.asdf/shims:/home/dsun/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib \\\n    PWD=/proc/self/cwd \\\n    TMPDIR=/tmp \\\n  /home/dsun/.cache/bazel/_bazel_dsun/install/616d39adb94ebfda4ae84d4ce81a9faf/linux-sandbox -t 15 -w /home/dsun/.cache/bazel/_bazel_dsun/c4e768d90cc77b1af937ef4d90fbb834/sandbox/linux-sandbox/161/execroot/istio_ecosystem_wasm_extensions -w /tmp -w /dev/shm -D -- /home/dsun/apps/clang+llvm-17.0.6-x86_64-linux-gnu-ubuntu-22.04/bin/clang-17 -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer \'-std=c++0x\' -MD -MF bazel-out/k8-fastbuild/bin/external/hedron_compile_commands/_objs/print_args/print_args.pic.d \'-frandom-seed=bazel-out/k8-fastbuild/bin/external/hedron_compile_commands/_objs/print_args/print_args.pic.o\' -fPIC -iquote external/hedron_compile_commands -iquote bazel-out/k8-fastbuild/bin/external/hedron_compile_commands -iquote external/bazel_tools -iquote bazel-out/k8-fastbuild/bin/external/bazel_tools \'-std=c++17\' \'--sysroot=/home/dsun/apps/clang\' -no-canonical-prefixes -Wno-builtin-macro-redefined \'-D__DATE__=\"redacted\"\' \'-D__TIMESTAMP__=\"redacted\"\' \'-D__TIME__=\"redacted\"\' -c external/hedron_compile_commands/print_args.cpp -o bazel-out/k8-fastbuild/bin/external/hedron_compile_commands/_objs/print_args/print_args.pic.o)"
spawn {
  code: NON_ZERO_EXIT
  spawn_exit_code: 1
}

I got stuck and have no idea how to solve this. Please help.

[cpsauer]

Hi Dante! Thanks for giving the tool a go. Sorry we got you snagged on this rough edge.

Here's my analysis:

1. We definitely don't want to make you specify the sysroot if you wouldn't otherwise. (Please confirm that you wouldn't otherwise be setting the sysroot except to satisfy this tool?)
2. Something's wrong with that sysroot that's causing the last error you copied. That's a (simple) c++ utility this tool uses, failing to compile because the build settings you're passing have somehow removed iostream from the header search paths.

@dantesun, could I ask you to try the following two things (without any sysroot workarounds you'd added to satisfy the tool):

1. Could you specifying the top-level target you care about building in a refresh_compile_commands rule (per instructions in the readme), just in case we need to inherit configuration from, e.g. a wasm_cc_binary to configure things and add the sysroot?
2. Separately, I just pushed up a change (backlinked above) that makes that not run that check unnecessarily on newer bazel versions. Could I ask you to grab the latest commit and see whether that avoids the issue? (Depends on whether the EM_BIN_PATH environment variable has been auto-set in these cc_libraries.

If those don't work, please tag kon72 and ask him if he know's what's up. He might be able to help better, since having last touched that code and worked on emscripten support.

Cheers, Chris

[dantesun]

@cpsauer Thanks for your quick response. I tried the two things you said but unfortunately my problem still exists. @kon72 Could you take a look at this issue, thanks.

I am using bazel 5.4.0.

BUILD

load("@hedron_compile_commands//:refresh_compile_commands.bzl", "refresh_compile_commands")

refresh_compile_commands(
    name = "refresh_compile_commands",
    targets = ["//extensions/...", "//common/..."]
)

WORKSPACE

workspace(name = "istio_ecosystem_wasm_extensions")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

# Hedron's Compile Commands Extractor for Bazel
# https://github.com/hedronvision/bazel-compile-commands-extractor
http_archive(
    name = "hedron_compile_commands",

    # Replace the commit hash (0e990032f3c5a866e72615cf67e5ce22186dcb97) in both places (below) with the latest (https://github.com/hedronvision/bazel-compile-commands-extractor/commits/main), rather than using the stale one here.
    # Even better, set up Renovate and let it do the work for you (see "Suggestion: Updates" in the README).
    url = "https://github.com/hedronvision/bazel-compile-commands-extractor/archive/5bcb0bd8a917b2b48fb5dc55818515f4be3b63ff.tar.gz",
    strip_prefix = "bazel-compile-commands-extractor-5bcb0bd8a917b2b48fb5dc55818515f4be3b63ff",
    # When you first run this tool, it'll recommend a sha256 hash to put here with a message like: "DEBUG: Rule 'hedron_compile_commands' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = ..."
)
load("@hedron_compile_commands//:workspace_setup.bzl", "hedron_compile_commands_setup")
hedron_compile_commands_setup()
load("@hedron_compile_commands//:workspace_setup_transitive.bzl", "hedron_compile_commands_setup_transitive")
hedron_compile_commands_setup_transitive()
load("@hedron_compile_commands//:workspace_setup_transitive_transitive.bzl", "hedron_compile_commands_setup_transitive_transitive")
hedron_compile_commands_setup_transitive_transitive()
load("@hedron_compile_commands//:workspace_setup_transitive_transitive_transitive.bzl", "hedron_compile_commands_setup_transitive_transitive_transitive")
hedron_compile_commands_setup_transitive_transitive_transitive()

load("//bazel:third_party.bzl", "wasm_extension_dependency")

wasm_extension_dependency()

load(
    "@io_bazel_rules_docker//repositories:repositories.bzl",
    container_repositories = "repositories",
)

container_repositories()

load("@io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")

container_deps()

PROXY_WASM_CPP_SDK_SHA = "fd0be8405db25de0264bdb78fae3a82668c03782"

PROXY_WASM_CPP_SDK_SHA256 = "c57de2425b5c61d7f630c5061e319b4557ae1f1c7526e5a51c33dc1299471b08"

http_archive(
    name = "proxy_wasm_cpp_sdk",
    sha256 = PROXY_WASM_CPP_SDK_SHA256,
    strip_prefix = "proxy-wasm-cpp-sdk-" + PROXY_WASM_CPP_SDK_SHA,
    url = "https://github.com/proxy-wasm/proxy-wasm-cpp-sdk/archive/" + PROXY_WASM_CPP_SDK_SHA + ".tar.gz",
)

load("@proxy_wasm_cpp_sdk//bazel/dep:deps.bzl", "wasm_dependencies")

wasm_dependencies()

load("@proxy_wasm_cpp_sdk//bazel/dep:deps_extra.bzl", "wasm_dependencies_extra")

wasm_dependencies_extra()

load("@istio_ecosystem_wasm_extensions//bazel:wasm.bzl", "wasm_libraries")

wasm_libraries()

# To import proxy wasm cpp host, which will be used in unit testing.
load("@proxy_wasm_cpp_host//bazel:repositories.bzl", "proxy_wasm_cpp_host_repositories")

proxy_wasm_cpp_host_repositories()

load("@proxy_wasm_cpp_host//bazel:dependencies.bzl", "proxy_wasm_cpp_host_dependencies")

proxy_wasm_cpp_host_dependencies()

I got errors when running bazel run :refresh_compile_commands

INFO: Build completed successfully, 5 total actions                                                                                              [INFO 19:47:06.180 src/main/cpp/blaze_util_posix.cc:308] Invoking binary /bin/bash in /home/dsun/.cache/bazel/_bazel_dsun/c4e768d90cc77b1af937ef4d90fbb834/execroot/istio_ecosystem_wasm_extensions/bazel-out/k8-fastbuild/bin/refresh_compile_commands.runfiles/istio_ecosystem_wasm_extensions  >>> Analyzing commands used in //extensions/...                                                                                                  [INFO 19:47:06.808 src/main/cpp/option_processor.cc:407] Looking for the following rc files: /etc/bazel.bazelrc,/home/dsun/work/gateway/plugins/wasm-cpp/.bazelrc,/home/dsun/.bazelrc                                                                                                             [INFO 19:47:06.808 src/main/cpp/rc_file.cc:56] Parsing the RcFile /home/dsun/work/gateway/plugins/wasm-cpp/.bazelrc                              [INFO 19:47:06.808 src/main/cpp/rc_file.cc:56] Parsing the RcFile /home/dsun/.bazelrc                                                            [INFO 19:47:06.808 src/main/cpp/blaze.cc:1628] Debug logging requested, sending all client log statements to stderr                              [INFO 19:47:06.808 src/main/cpp/blaze.cc:1511] Acquired the client lock, waited 0 milliseconds                                                   [INFO 19:47:06.811 src/main/cpp/blaze.cc:1704] Trying to connect to server (timeout: 30 secs)...                                                 [INFO 19:47:06.857 src/main/cpp/blaze.cc:1266] Connected (server pid=1573542).                                                                   [INFO 19:47:06.857 src/main/cpp/blaze.cc:1980] Releasing client lock, let the server manage concurrent requests.                                 WARNING: Option 'host_javabase' is deprecated                                                                                                    WARNING: Option 'javabase' is deprecated

Traceback (most recent call last):                                                                                                                 File "/home/dsun/.cache/bazel/_bazel_dsun/c4e768d90cc77b1af937ef4d90fbb834/execroot/istio_ecosystem_wasm_extensions/bazel-out/k8-fastbuild/bin/refresh_compile_commands.runfiles/istio_ecosystem_wasm_extensions/refresh_compile_commands.check_python_version.py", line 15, in <module>
    refresh_compile_commands.main()
  File "/home/dsun/.cache/bazel/_bazel_dsun/c4e768d90cc77b1af937ef4d90fbb834/execroot/istio_ecosystem_wasm_extensions/bazel-out/k8-fastbuild/bin/refresh_compile_commands.py", line 1425, in main
    compile_command_entries.extend(_get_commands(target, flags))
  File "/home/dsun/.cache/bazel/_bazel_dsun/c4e768d90cc77b1af937ef4d90fbb834/execroot/istio_ecosystem_wasm_extensions/bazel-out/k8-fastbuild/bin/refresh_compile_commands.py", line 1286, in _get_commands
    yield from _convert_compile_commands(parsed_aquery_output)
  File "/home/dsun/.cache/bazel/_bazel_dsun/c4e768d90cc77b1af937ef4d90fbb834/execroot/istio_ecosystem_wasm_extensions/bazel-out/k8-fastbuild/bin/refresh_compile_commands.py", line 1166, in _convert_compile_commands
    for source_files, header_files, compile_command_args in outputs:
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/dsun/.cache/bazel/_bazel_dsun/c4e768d90cc77b1af937ef4d90fbb834/execroot/istio_ecosystem_wasm_extensions/bazel-out/k8-fastbuild/bin/refresh_compile_commands.py", line 1126, in _get_cpp_command_for_files
    compile_action.arguments = _emscripten_platform_patch(compile_action)
  File "/home/dsun/.cache/bazel/_bazel_dsun/c4e768d90cc77b1af937ef4d90fbb834/execroot/istio_ecosystem_wasm_extensions/bazel-out/k8-fastbuild/bin/refresh_compile_commands.py", line 828, in _emscripten_platform_patch
    assert sysroot, f'Emscripten sysroot not detected in CMD: {compile_action.arguments}'
AssertionError: Emscripten sysroot not detected in CMD: ['external/proxy_wasm_cpp_sdk/toolchain/emcc.sh', '-MD', '-MF', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/extensions/sni_misdirect/_objs/_wasm_sni_misdirect.wasm/plugin.d', '-frandom-seed=bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/extensions/sni_misdirect/_objs/_wasm_sni_misdirect.wasm/plugin.o', '-iquote', '.', '-iquote', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin', '-iquote', 'external/com_google_absl', '-iquote', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/external/com_google_absl', '-iquote', 'external/proxy_wasm_cpp_sdk', '-iquote', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/external/proxy_wasm_cpp_sdk', '-iquote', 'external/bazel_tools', '-iquote', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/external/bazel_tools', '-std=c++17', '-no-canonical-prefixes', '-O3', '-ffunction-sections', '-fdata-sections', '-flto', '-c', 'extensions/sni_misdirect/plugin.cc', '-o', 'bazel-out/wasm32-fastbuild-ST-4bf3c0394d2a/bin/extensions/sni_misdirect/_objs/_wasm_sni_misdirect.wasm/plugin.o']

WASM_SDK

    PROXY_WASM_CPP_HOST_SHA = "f38347360feaaf5b2a733f219c4d8c9660d626f0"
    PROXY_WASM_CPP_HOST_SHA256 = "bf10de946eb5785813895c2bf16504afc0cd590b9655d9ee52fb1074d0825ea3"

    http_archive(
        name = "proxy_wasm_cpp_host",
        sha256 = PROXY_WASM_CPP_HOST_SHA256,
        strip_prefix = "proxy-wasm-cpp-host-" + PROXY_WASM_CPP_HOST_SHA,
        url = "https://github.com/proxy-wasm/proxy-wasm-cpp-host/archive/" + PROXY_WASM_CPP_HOST_SHA +".tar.gz",
    )

[dantesun]

@cpsauer I can't find EM_BIN_PATH declared in emscripten_toolchain 2.0.7 which my proxy_wasm_cpp_sdk dependes on.

[kon72]

Hello @dantesun,

Thanks for providing the information. That helps me a lot to figure out the issue.

When I implemented the emscripten support, I expected your projects to use emsdk's official Bazel toolchain, so this tool is relying on sysroot flag that is supposed to be set inside that toolchain in order to make up the environment variable that is necessary to invoke emcc.sh wrapper script. However, it seems that your version of proxy_wasm_cpp_sdk implements its own Bazel toolchain, and it doesn't set sysroot flag.

I think we have two options to resolve this issue:

1. Since recent versions of proxy_wasm_cpp_sdk moved out of its own toolchain to emsdk's one, you can try to update your proxy_wasm_cpp_sdk and see if it resolves the issue.

2. If upgrade is not an option, we could then make a simple patch to this tool so that if sysroot flag is not found, it will not set EM_BIN_PATH environment variable. This is because proxy_wasm_cpp_sdk's emcc.sh wrapper doesn't seem to require any environment variable to be set.

[cpsauer]

(Thanks so much, Kon!)