bazel links program using clang rather than clang++, incompatible with -fsanitize=undefined (UBSan)

[trittweiler]

Description of the problem / feature request:

Linking a C++ program with clang -fsanitize=undefined results in undefined symbols. It is necessary to use clang++ -fsanitize=undefined instead.

And bazel as of 3.7.2 tries to link C++ object files using clang rather than clang++.

What operating system are you running Bazel on?

Ubuntu Focal 20.04.1 LTS.

The test case was done using clang 10.0.0-4ubuntu1, that's simply from the clang-10 Ubuntu package.

What's the output of bazel info release?

release 3.7.2

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Here's a test case:

// ubsan_test.cc
#include <iostream>

int main(int argc, char **argv) {
  std::cout << "program " << argv[0] << " started" << std::endl;
  int k = 0x7fffffff;
  k += argc;
  return 0;
}

# BUILD
cc_binary(
    name = "ubsan_test",
    srcs = [ "ubsan_test.cc" ]
)

# WORKSPACE

And we now try to build, link and run :ubsan_test as follows:

> bazel run :ubsan_test -c dbg --repo_env=CC=clang --repo_env=CXX=clang++ --copt -fsanitize=undefined --copt -O1 --copt -fno-omit-frame-pointer --linkopt -fsanitize=undefined --subcommands
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:ubsan_test (15 packages loaded, 53 targets configured).
INFO: Found 1 target...
SUBCOMMAND: # //:ubsan_test [action 'Compiling ubsan_test.cc', configuration: 9ba3d5284ce70f920a40e5da377608177842fbf5b6c778ffecc875296ea059e1, execution platform: @local_config_platform//:host]
(cd /home/trittweiler/.cache/bazel/_bazel_trittweiler/ce7c0737658cab18554c03aaea49c3e0/execroot/__main__ && \
  exec env - \
    PATH=/home/trittweiler/.local/bin:/home/trittweiler/.local/software/yED/jre/bin:/home/trittweiler/.local/software/watchman/bin:/home/trittweiler/.local/software/rtags/bin:/home/trittweiler/.local/software/gopls/bin:/home/trittweiler/.local/software/emacs-git/bin:/home/trittweiler/.local/software/buck/bin:/opt/intellij-idea-community/bin:/opt/containerd/bin:/opt/android-studio/bin:/home/trittweiler/software/mtr/sbin/:/home/trittweiler/software/youtube-dl/bin/:/home/trittweiler/software/stumpwm/bin/:/home/trittweiler/software/sbcl/bin/:/home/trittweiler/software/cutter/bin/:/home/trittweiler/software/ccl-1.11.5/bin/:/home/trittweiler/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
  /usr/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer -g '-std=c++0x' -MD -MF bazel-out/k8-dbg/bin/_objs/ubsan_test/ubsan_test.pic.d '-frandom-seed=bazel-out/k8-dbg/bin/_objs/ubsan_test/ubsan_test.pic.o' -fPIC -iquote . -iquote bazel-out/k8-dbg/bin -iquote external/bazel_tools -iquote bazel-out/k8-dbg/bin/external/bazel_tools '-fsanitize=undefined' -O1 -fno-omit-frame-pointer -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c ubsan_test.cc -o bazel-out/k8-dbg/bin/_objs/ubsan_test/ubsan_test.pic.o)
SUBCOMMAND: # //:ubsan_test [action 'Linking ubsan_test', configuration: 9ba3d5284ce70f920a40e5da377608177842fbf5b6c778ffecc875296ea059e1, execution platform: @local_config_platform//:host]
(cd /home/trittweiler/.cache/bazel/_bazel_trittweiler/ce7c0737658cab18554c03aaea49c3e0/execroot/__main__ && \
  exec env - \
    PATH=/home/trittweiler/.local/bin:/home/trittweiler/.local/software/yED/jre/bin:/home/trittweiler/.local/software/watchman/bin:/home/trittweiler/.local/software/rtags/bin:/home/trittweiler/.local/software/gopls/bin:/home/trittweiler/.local/software/emacs-git/bin:/home/trittweiler/.local/software/buck/bin:/opt/intellij-idea-community/bin:/opt/containerd/bin:/opt/android-studio/bin:/home/trittweiler/software/mtr/sbin/:/home/trittweiler/software/youtube-dl/bin/:/home/trittweiler/software/stumpwm/bin/:/home/trittweiler/software/sbcl/bin/:/home/trittweiler/software/cutter/bin/:/home/trittweiler/software/ccl-1.11.5/bin/:/home/trittweiler/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
  /usr/bin/clang @bazel-out/k8-dbg/bin/ubsan_test-2.params)
ERROR: /home/trittweiler/tmp/bazel-bug/BUILD:1:10: Linking of rule '//:ubsan_test' failed (Exit 1): clang failed: error executing command /usr/bin/clang @bazel-out/k8-dbg/bin/ubsan_test-2.params

Use --sandbox_debug to see verbose messages from the sandbox clang failed: error executing command /usr/bin/clang @bazel-out/k8-dbg/bin/ubsan_test-2.params

Use --sandbox_debug to see verbose messages from the sandbox
ubsan_test.cc:4: error: undefined reference to '__ubsan_vptr_type_cache'
ubsan_test.cc:4: error: undefined reference to '__ubsan_handle_dynamic_type_cache_miss'
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/locale_facets.h:874: error: undefined reference to '__ubsan_vptr_type_cache'
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/locale_facets.h:874: error: undefined reference to '__ubsan_handle_dynamic_type_cache_miss'
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/locale_facets.h:875: error: undefined reference to '__ubsan_handle_dynamic_type_cache_miss'
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/locale_facets.h:876: error: undefined reference to '__ubsan_handle_dynamic_type_cache_miss'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //:ubsan_test failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6.665s, Critical Path: 0.43s
INFO: 8 processes: 7 internal, 1 linux-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Now I am going to manually repeat the link step (the /usr/bin/clang @bazel-out/k8-dbg/bin/ubsan_test-2.params) but replaceclang with clang++:

> (cd /home/trittweiler/.cache/bazel/_bazel_trittweiler/ce7c0737658cab18554c03aaea49c3e0/execroot/__main__ && \
>   exec env - \
>     PATH=/home/trittweiler/.local/bin:/home/trittweiler/.local/software/yED/jre/bin:/home/trittweiler/.local/software/watchman/bin:/home/trittweiler/.local/software/rtags/bin:/home/trittweiler/.local/software/gopls/bin:/home/trittweiler/.local/software/emacs-git/bin:/home/trittweiler/.local/software/buck/bin:/opt/intellij-idea-community/bin:/opt/containerd/bin:/opt/android-studio/bin:/home/trittweiler/software/mtr/sbin/:/home/trittweiler/software/youtube-dl/bin/:/home/trittweiler/software/stumpwm/bin/:/home/trittweiler/software/sbcl/bin/:/home/trittweiler/software/cutter/bin/:/home/trittweiler/software/ccl-1.11.5/bin/:/home/trittweiler/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
>     PWD=/proc/self/cwd \
>   /usr/bin/clang++ @bazel-out/k8-dbg/bin/ubsan_test-2.params)

And that succeeded just fine. And when I manually run the produced binary, it works just fine:

 UBSAN_OPTIONS=print_stacktrace=1 ./bazel-out/k8-dbg/bin/ubsan_test
program ./bazel-out/k8-dbg/bin/ubsan_test started
ubsan_test.cc:6:5: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
    #0 0x424f3c in main /proc/self/cwd/ubsan_test.cc:6:5
    #1 0x7f20290380b2 in __libc_start_main /build/glibc-ZN95T4/glibc-2.31/csu/../csu/libc-start.c:308:16
    #2 0x40352d in _start (/home/trittweiler/.cache/bazel/_bazel_trittweiler/ce7c0737658cab18554c03aaea49c3e0/execroot/__main__/bazel-out/k8-dbg/bin/ubsan_test+0x40352d)

[trittweiler]

Maybe I am just mistaken, and bazel does not actually look at the CXX environment variable?

It seems to be compiling the ubsan_test.cc also with clang, i.e. the contents of the CC variable.

So am I simply using it wrong?

[trittweiler]

Assuming bazel does indeed not look at CXX at this time, this issue is essentially evidence for someone tripping over what is covered by Issue #5186.

[chancila]

bazel currently compiles through the base clang/gcc driver and doesn't support the c++ drivers of these commands, bazel c++ toolchains generally have to be explicit about any implicit c++ stuff that would otherwise happen when you use the c++ drivers for clang/gcc.

if you run -v on your invocation with clang++ you'll probably see something like --whole-archive /usr/lib/clang/11.0.0/lib/linux/libclang_rt.ubsan_standalone_cxx-x86_64.a --no-whole-archive --dynamic-list=/usr/lib/clang/11.0.0/lib/linux/libclang_rt.ubsan_standalone_cxx-x86_64.a.syms

you effectively need to do this manually in your toolchain (or via --linkopts, or via the linkopts rule attributes), afaik this is a known limitation of the toolchain design, the link invocation is also always assumed to be c++ which isn't great but is typically benign because c code won't emit any references to c++ symbols.

[fmeum]

See https://github.com/bazelbuild/bazel/issues/11122#issuecomment-896613570 for two easy ways to fix this problem. There is no need to hardcode library paths. Appending either

--linkopt -fsanitize-link-c++-runtime

or

--linkopt --driver-mode=g++ 

to the commandline above makes clang link the UBSan C++ runtime libraries without the need to use clang++. In fact, the latter flag makes clang behave as if it were clang++.

[matts1]

This workaround solves the problem of compiling C++ code with a C compiler by trying to act like a C++ compiler. I was successfully able to compile C++ code with this workaround, but doing so broke the compilation of some of our dependencies that use C (see below).

IMO, the fact that we need workarounds for this is evidence that we should allow specifying both the C and C++ compiler.

We get errors like:

bazel build @zstd//:zstd
INFO: Analyzed target @zstd//:zstd (2 packages loaded, 16111 targets configured).
INFO: Found 1 target...
ERROR: /usr/local/google/home/msta/.cache/bazel/_bazel_msta/ed8a9562ed8b7b4d013265ed31cbccd4/external/zstd/BUILD.bazel:48:10: Linking external/zstd/zstd failed: (Exit 1): x86_64-cros-linux-gnu-clang failed: error executing command (from target @zstd//:zstd) /usr/local/google/home/msta/.cache/bazel/_bazel_msta/ed8a9562ed8b7b4d013265ed31cbccd4/external/_main~toolchains~toolchain_sdk/bin/x86_64-cros-linux-gnu-clang ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
ld.lld: error: undefined symbol: __cxa_begin_catch
>>> referenced by benchzstd.c
>>>               bazel-out/k8-dbg/bin/external/zstd/_objs/zstd/benchzstd.pic.o:(__clang_call_terminate)

ld.lld: error: undefined symbol: std::terminate()
>>> referenced by benchzstd.c
>>>               bazel-out/k8-dbg/bin/external/zstd/_objs/zstd/benchzstd.pic.o:(__clang_call_terminate)

ld.lld: error: undefined symbol: __gxx_personality_v0
>>> referenced by benchzstd.c
>>>               bazel-out/k8-dbg/bin/external/zstd/_objs/zstd/benchzstd.pic.o:(DW.ref.__gxx_personality_v0)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target @zstd//:zstd failed to build

And hundreds of warnings like the following:

INFO: From Compiling lib/compress/zstd_lazy.c:
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

[fmeum]

@comius @pzembrod I am thinking of extending the auto-configured toolchain in a way that ends up using g++/clang++ for C++ compilation actions, ideally in a backwards compatible way. This would bring Bazel in line with other build tools and remove subtle behavioral differences (https://github.com/bazel-contrib/toolchains_llvm/issues/372 just came up as another example). What do you think of that?