Support cc_toolchain specifying runtime dynamic libraries

[illicitonion]

Description of the feature request:

When using a cc_toolchain which contains self-contained tools (e.g. clang + lld + libstdc++), and dynamically linking, produced binaries have a runtime dependency on some libstdc++ being present on the machine.

Ideally, we would like our remote execution environment not to have a libstdc++ present, and would like to specify our hermetically supplied libstdc++ from our toolchain as a runtime dependency when a binary is run as an action (e.g. when used in a genrule).

Currently, we're working around this by installing libstdc++ in our remote execution environment, but we'd really like to get rid of this if possible. I couldn't find any way of configuring a cc_toolchain to set this up.

What underlying problem are you trying to solve with this feature?

Being able to run Bazel actions in purely hermetic environments, so that each team using our remote execution infrastructure is forced to bring their own libstdc++ or libc++ long in their toolchain, rather than falling back to a shared version.

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 5.3.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

Brief discussion on Slack: https://bazelbuild.slack.com/archives/CA31HN1T3/p1663676883083889

Any other information, logs, or outputs that you want to share?

No response

[jfirebaugh]

Another situation where this is necessary is if you want to create a hermetic llvm toolchain that supports both macOS and sanitizers. On macOS sanitizer runtimes must be dynamically linked. Thus the macOS LLVM distribution provides prebuilt sanitizer dylibs, e.g. libclang_rt.asan_osx_dynamic.dylib. The custom toolchain needs to be able to:

• Construct an appropriate -Wl,-rpath,@loader_path/... linker argument. The part after @loader_path needs to be a relative path from the cc_binary output directory to a path in the llvm external repository. It's something like @loader_path/../../external/llvm/lib/clang/15.0.0/lib/darwin, where the number of .. segments depends on the depth of the package containing the cc_binary target.
• Inject the DSO into the runfiles of the target.

Neither of these are currently possible to implement with the current toolchain APIs.

A hacky workaround suggested to me by @chancila is to (ab)use --custom_malloc with a cc_import(shared_library = ...) to inject a dependency on the sanitizer runtime DSO. This does work, but it's far from ideal.

[adrianimboden]

I also have the same challenge.

I tried quite some hours now to do the same thing using dynamic_runtime_lib in the cc_toolchain call now.

It seems to me that this parameter would be the canonical way to support this use case, no? This however does not work because of this cyclic dependency:

    //tmptest:foo (22fb1f1bd7b5d148386050801e8ccada95e2d7fb199b513d22942625b68f3cd1)
.-> @local_config_cc//:cc-compiler-k8 (22fb1f1bd7b5d148386050801e8ccada95e2d7fb199b513d22942625b68f3cd1)
|   @local_config_cc//:libc++ (22fb1f1bd7b5d148386050801e8ccada95e2d7fb199b513d22942625b68f3cd1)
`-- @local_config_cc//:cc-compiler-k8 (22fb1f1bd7b5d148386050801e8ccada95e2d7fb199b513d22942625b68f3cd1)

@local_config_cc//:libc++:

cc_import(
    name="libc++",
    hdrs=[],
    shared_library="install/cxxlib/fastbuild/lib/libc++.so.2",
)

It kind of makes sense, but what I do not understand is, why with the --custom_malloc works and dynamic_runtime_lib does not. After all, the same circular dependency should be an issue.

Or could the dynamic_runtime_lib maybe a little bit changed so that the toolchain dependency is handled like --custom_malloc?

[fmeum]

It kind of makes sense, but what I do not understand is, why with the --custom_malloc works and dynamic_runtime_lib does not. After all, the same circular dependency should be an issue.

The difference may be explained by --custom_malloc being a direct dependency of all cc_* target rather than one brought in by the toolchain. Essentially, with --custom_malloc, the dependency chain should look like this instead - free of cycles:

    //tmptest:foo (22fb1f1bd7b5d148386050801e8ccada95e2d7fb199b513d22942625b68f3cd1)
    @local_config_cc//:libc++ (22fb1f1bd7b5d148386050801e8ccada95e2d7fb199b513d22942625b68f3cd1)
    @local_config_cc//:cc-compiler-k8 (22fb1f1bd7b5d148386050801e8ccada95e2d7fb199b513d22942625b68f3cd1)

Since dynamic_runtime_lib is brought in by the toolchain, it can't itself depend on the toolchain.

Instead, I think that you are supposed to pass in file targets rather than cc_* targets. The Starlark version of cc_binary uses dynamic_runtime_lib only here: https://github.com/bazelbuild/bazel/blob/4201c697b9249acc02197b8d9951eaf09c2cddf4/src/main/starlark/builtins_bzl/common/cc/cc_binary.bzl#L251-L254

Its files are added to the runfiles of every cc_binary. If you replace your cc_import with a filegroup, they should thus be able to find the library at runtime. This part of CcToolchainProvider should ensure that the library is symlinked into the location that the default RPATHs point to.

@adrianimboden Could you try if that works?

[adrianimboden]

ah interesting, I thought that it must be a cc_import/cc_library so that the rpath thing works, but I have not tested the filegroup approach at all.

I will try that soon and write back here :+1:. Thank you for your fast response.

[adrianimboden]

I shortly tried it with this target:

filegroup(
    name="runtimelibs_group",
    srcs=[
        "install/cxxlib/fastbuild/lib/libc++.so.2",
        "install/cxxlib/fastbuild/lib/libunwind.so.1",
    ]
)

and

dynamic_runtime_lib=":runtimelibs_group", 

It does not link against the library. When I put --linkopt -lc++ to make it "work", the bazel-run directory also does not have the .so files in the tree at all:

$ find bazel-bin/
bazel-bin/
bazel-bin/tmptest
bazel-bin/tmptest/foo
bazel-bin/tmptest/_objs
bazel-bin/tmptest/_objs/foo
bazel-bin/tmptest/_objs/foo/main.o
bazel-bin/tmptest/_objs/foo/main.d
bazel-bin/tmptest/foo.runfiles_manifest
bazel-bin/tmptest/foo.runfiles
bazel-bin/tmptest/foo.runfiles/__main__
bazel-bin/tmptest/foo.runfiles/__main__/tmptest
bazel-bin/tmptest/foo.runfiles/__main__/tmptest/foo
bazel-bin/tmptest/foo.runfiles/MANIFEST

I think, using cc_import or cc_library is needed to make linking and rpath possible.

[yuzhy8701]

@adrianimboden I think your problem is that bazel by default links binary in static mode - and it will use static_runtime_lib, not dynamic_runtime_lib. (You can add "-v" to the compiler flags to see the actual linker command called by clang.) (Note: you cannot feed so files to static_runtime_lib or bazel will crash with hard-to-understand errors).

Currently there are 2 conditions to make dynamic_runtime_lib work:

1. You have to enable the feature static_link_cpp_runtimes (If you did not enable the no_legacy_features feature, it should be auto added by bazel).
2. You have to link the binary in dynamic mode (either the --dynamic_mode=fully build flag or the linkstatic=False target attribute).

A side effect of doing 2 above is that all your cc_library targets will also get linked dynamically to your binary - with some long and ugly names to ensure uniqueness.

Alternatively, you can link the runtime statically, by feeding libc++.a etc to the static_runtime_lib.

I think right now the real problem is, it is impossible to enforce at toolchain level, that a library is always dynamically linked and in RPATH, regardless of link mode (which is what John's comment mentions).

[adrianimboden]

I will try that when I get back to this variant later. I will go on with the --custom_malloc approach for now. This seems to be easy to change to the toolchain approach once it is supported.

Normally, I link statically also. There everything works as expected. I need the dynamic way only for certain cases with sanitizers and such.

Thank you very much for your help.

For future reference, If I can test anything corresponding to this feature request, I will happily do some testing :smile:

[adrianimboden]

Ok i have to admit that I did not come very far with --custom_malloc :smile:. The reason is that tools that are being used in the build process itself (in my case ubp from grpc) will not use the given --custom_malloc. I don't now why that makes sense at all, but so be it.

So I tried your suggestion anyway and adding static_link_cpp_runtimes in combination with dynamic_runtime_lib works. I had to add static_runtime_lib with an empty srcs set because there was some error:

ERROR: /build_cache/bzl/root/690491f57e04da302046e3d5841bd6e3/external/bazel_tools/src/tools/launcher/BUILD:9:14: in cc_binary rule @bazel_tools//src/tools/launcher:launcher: Toolchain supports embedded runtimes, but didn't provide static_runtime_lib attribute.
ERROR: /build_cache/bzl/root/690491f57e04da302046e3d5841bd6e3/external/bazel_tools/src/tools/launcher/BUILD:9:14: in cc_binary rule @bazel_tools//src/tools/launcher:launcher: 
Traceback (most recent call last):                                                                                                      
        File "/virtual_builtins_bzl/common/cc/cc_binary.bzl", line 921, column 59, in _impl
        File "/virtual_builtins_bzl/common/cc/cc_binary.bzl", line 744, column 108, in cc_binary_impl
        File "/virtual_builtins_bzl/common/cc/cc_binary.bzl", line 509, column 40, in _create_transitive_linking_actions
Error in link: Toolchain supports embedded runtimes, but didn't provide static_runtime_lib attribute.
ERROR: /build_cache/bzl/root/690491f57e04da302046e3d5841bd6e3/external/bazel_tools/src/tools/launcher/BUILD:9:14: Analysis of target '@bazel_tools//src/tools/launcher:launcher' failed

However, even after this change, tools seem to get built with the static runtime all the time. Is there a way to build tools with dynamic dependencies as well? Tool example:

ERROR: /build_cache/bzl/root/690491f57e04da302046e3d5841bd6e3/external/upb/upbc/BUILD:44:10: Linking external/upb/upbc/protoc-gen-upb [for tool] failed

Edit: or is there a way to select a different toolchain/compiler for the tools altogether? I did not find documentation regarding tools.

[adrianimboden]

Also, interestingly cc_test targets do not use the given dynamic_runtime_lib or static_runtime_lib. cc_binary targets work fine.

Edit: seems to be related to https://github.com/bazelbuild/bazel/issues/3592

[fmeum]

That's because both --custom_malloc and --dynamic_mode are reset to their default values in the exec configuration (https://cs.opensource.google/bazel/bazel/+/b927d81ea4bb9d2b99f23109d66454e108914cb8:src/main/java/com/google/devtools/build/lib/rules/cpp/CppOptions.java;l=345;drc=4201c697b9249acc02197b8d9951eaf09c2cddf4;bpv=1;bpt=1) and there don't seem to be corresponding --host_ flags. Whether that's an oversight or intentional I don't know.

Starlark cc_test doesn't contain code to use the runtime libraries - again, I don't know why.

[trybka]

Starlark cc_test doesn't contain code to use the runtime libraries - again, I don't know why.

Can you elaborate? cc_test in Starlark just delegates its linking to the cc_binary implementation (much the same way the Java version did).

Does the Starlark version behave differently from the native impl?

[fmeum]

Starlark cc_test doesn't contain code to use the runtime libraries - again, I don't know why.

Can you elaborate? cc_test in Starlark just delegates its linking to the cc_binary implementation (much the same way the Java version did).

Does the Starlark version behave differently from the native impl?

Sorry for the false alarm, cc_test does inherit the dynamic runtime library handling from cc_binary. I don't know how the native implementation behaved, but if it delegated in the same way, nothing should have changed.

[adrianimboden]

Yeah I think the cc_test part was more related to https://github.com/bazelbuild/bazel/issues/3592, which is a different thing.

In my current setup, it is no longer a problem (but I did a lot of changes since then, most with your help here, thanks again).

Right now, it looks like this (cc_test which uses genrule which uses cc_binary):

$ bazel build --dynamic_mode=off [snip]
$ ldd bazel-bin/some/cc_binary/bin
        linux-vdso.so.1 (0x00007ffe411d4000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f390ce93000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f390cd44000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f390cd3e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f390cb4c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f390d790000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f390cb40000)
$ ldd bazel-bin/genrule/tool/bin
        linux-vdso.so.1 (0x00007ffdd547f000)
        libc++.so.2 => /home/thingdust/src/bazel-bin/generators/../_solib___Ccc-compiler-k8/libc++.so.2 (0x00007fe029574000)
        libunwind.so.1 => /home/thingdust/src/bazel-bin/generators/../_solib___Ccc-compiler-k8/libunwind.so.1 (0x00007fe029549000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe029526000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe0293d5000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe0293cf000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe0293c5000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe0291d3000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe02a473000)
        libatomic.so.1 => /lib/x86_64-linux-gnu/libatomic.so.1 (0x00007fe0291c9000)
$ ldd bazel-bin/cc_test/bin
        linux-vdso.so.1 (0x00007ffe915f8000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f71015c2000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7101473000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f710146d000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f710127b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7102af5000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f710126f000)

It is not really a problem, I was just suprised that the tool builds do not follow the same rules as the other cc_binary rules.

After all, It may be better to have a toolchain that works for static and dynamically linked builds :) This I did not have before.

[sauravnith]

Hello just want to check what was the resolution here . We are also using a custom c++ toolchain and binaries created have a dependency on libstdc++. For now we are specifying "--test_env" in .bazelrc to set LD_LIBRARY_PATH to the location of "libstdc++".

[yuzhy8701]

We recently also came across the need to always dynamically link a hermetic libc++ on macOS (together with the hermetic sanitizer libs that requires dynamic linking). --custom_malloc won't work because it only accepts a single lib (and, it doesn't work well with toolchain resolution).

Considering what is currently possible with a custom toolchain and https://github.com/bazelbuild/bazel/issues/16520#issuecomment-1315627908, I think this issue can be resolved by adding a additional_dynamic_libs (?) attribute to cc_toolchain rule, which does only the following:

1. If not empty, issue a rapth entry to the toolchain's solib directory when linking binaries.
2. If not empty, create the solib directory and include the lib files in the runfiles so the binaries can find them during runtime.

Compared with the current dynamic_runtime_lib attribute, the proposed attribute:

• Does not bother with adding library search paths or the lib names to the link command.
• Does not care what linking mode is being used (since it doesn't actually link the libs).
• Does not check the file extensions or parse file names in any way.

This should make the attribute straightforward to implement, while leaving the rest to toolchain authors. Toolchain authors can just write custom rules to check inputs, generate features with link flags, pass the files to this attribute and the loop is closed.

@fmeum @oquenchil any thoughts?