The path of `lld` in not included in `-B` argument

YangKeao commented 1 year ago

Description of the bug:

I have both gcc and clang, and the gcc is the default compiler in /usr/bin/gcc. For the linker, I have both ld (in /usr/bin/ld, which is a soft link to /usr/bin/x86_64-pc-linux-gnu-ld), and lld (which located at /usr/lib/llvm/15/bin/lld, and /usr/lib/llvm/15/bin is in the $PATH).

Then the bazel priorities the lld, and passes the -fuse-ld=lld to the gcc. However, the $PATH is not passed to the sandbox, so gcc failed to find the lld. Add an argument -B/usr/lib/llvm/15/bin should fix this problem.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

git clone https://github.com/pingcap/tidb.git
cd tidb
make bazel_prepare

Which operating system are you running Bazel on?

Linux / Gentoo

What is the output of `bazel info release`?

release 6.2.1

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

No response

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

/home/yangkeao/.cache/bazel/_bazel_yangkeao/install/50a5a3eaeaa5297d31568ab6a3a702d9/linux-sandbox -t 15 -w /home/yangkeao/.cache/bazel/_bazel_yangkeao/d58c0161a9f3fe46dadee0c01600b75a/sandbox/linux-sandbox/1025/execroot/__main__ -w /tmp -w /dev/shm -S /home/yangkeao/.cache/bazel/_bazel_yangkeao/d58c0161a9f3fe46dadee0c01600b75a/sandbox/linux-sandbox/1025/stats.out -D -- bazel-out/k8-opt-exec-2B5CBBC6-ST-b62283a3abad/bin/external/go_sdk/builder_reset/builder '-param=bazel-out/k8-opt-exec-2B5CBBC6-ST-b62283a3abad/bin/build/tidb_nogo_actual_/tidb_nogo_actual-0.params' -- -extld /usr/bin/gcc '-buildid=redacted' -s -w -extldflags '-fuse-ld=lld -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lm')
....
external/go_sdk/pkg/tool/linux_amd64/link: running /usr/bin/gcc failed: exit status 1
collect2: fatal error: cannot find 'ld'
compilation terminated.

link: error running subcommand external/go_sdk/pkg/tool/linux_amd64/link: exit status 2
1688388867.058658083: src/main/tools/linux-sandbox-pid1.cc:538: wait returned pid=2, status=0x100
1688388867.058681773: src/main/tools/linux-sandbox-pid1.cc:556: child exited normally with code 1
1688388867.059128293: src/main/tools/linux-sandbox.cc:233: child exited normally with code 1

The external/local_config_cc/BUILD shows:

...
    link_flags = ["-fuse-ld=lld",
    "-Wl,-no-as-needed",
    "-Wl,-z,relro,-z,now",
    "-B/usr/bin",
    "-pass-exit-codes"],
...

YangKeao commented 1 year ago

I have tested the following patch:

diff --git a/tools/cpp/unix_cc_configure.bzl b/tools/cpp/unix_cc_configure.bzl
index df2034786b..13012769ae 100644
--- a/tools/cpp/unix_cc_configure.bzl
+++ b/tools/cpp/unix_cc_configure.bzl
@@ -451,6 +451,9 @@ def configure_unix_toolchain(repository_ctx, cpu_value, overriden_tools):
         ld_path = repository_ctx.path(tool_paths["ld"])
         if ld_path.dirname != cc_path.dirname:
             bin_search_flags.append("-B" + str(ld_path.dirname))
+    if gold_or_lld_linker_path and not is_clang:
+        linker_path = repository_ctx.path(repository_ctx.which(gold_or_lld_linker_path))
+        bin_search_flags.append("-B" + str(linker_path.dirname))
     coverage_compile_flags, coverage_link_flags = _coverage_flags(repository_ctx, darwin)
     print_resource_dir_supported = _is_compiler_option_supported(
         repository_ctx,

This patch actually solved the problem. It adds the -B/usr/lib/llvm/15/bin to the arguments.

But I'm not sure whether go compiler supports mix using gcc and lld, when I tested it on my project (tidb) it fails to compile:

fatal error: invalid function symbol table
runtime: panic before malloc heap initialized

runtime stack:
runtime.throw({0x55fc2fc27cd3?, 0x0?})
        GOROOT/src/runtime/panic.go:1047 +0x5d fp=0x7ffe2853fda0 sp=0x7ffe2853fd70 pc=0x55fc30025c9d
runtime.moduledataverify1(0x7ffe2853fecc?)
        GOROOT/src/runtime/symtab.go:627 +0x806 fp=0x7ffe2853fea0 sp=0x7ffe2853fda0 pc=0x55fc300435a6
runtime.moduledataverify(...)
        GOROOT/src/runtime/symtab.go:613
runtime.schedinit()
        GOROOT/src/runtime/proc.go:710 +0x3a fp=0x7ffe2853ff00 sp=0x7ffe2853fea0 pc=0x55fc3002977a
runtime.rt0_go()
        src/runtime/asm_amd64.s:349 +0x11c fp=0x7ffe2853ff08 sp=0x7ffe2853ff00 pc=0x55fc30054ddc

Maybe it's not correct to prioritize the lld and gold linker rather than the default one :thinking: (but I'm not sure about the behavior, and why they are specially handled in the bazel code base).

BTW, CC=clang is a workaround for me now :facepalm:.

jcpetruzza commented 1 year ago

Fwiw, this also happens when trying to use bazel from GitHub Actions on an Ubuntu host: lld is picked there, the path is wrong so the build fails with:

collect2: fatal error: cannot find 'ld'

Passing --linkopt=-fuse-ld=gold seems to be a workaround

bazelbuild / bazel