uber / hermetic_cc_toolchain

Bazel C/C++ toolchain for cross-compiling C/C++ programs
MIT License
273 stars 44 forks source link

Fail to link Go binary on 1.22 #171

Open sluongng opened 4 months ago

sluongng commented 4 months ago

Example failure here https://app.buildbuddy.io/invocation/abd101b0-f5e3-4c5a-83ad-a459949a2905 with the remotely executed action here https://app.buildbuddy.io/invocation/abd101b0-f5e3-4c5a-83ad-a459949a2905?actionDigest=2141d6f2153f2060aef65156f6f74d66f2737bf6a25a818bb3e2419fb47c01b1%2F261&executeResponseDigest=77fe59ae3016c296887efe46b0b4d5c782385a865a92dc511672fd971ebf8947%2F127#action

Error message

runtime.gcdata: missing Go type information for global symbol .dynsym: size 72
net(.text): relocation target __errno_location not defined
net(.text): relocation target getaddrinfo not defined
net(.text): relocation target free not defined
net(.text): relocation target freeaddrinfo not defined
net(.text): relocation target gai_strerror not defined
runtime/cgo(.text): relocation target stderr not defined
runtime/cgo(.text): relocation target fwrite not defined
runtime/cgo(.text): relocation target vfprintf not defined
runtime/cgo(.text): relocation target fputc not defined
runtime/cgo(.text): relocation target abort not defined
runtime/cgo(.text): relocation target pthread_create not defined
runtime/cgo(.text): relocation target nanosleep not defined
runtime/cgo(.text): relocation target pthread_detach not defined
runtime/cgo(.text): relocation target stderr not defined
runtime/cgo(.text): relocation target strerror not defined
runtime/cgo(.text): relocation target fprintf not defined
runtime/cgo(.text): relocation target abort not defined
runtime/cgo(.text): relocation target pthread_mutex_lock not defined
runtime/cgo(.text): relocation target pthread_cond_wait not defined
runtime/cgo(.text): relocation target pthread_key_create not defined
external/go_sdk_linux/pkg/tool/linux_amd64/link: too many errors
link: error running subcommand PATH=external/zig_sdk/tools:external/zig_sdk/tools/x86_64-linux-musl:/bin:/usr/bin \
container=podman \
GCOV=/dev/null \
LANGUAGE=C.UTF-8 \
GOEXPERIMENT=nocoverageredesign \
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 \
TERM=xterm \
CC=/usr/local/bin/clang \
GOROOT_FINAL=GOROOT \
GOOS=linux \
CGO_ENABLED=1 \
GOPATH= \
GOROOT=bazel-out/linux_x86_64-opt-exec/bin/external/io_bazel_rules_go/stdlib_ \
MSAN_SYMBOLIZER_PATH=/usr/local/bin/llvm-symbolizer \
TSAN_SYMBOLIZER_PATH=/usr/local/bin/llvm-symbolizer \
DEBIAN_FRONTEND=noninteractive \
UBSAN_SYMBOLIZER_PATH=/usr/local/bin/llvm-symbolizer \
LD_LIBRARY_PATH=/usr/local/lib \
GOARCH=amd64 \
LC_ALL=C.UTF-8 \
ASAN_SYMBOLIZER_PATH=/usr/local/bin/llvm-symbolizer \
LANG=C.UTF-8 \
GOTOOLCHAIN=local \
HOSTNAME=localhost \
HOME=/root \
external/go_sdk_linux/pkg/tool/linux_amd64/link -importcfg /buildbuddy/remotebuilds/9cda9ace-9c5f-4a7d-bf72-7bb3386c6c56/bazel-out/linux_x86_64-opt-exec-ST-da86d8d417d3/bin/vet_actual_/importcfg1901904132 -X github.com/sluongng/nogo-analyzer/staticcheck/QF1002.name=QF1002 -X github.com/sluongng/nogo-analyzer/staticcheck/QF1007.name=QF1007 -X github.com/sluongng/nogo-analyzer/staticcheck/QF1009.name=QF1009 -X github.com/sluongng/nogo-analyzer/staticcheck/QF1010.name=QF1010 -X github.com/sluongng/nogo-analyzer/staticcheck/S1000.name=S1000 -X github.com/sluongng/nogo-analyzer/staticcheck/S1001.name=S1001 -X github.com/sluongng/nogo-analyzer/staticcheck/S1002.name=S1002 -X github.com/sluongng/nogo-analyzer/staticcheck/S1003.name=S1003 -X github.com/sluongng/nogo-analyzer/staticcheck/S1004.name=S1004 -X github.com/sluongng/nogo-analyzer/staticcheck/S1005.name=S1005 -X github.com/sluongng/nogo-analyzer/staticcheck/S1006.name=S1006 -X github.com/sluongng/nogo-analyzer/staticcheck/S1007.name=S1007 -X github.com/sluongng/nogo-analyzer/staticcheck/S1008.name=S1008 -X github.com/sluongng/nogo-analyzer/staticcheck/S1009.name=S1009 -X github.com/sluongng/nogo-analyzer/staticcheck/S1010.name=S1010 -X github.com/sluongng/nogo-analyzer/staticcheck/S1011.name=S1011 -X github.com/sluongng/nogo-analyzer/staticcheck/S1012.name=S1012 -X github.com/sluongng/nogo-analyzer/staticcheck/S1016.name=S1016 -X github.com/sluongng/nogo-analyzer/staticcheck/S1017.name=S1017 -X github.com/sluongng/nogo-analyzer/staticcheck/S1018.name=S1018 -X github.com/sluongng/nogo-analyzer/staticcheck/S1019.name=S1019 -X github.com/sluongng/nogo-analyzer/staticcheck/S1020.name=S1020 -X github.com/sluongng/nogo-analyzer/staticcheck/S1021.name=S1021 -X github.com/sluongng/nogo-analyzer/staticcheck/S1023.name=S1023 -X github.com/sluongng/nogo-analyzer/staticcheck/S1024.name=S1024 -X github.com/sluongng/nogo-analyzer/staticcheck/S1025.name=S1025 -X github.com/sluongng/nogo-analyzer/staticcheck/S1028.name=S1028 -X github.com/sluongng/nogo-analyzer/staticcheck/S1029.name=S1029 -X github.com/sluongng/nogo-analyzer/staticcheck/S1030.name=S1030 -X github.com/sluongng/nogo-analyzer/staticcheck/S1031.name=S1031 -X github.com/sluongng/nogo-analyzer/staticcheck/S1032.name=S1032 -X github.com/sluongng/nogo-analyzer/staticcheck/S1033.name=S1033 -X github.com/sluongng/nogo-analyzer/staticcheck/S1034.name=S1034 -X github.com/sluongng/nogo-analyzer/staticcheck/S1035.name=S1035 -X github.com/sluongng/nogo-analyzer/staticcheck/S1036.name=S1036 -X github.com/sluongng/nogo-analyzer/staticcheck/S1037.name=S1037 -X github.com/sluongng/nogo-analyzer/staticcheck/S1038.name=S1038 -X github.com/sluongng/nogo-analyzer/staticcheck/S1039.name=S1039 -X github.com/sluongng/nogo-analyzer/staticcheck/S1040.name=S1040 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1000.name=SA1000 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1001.name=SA1001 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1002.name=SA1002 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1003.name=SA1003 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1004.name=SA1004 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1005.name=SA1005 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1006.name=SA1006 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1007.name=SA1007 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1008.name=SA1008 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1010.name=SA1010 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1011.name=SA1011 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1012.name=SA1012 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1013.name=SA1013 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1014.name=SA1014 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1015.name=SA1015 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1016.name=SA1016 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1017.name=SA1017 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1018.name=SA1018 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1020.name=SA1020 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1021.name=SA1021 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1023.name=SA1023 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1025.name=SA1025 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1026.name=SA1026 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1027.name=SA1027 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1028.name=SA1028 -X github.com/sluongng/nogo-analyzer/staticcheck/SA1030.name=SA1030 -X github.com/sluongng/nogo-analyzer/staticcheck/SA2000.name=SA2000 -X github.com/sluongng/nogo-analyzer/staticcheck/SA2001.name=SA2001 -X github.com/sluongng/nogo-analyzer/staticcheck/SA2002.name=SA2002 -X github.com/sluongng/nogo-analyzer/staticcheck/SA2003.name=SA2003 -X github.com/sluongng/nogo-analyzer/staticcheck/SA3000.name=SA3000 -X github.com/sluongng/nogo-analyzer/staticcheck/SA3001.name=SA3001 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4000.name=SA4000 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4001.name=SA4001 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4003.name=SA4003 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4004.name=SA4004 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4005.name=SA4005 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4006.name=SA4006 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4008.name=SA4008 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4012.name=SA4012 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4013.name=SA4013 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4014.name=SA4014 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4015.name=SA4015 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4016.name=SA4016 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4017.name=SA4017 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4018.name=SA4018 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4019.name=SA4019 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4020.name=SA4020 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4021.name=SA4021 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4022.name=SA4022 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4023.name=SA4023 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4024.name=SA4024 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4025.name=SA4025 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4026.name=SA4026 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4027.name=SA4027 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4028.name=SA4028 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4029.name=SA4029 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4030.name=SA4030 -X github.com/sluongng/nogo-analyzer/staticcheck/SA4031.name=SA4031 -X github.com/sluongng/nogo-analyzer/staticcheck/SA5000.name=SA5000 -X github.com/sluongng/nogo-analyzer/staticcheck/SA5002.name=SA5002 -X github.com/sluongng/nogo-analyzer/staticcheck/SA5003.name=SA5003 -X github.com/sluongng/nogo-analyzer/staticcheck/SA5004.name=SA5004 -X github.com/sluongng/nogo-analyzer/staticcheck/SA5005.name=SA5005 -X github.com/sluongng/nogo-analyzer/staticcheck/SA5007.name=SA5007 -X github.com/sluongng/nogo-analyzer/staticcheck/SA5008.name=SA5008 -X github.com/sluongng/nogo-analyzer/staticcheck/SA5009.name=SA5009 -X github.com/sluongng/nogo-analyzer/staticcheck/SA5010.name=SA5010 -X github.com/sluongng/nogo-analyzer/staticcheck/SA5012.name=SA5012 -X github.com/sluongng/nogo-analyzer/staticcheck/SA6000.name=SA6000 -X github.com/sluongng/nogo-analyzer/staticcheck/SA6001.name=SA6001 -X github.com/sluongng/nogo-analyzer/staticcheck/SA6002.name=SA6002 -X github.com/sluongng/nogo-analyzer/staticcheck/SA6003.name=SA6003 -X github.com/sluongng/nogo-analyzer/staticcheck/SA6005.name=SA6005 -X github.com/sluongng/nogo-analyzer/staticcheck/SA9002.name=SA9002 -X github.com/sluongng/nogo-analyzer/staticcheck/SA9003.name=SA9003 -X github.com/sluongng/nogo-analyzer/staticcheck/SA9004.name=SA9004 -X github.com/sluongng/nogo-analyzer/staticcheck/SA9005.name=SA9005 -X github.com/sluongng/nogo-analyzer/staticcheck/SA9006.name=SA9006 -X github.com/sluongng/nogo-analyzer/staticcheck/SA9007.name=SA9007 -X github.com/sluongng/nogo-analyzer/staticcheck/SA9008.name=SA9008 -X github.com/sluongng/nogo-analyzer/staticcheck/ST1001.name=ST1001 -X github.com/sluongng/nogo-analyzer/staticcheck/ST1011.name=ST1011 -X github.com/sluongng/nogo-analyzer/staticcheck/ST1013.name=ST1013 -X github.com/sluongng/nogo-analyzer/staticcheck/ST1015.name=ST1015 -X github.com/sluongng/nogo-analyzer/staticcheck/ST1018.name=ST1018 -X github.com/sluongng/nogo-analyzer/staticcheck/ST1019.name=ST1019 -X github.com/sluongng/nogo-analyzer/staticcheck/U1000.name=U1000 -o /buildbuddy/remotebuilds/9cda9ace-9c5f-4a7d-bf72-7bb3386c6c56/bazel-out/linux_x86_64-opt-exec-ST-da86d8d417d3/bin/vet_actual_/vet_actual -extld external/zig_sdk/tools/x86_64-linux-musl/c++ -buildid=redacted -s -w -extldflags "-O2 -DNDEBUG" /buildbuddy/remotebuilds/9cda9ace-9c5f-4a7d-bf72-7bb3386c6c56/bazel-out/linux_x86_64-opt-exec-ST-da86d8d417d3/bin/vet_actual~nogo.a: exit status 2

On the surface, this looks like an incompatibility between Go 1.22 and @zig_sdk//toolchain:linux_amd64_musl like reported in https://github.com/golang/go/issues/44695

Additional info:

sluongng commented 4 months ago

FWIW, I can reproduce this with local execution as well, so this is not an RBE-related bug.

motiejus commented 4 months ago

There isn't much we can do but report this upstream somewhere. So I will offer some guidance on what to do next and close this issue.

Please create a small test case case: a Go file and a specific zig cc command. Then file an issue to Zig or Go.

motiejus commented 4 months ago

Also, feel free to add the link to the task to this issue, so whoever is following have a trail. I will subscribe to whatever you register and keep an eye on.

sluongng commented 4 months ago

I think closing this would delegate the burden of troubleshooting to the user of the toolchain. But users may not have the skill/knowledge needed to properly create a reproduce.

For example: event though Zig toolchain is being used, in the action above, all I could see is

CC=/usr/local/bin/clang

It's unclear to me, a user, how to map that and the PATH=...:external/zig_sdk/tools/x86_64-linux-musl:... to a zig cc call 🤔

Furthermore, I think this toolchain repo should consider upgrading rules_go dependency + go_sdk to verify the reported issue using the existing test suite and avoid regression. So if breakage is found in that process, the toolchain maintainer should investigate and potentially fix this issue instead of closing it prematurely.

eko commented 4 months ago

Hi @sluongng, have you come to any solution please? I have the same issue after upgrading to Go 1.22

sluongng commented 4 months ago

We are weighing our options. Most likely we gonna stop using x86_64-linux-musl and/or hermetic_cc_toolchain entirely until the problem is fixed, or a better musl toolchain becomes available (such as https://github.com/bazel-contrib/musl-toolchain).

I am testing out https://github.com/bazel-contrib/toolchains_llvm/ with a custom sysroot as an alternative solution. But our problem is rather narrow and unique so folks will have to decide for themselves.

motiejus commented 4 months ago

I think closing this would delegate the burden of troubleshooting to the user of the toolchain.

Sorry about this, you were right. Reopening and will pin for higher visibility.

motiejus commented 4 months ago

Afaik if things are like they were a year ago Uber may have this problem too, since it uses musl for some targets.

@linzhp have you upgraded to 1.22 yet?

linzhp commented 4 months ago

We are on 1.22.1, but we are using @zig_sdk//toolchain:linux_amd64_gnu.2.19 and @zig_sdk//toolchain:linux_arm64_gnu.2.28 by default as you configured before. It's working.

There is only one shell script using musl to compile a Cgo target, and it did fail with the same error, but I guess no one is calling that shell script...