Fleetbench-proto build fails with Clang+ThinLTO

[iqbalIntel]

ERROR: /root/fleetbench/fleetbench/proto/BUILD:164:28: Linking fleetbench/proto/proto_benchmark failed: (Exit 1): clang failed: error executing command   (cd /root/.cache/bazel/_bazel_root/9906b1d63bb73ec34cafd40427c4e498/sandbox/linux-sandbox/476/execroot/com_google_fleetbench && \   exec env - \     PATH=/root/.cargo/bin:/root/dcsomc:/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin:/root/bin \     PWD=/proc/self/cwd \   /usr/bin/clang @bazel-out/k8-opt/bin/fleetbench/proto/proto_benchmark-2.params) Execution platform: @local_config_platform//:host Use --sandbox_debug to see verbose messages from the sandbox /usr/bin/ld.gold: error: bazel-out/k8-opt/bin/fleetbench/proto/_objs/proto_benchmark/benchmark.o:1:3: invalid character /usr/bin/ld.gold: error: bazel-out/k8-opt/bin/fleetbench/proto/_objs/proto_benchmark/benchmark.o:1:3: syntax error, unexpected $end /usr/bin/ld.gold: error: bazel-out/k8-opt/bin/fleetbench/proto/_objs/proto_benchmark/benchmark.o: not an object or archive /usr/bin/ld.gold: error: bazel-out/k8-opt/bin/fleetbench/proto/_objs/benchmark_lib/lifecycle.o:1:3: invalid character /usr/bin/ld.gold: error: bazel-out/k8-opt/bin/fleetbench/proto/_objs/benchmark_lib/lifecycle.o:1:3: syntax error, unexpected $end /usr/bin/ld.gold: error: bazel-out/k8-opt/bin/fleetbench/proto/_objs/benchmark_lib/lifecycle.o: not an object or archive /usr/bin/ld.gold: internal error in remove_blocker, at token.h:161 clang-12: error: linker command failed with exit code 1 (use -v to see invocation) Target //fleetbench/proto:proto_benchmark failed to build

[rjogrady]

Thanks for the report!

Thin LTO doesn't work for me either, and I don't see an obvious fix. Will have to debug it.

However, I get different output from yours.

My command line:

$ bazel run --features=thin_lto --config=clang fleetbench/proto:proto_benchmark

I get some linker errors.

Your output sounds like maybe an out of date version of something.

This is on my system: $ bazel --version bazel 6.0.0

$ clang --version Debian clang version 14.0.6 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin

I don't think we have identified a minimum supported version yet, but it would be helpful to know what your command line was and what version of these tools you are using.

[iqbalIntel]

I am using older version of bazel and clang. I tried to upgrade but seems like fleetbench doesn't build with the latest version bazel (v6) + clang (v15)

$bazel --version bazel 4.2.1 $ clang --version clang version 12.0.0 (Red Hat 12.0.0-1.module_el8.5.0+840+21214faf) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /usr/bin

Here is my .bazelrc content - build --cxxopt='-std=c++17' build --cxxopt='-fcf-protection=none' build --copt='-fcf-protection=none' build --copt='-include' build --copt='cet.h' build --define=pfm=1 build --cxxopt='-O3' build --copt='-O3' build --cxxopt="-flto=thin" build --copt='-flto=thin'

Define the --config=asan-libfuzzer configuration.

build:asan-libfuzzer --@rules_fuzzing//fuzzing:cc_engine=@rules_fuzzing//fuzzing/engines:libfuzzer build:asan-libfuzzer --@rules_fuzzing//fuzzing:cc_engine_instrumentation=libfuzzer build:asan-libfuzzer --@rules_fuzzing//fuzzing:cc_engine_sanitizer=asan

Define --config=clang. clang must be in the PATH.

build:clang --repo_env=CC=clang --repo_env=CXX=clang++

Prevent warnings about gcc-specific arguments from --fdo_optimize.

build:clang --copt='-Wno-ignored-optimization-argument'

Allow instrprof_error::hash_mismatch profiles.

build:clang --copt='-Wno-error=backend-plugin'

Cache clang artifacts in a different directory.

build:clang --platform_suffix=clang

[rjogrady]

Thanks for that info.

It should build with bazel 6 now- as of a few weeks ago. Can you share the error you're seeing? Also might try a bazel clean --expunge to clear its cache after upgrading.

[iqbalIntel]

This is the error I am seeing with bazel5 + clang15

ERROR: /root/.cache/bazel/_bazel_root/9906b1d63bb73ec34cafd40427c4e498/external/com_google_tcmalloc/tcmalloc/internal/BUILD:529:11: Compiling tcmalloc/internal/percpu.cc failed: (Exit 1): clang failed: error executing command (cd /root/.cache/bazel/_bazel_root/9906b1d63bb73ec34cafd40427c4e498/sandbox/linux-sandbox/333/execroot/com_google_fleetbench && \ exec env - \ PATH=/root/.cargo/bin:/root/dcsomc:/root/vchua/miniconda3/condabin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin:/root/bin \ PWD=/proc/self/cwd \ /usr/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/external/com_google_tcmalloc/tcmalloc/internal/_objs/percpu/percpu.d '-frandom-seed=bazel-out/k8-opt/bin/external/com_google_tcmalloc/tcmalloc/internal/_objs/percpu/percpu.o' -iquote external/com_google_tcmalloc -iquote bazel-out/k8-opt/bin/external/com_google_tcmalloc -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl '-std=c++17' -Werror -Wno-deprecated-declarations -Wno-deprecated-volatile -Wno-implicit-int-float-conversion -Wno-sign-compare -Wno-uninitialized -Wno-unused-function -Wno-unused-variable -no-canonical-prefixes -Wno-builtin-macro-redefined '-DDATE="redacted"' '-DTIMESTAMP="redacted"' '-DTIME="redacted"' -c external/com_google_tcmalloc/tcmalloc/internal/percpu.cc -o bazel-out/k8-opt/bin/external/com_google_tcmalloc/tcmalloc/internal/_objs/percpu/percpu.o) Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox In file included from external/com_google_tcmalloc/tcmalloc/internal/percpu.cc:14: In file included from external/com_google_tcmalloc/tcmalloc/internal/percpu.h:57: In file included from external/com_google_tcmalloc/tcmalloc/internal/logging.h:28: In file included from external/com_google_absl/absl/strings/str_cat.h:64: In file included from external/com_google_absl/absl/strings/numbers.h:43: In file included from external/com_google_absl/absl/base/internal/endian.h:22: In file included from external/com_google_absl/absl/base/casts.h:38: external/com_google_absl/absl/meta/type_traits.h:304:36: error: builtin has_trivial_destructor is deprecated; use is_trivially_destructible instead [-Werror,-Wdeprecated-builtins] : std::integral_constant<bool, has_trivial_destructor(T) && ^ external/com_google_absl/absl/meta/type_traits.h:357:36: error: builtin has_trivial_constructor is deprecated; use is_trivially_constructible instead [-Werror,-Wdeprecated-builtins] : std::integral_constant<bool, has_trivial_constructor(T) && ^ external/com_google_absl/absl/meta/type_traits.h:509:17: error: builtin has_trivial_assign is deprecated; use is_trivially_assignable instead [-Werror,-Wdeprecated-builtins] bool, __has_trivial_assign(typename std::remove_reference::type) && ^ 3 errors generated. Target //fleetbench/proto:proto_benchmark failed to build INFO: Elapsed time: 14.895s, Critical Path: 3.45s INFO: 581 processes: 276 internal, 305 linux-sandbox. FAILED: Build did NOT complete successfully

[rjogrady]

I've just updated our version of abseil to handle this warning from clang 15 and above. If you get latest and try again, do you get any further? Thanks for your patience!

[iqbalIntel]

Thanks. It builds now with clang15. However, ThinLTO still failing the error below.

ERROR: /root/fleetbench/fleetbench/proto/BUILD:164:28: Linking fleetbench/proto/proto_benchmark failed: (Exit 1): clang-15 failed: error executing command (cd /root/.cache/bazel/_bazel_root/9906b1d63bb73ec34cafd40427c4e498/sandbox/linux-sandbox/959/execroot/com_google_fleetbench && \ exec env - \ PATH=/root/.cargo/bin:/root/dcsomc:/root/vchua/miniconda3/condabin:/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin:/root/bin \ PWD=/proc/self/cwd \ /usr/bin/clang-15 @bazel-out/k8-opt-clang/bin/fleetbench/proto/proto_benchmark-2.params)

Configuration: 7abdad4ee09316d659be3a11299851c41489296ae13b90618ea7384d905bdc68

Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging bazel-out/k8-opt-clang/bin/fleetbench/proto/proto_benchmark.lto/external/com_google_benchmark/_objs/benchmark/benchmark.o:benchmark.cc:function _GLOBALsub_I_benchmark.cc: error: undefined reference to 'pfm_initialize' bazel-out/k8-opt-clang/bin/fleetbench/proto/proto_benchmark.lto/external/com_google_benchmark/_objs/benchmark/benchmark_runner.o:benchmark_runner.cc:function _GLOBALsub_I_benchmark_runner.cc: error: undefined reference to 'pfm_initialize' bazel-out/k8-opt-clang/bin/fleetbench/proto/proto_benchmark.lto/external/com_google_benchmark/_objs/benchmark/perf_counters.o:perf_counters.cc:function benchmark::internal::PerfCounters::Create(std::vector<std::cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::cxx11::basic_string<char, std::char_traits, std::allocator > > > const&): error: undefined reference to 'pfm_get_os_event_encoding' bazel-out/k8-opt-clang/bin/fleetbench/proto/proto_benchmark.lto/external/com_google_benchmark/_objs/benchmark/perf_counters.o:perf_counters.cc:function _GLOBAL__sub_I_perf_counters.cc: error: undefined reference to 'pfm_initialize' bazel-out/k8-opt-clang/bin/fleetbench/proto/proto_benchmark.lto/external/com_google_tcmalloc/tcmalloc/_objs/tcmalloc/tcmalloc.o:tcmalloc.cc:function tcmalloc::tcmalloc_internal::cpu_cache_internal::CpuCache::Deallocate(void, unsigned long)::Helper::Overflow(int, unsigned long, void, void): error: undefined reference to 'TcmallocSlab_Internal_PopBatch' bazel-out/k8-opt-clang/bin/fleetbench/proto/proto_benchmark.lto/external/com_google_tcmalloc/tcmalloc/_objs/tcmalloc/tcmalloc.o:tcmalloc.cc:function tcmalloc::tcmalloc_internal::cpu_cache_internal::CpuCache::UpdateCapacity(int, unsigned long, unsigned long, bool, tcmalloc::tcmalloc_internal::cpu_cache_internal::CpuCache::ObjectsToReturn): error: undefined reference to 'TcmallocSlab_Internal_PerCpuCmpxchg64' bazel-out/k8-opt-clang/bin/fleetbench/proto/proto_benchmark.lto/external/com_google_tcmalloc/tcmalloc/_objs/tcmalloc/tcmalloc.o:tcmalloc.cc:function tcmalloc::tcmalloc_internal::cpu_cache_internal::CpuCache::UpdateCapacity(int, unsigned long, unsigned long, bool, tcmalloc::tcmalloc_internal::cpu_cache_internal::CpuCache::ObjectsToReturn*): error: undefined reference to 'TcmallocSlab_Internal_PerCpuCmpxchg64' bazel-out/k8-opt-clang/bin/fleetbench/proto/proto_benchmark.lto/external/com_google_tcmalloc/tcmalloc/_objs/tcmalloc/tcmalloc.o:tcmalloc.cc:function tcmalloc::tcmalloc_internal::cpu_cache_internal::CpuCache::Refill(int, unsigned long): error: undefined reference to 'TcmallocSlab_Internal_PushBatch' clang-15: error: linker command failed with exit code 1 (use -v to see invocation) Target //fleetbench/proto:proto_benchmark failed to build INFO: Elapsed time: 68.459s, Critical Path: 62.84s INFO: 1119 processes: 161 internal, 958 linux-sandbox. FAILED: Build did NOT complete successfully

[rjogrady]

This likely means you are using the 'gold' linker rather than 'lld'. There seems to be a bug with gold.

Please try installing lld (sudo apt-get install lld, or whatever is appropriate for your distro) and doing a bazel clean --expunge and see if that helps.

[iqbalIntel]

Thanks. It worked with lld and clang15. I tried with lld before with clang12 which didn't work.

[rjogrady]

Great, thanks for working through that. I'm sure it'll help the next person, and we'll keep working on the docs. Will close this if it's working for you now.