How to build cpp with distributed thinlto?

[mheiber]

How does one build cpp with distributed thinlto? I tried adding enable_distributed_thinlto = True, to the attributes for cxx_binary in examples/with_prelude/cpp/hello_world, but this doesn't seem to do anything: buck2 log what-ran and inspecting buck-out don't show any .bc files.

I'm not blocked and am asking out of curiosity so I can play with distributed thinlto. Thanks for your help

[ndmitchell]

You also need a C++ toolchain that has supports_distributed_thinlto at least. I'm not sure if that is sufficient or if there are other steps required. We should really document this somewhere. I'll reach out to the people who wrote it internally.

[stolyaroleh]

It looks like this patch is also required. I applied it and built a custom Clang 16 toolchain, only to end up with duplicate symbol errors:

ld.lld: error: duplicate symbol: __EH_FRAME_LIST_END__
>>> defined at /nix/store/zv7qi2ca0aylln5hi0sg0hqcfglhqsym-clang-wrapper-16.0.1/resource-root/lib/linux/clang_rt.crtend-x86_64.o:(__EH_FRAME_LIST_END__)
>>> defined at /nix/store/zv7qi2ca0aylln5hi0sg0hqcfglhqsym-clang-wrapper-16.0.1/resource-root/lib/linux/clang_rt.crtend-x86_64.o:(.eh_frame+0x0)

ld.lld: error: duplicate symbol: __dso_handle
>>> defined at /nix/store/zv7qi2ca0aylln5hi0sg0hqcfglhqsym-clang-wrapper-16.0.1/resource-root/lib/linux/clang_rt.crtbegin-x86_64.o:(__dso_handle)
>>> defined at /nix/store/zv7qi2ca0aylln5hi0sg0hqcfglhqsym-clang-wrapper-16.0.1/resource-root/lib/linux/clang_rt.crtbegin-x86_64.o:(.data.rel.local+0x0)

ld.lld: error: duplicate symbol: _init
>>> defined at /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/crti.o:(.init+0x0)
>>> defined at /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/crti.o:(.init+0x0)

ld.lld: error: duplicate symbol: _fini
>>> defined at /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/crti.o:(.fini+0x0)
>>> defined at /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/crti.o:(.fini+0x0)

ld.lld: error: duplicate symbol: _start
>>> defined at /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/Scrt1.o:(_start)
>>> defined at /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/Scrt1.o:(.text+0x0)

ld.lld: error: duplicate symbol: _IO_stdin_used
>>> defined at /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/Scrt1.o:(_IO_stdin_used)
>>> defined at /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/Scrt1.o:(.rodata.cst4+0x0)

ld.lld: error: duplicate symbol: __data_start
>>> defined at /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/Scrt1.o:(.data+0x0)
>>> defined at /nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/Scrt1.o:(.data+0x0)

What version of Clang did you use?

[christycylee]

Right now the only C++ build toolchain that supports_distributed_thinlto is llvm. Internally, we use clang12 and clang15 with required patches backported, the most important ones are https://reviews.llvm.org/D130229 and https://reviews.llvm.org/D133740. Trunk should work as well.

I applied it and built a custom Clang 16 toolchain, only to end up with duplicate symbol errors:

This looks like the same file was passed to the link step multiple times. Is this the thin link (thin_lto_index action) or final link (thin_lto_link action)?

[stolyaroleh]

Internally, we use clang12 and clang15 with required patches backported

Seems like I am missing some important patches.

Is this the thin link (thin_lto_index action) or final link (thin_lto_link action)?

Final link

This looks like the same file was passed to the link step multiple times.

I can see duplicate libunwind.so and libc++abi.so in my final_link_index:

~/s/buck2-experiment ❯❯❯ cat buck-out/v2/gen/root/524f8da68ea2a374/__main__/main.final_link_index

/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/Scrt1.o
/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/crti.o
/nix/store/zv7qi2ca0aylln5hi0sg0hqcfglhqsym-clang-wrapper-16.0.1/resource-root/lib/linux/clang_rt.crtbegin-x86_64.o
/nix/store/c3zgr64h1pfjb2i65vb28sxkd1a7w243-libcxxabi-16.0.1/lib/libc++abi.so
buck-out/v2/gen/root/524f8da68ea2a374/__main__/__objects__/main.cpp.pic.o
buck-out/v2/gen/root/524f8da68ea2a374/__print__/lib_print.so
/nix/store/a01vfl384lkmps0fr4hkl1ywyq5ykj15-libcxx-16.0.1/lib/libc++.so.1
/nix/store/c3zgr64h1pfjb2i65vb28sxkd1a7w243-libcxxabi-16.0.1/lib/libc++abi.so
/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/libm.so.6
/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/libmvec.so.1
/nix/store/5fqm512bd17cccsvlw06w55gba1hkflz-libunwind-16.0.1/lib/libunwind.so
/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/libc.so.6
/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/ld-linux-x86-64.so.2
/nix/store/5fqm512bd17cccsvlw06w55gba1hkflz-libunwind-16.0.1/lib/libunwind.so
/nix/store/zv7qi2ca0aylln5hi0sg0hqcfglhqsym-clang-wrapper-16.0.1/resource-root/lib/linux/clang_rt.crtend-x86_64.o
/nix/store/yaz7pyf0ah88g2v505l38n0f3wg2vzdj-glibc-2.37-8/lib/crtn.o

[christycylee]

I can see duplicate libunwind.so and libc++abi.so in my final_link_index:

The prefixes suggest that are not built from source, perhaps they came from the toolchain or the platform? buck2 build <target_name>[index.argsfile] will return the path to the argsfile used in thin link, while buck2 build <target_name>[linker.argsfile] will return the path to the argsfile used in the final link.