Compiling mold with `icpx`

[TApplencourt]

Hi,

I work at Argonne National Lab on the Aurora project (https://www.alcf.anl.gov/aurora). icpx is the new Intel LLVM-based compiler that is part of the OneAPI. This thing is free, you can download and install it.

(To increase naming confusion, the icpx compiler is more or less just https://github.com/intel/llvm + some proprietary stuff)

For fun, I try to see if I can use mold with this compiler. I found two bugs. One I can resolve, one where I need help :)

Unknow argument -Kc++

$icpx -DMI_MALLOC_OVERRIDE -DMI_STATIC_LIB -I/home/tapplencourt/tmp/mold/third-party/mimalloc/include -O3 -DNDEBUG -fPIC -Kc++ -Wall -fvisibility=hidden -ftls-model=initial-exec -fno-builtin-malloc -std=gnu++17 -MD -MT CMakeFiles/mimalloc-static.dir/src/stats.c.o -MF CMakeFiles/mimalloc-static.dir/src/stats.c.o.d -o CMakeFiles/mimalloc-static.dir/src/stats.c.o -c /home/tapplencourt/tmp/mold/third-party/mimalloc/src/stats.c
clang++: error: unknown argument: '-Kc++'

PS: clang++ should be read icpx in the error message but 🤷🏽 . The fix is the following

index 8127e096..f5458453 100644
--- a/third-party/mimalloc/CMakeLists.txt
+++ b/third-party/mimalloc/CMakeLists.txt
@@ -179,7 +179,7 @@ if(MI_USE_CXX)
   if(CMAKE_CXX_COMPILER_ID MATCHES "AppleClang|Clang")
     list(APPEND mi_cflags -Wno-deprecated)
   endif()
-  if(CMAKE_CXX_COMPILER_ID MATCHES "Intel")
+  if(CMAKE_CXX_COMPILER_ID MATCHES "Intel" AND NOT CMAKE_CXX_COMPILER_ID MATCHES "IntelLLVM")
     list(APPEND mi_cflags -Kc++)
   endif()
 endif()
lines 1-13/13 (END)

Indeed -Kc++ is a legacy icpc compiler and not supported by icpx. CMake use IntelLLVM as a label for the new Intel compiler. I can make a PR if you want.

/usr/bin/ld: out/tbb/libs/libtbb.a: error adding symbols: archive has no index; run ranlib to add one

$icpx out/tar.o out/strerror.o out/perf.o out/main.o out/hyperloglog.o out/filepath.o out/demangle.o out/compress.o out/elf/version-matcher.o out/elf/subprocess.o out/elf/relocatable.o out/elf/passes.o out/elf/output-file.o out/elf/output-chunks.o out/elf/mapfile.o out/elf/main.o out/elf/lto.o out/elf/linker-script.o out/elf/input-sections.o out/elf/input-files.o out/elf/icf.o out/elf/glob.o out/elf/gc-sections.o out/elf/dwarf.o out/elf/cmdline.o out/elf/arch-x86-64.o out/elf/arch-riscv64.o out/elf/arch-i386.o out/elf/arch-arm64.o out/elf/arch-arm32.o out/macho/yaml.o out/macho/tapi.o out/macho/output-file.o out/macho/output-chunks.o out/macho/object-file.o out/macho/mapfile.o out/macho/main.o out/macho/input-sections.o out/macho/dumper.o out/macho/dead-strip.o out/macho/cmdline.o out/macho/arch-x86-64.o out/macho/arch-arm64.o -o mold -pthread -lz -lm -ldl -Wl,-whole-archive out/mimalloc/libmimalloc.a -Wl,-no-whole-archive out/tbb/libs/libtbb.a -Wl,-push-state -Wl,-as-needed -lrt -Wl,-pop-state  -lcrypto -v
Intel(R) oneAPI DPC++/C++ Compiler XXX
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: compiler/linux/bin-llvm
Found candidate GCC installation: /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib/gcc/x86_64-pc-linux-gnu/10.2.0
Selected GCC installation: /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib/gcc/x86_64-pc-linux-gnu/10.2.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -inputs=out/mimalloc/libmimalloc.a -check-section
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -inputs=out/mimalloc/libmimalloc.a -check-section
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -inputs=out/mimalloc/libmimalloc.a -check-section
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -inputs=out/mimalloc/libmimalloc.a -check-section
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -inputs=out/tbb/libs/libtbb.a -check-section
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -inputs=out/tbb/libs/libtbb.a -check-section
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -inputs=out/tbb/libs/libtbb.a -check-section
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -inputs=out/tbb/libs/libtbb.a -check-section
 "/usr/bin/ld" --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o mold /usr/lib/../lib64/crt1.o /usr/lib/../lib64/crti.o /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib/gcc/x86_64-pc-linux-gnu/10.2.0/crtbegin.o -L [...] out/tar.o out/strerror.o out/perf.o out/main.o out/hyperloglog.o out/filepath.o out/demangle.o out/compress.o out/elf/version-matcher.o out/elf/subprocess.o out/elf/relocatable.o out/elf/passes.o out/elf/output-file.o out/elf/output-chunks.o out/elf/mapfile.o out/elf/main.o out/elf/lto.o out/elf/linker-script.o out/elf/input-sections.o out/elf/input-files.o out/elf/icf.o out/elf/glob.o out/elf/gc-sections.o out/elf/dwarf.o out/elf/cmdline.o out/elf/arch-x86-64.o out/elf/arch-riscv64.o out/elf/arch-i386.o out/elf/arch-arm64.o out/elf/arch-arm32.o out/macho/yaml.o out/macho/tapi.o out/macho/output-file.o out/macho/output-chunks.o out/macho/object-file.o out/macho/mapfile.o out/macho/main.o out/macho/input-sections.o out/macho/dumper.o out/macho/dead-strip.o out/macho/cmdline.o out/macho/arch-x86-64.o out/macho/arch-arm64.o -lz -Bstatic -limf -Bdynamic -lm -ldl -whole-archive out/mimalloc/libmimalloc.a -no-whole-archive out/tbb/libs/libtbb.a -push-state -as-needed -lrt -pop-state -lcrypto -Bstatic -lsvml -Bdynamic -Bstatic -lirng -Bdynamic -lstdc++ -Bstatic -limf -Bdynamic -lm -lgcc_s -lgcc -Bstatic -lirc -Bdynamic -ldl -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc -Bstatic -lirc_s -Bdynamic /soft/packaging/spack-builds/linux-opensuse_leap15-x86_64/gcc-10.2.0/gcc-10.2.0-yudlyezca7twgd5o3wkkraur7wdbngdn/lib/gcc/x86_64-pc-linux-gnu/10.2.0/crtend.o /usr/lib/../lib64/crtn.o
/usr/bin/ld: out/tbb/libs/libtbb.a: error adding symbols: archive has no index; run ranlib to add one
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

This one I don't know. Linker is too magic for me. Maybe related to this clang-offload-bundler?

Thanks a lot for this promising project!

Thomas

[rui314]

For the first issue, can you send a PR to https://github.com/microsoft/mimalloc? We just copied source files from that project.

For the second issue, I guess that ipcx enables LTO by default and therefore your .o files aren't in the ELF file format. My theory is that ar couldn't read symbol tables of your .o files and therefore failed to create an index. Fortunately, lld doesn't need a symbol table in an archive, so please try again with make LDFLAGS=-fuse-ld=lld.

[TApplencourt]

Thanks for the quick reply!

For the first issue, can you send a PR to https://github.com/microsoft/mimalloc? We just copied source files from that project.

I opened the PR in minimalloc. https://github.com/microsoft/mimalloc/pull/588

For the second issue, I guess that ipcx enables LTO by default and therefore your .o files aren't in the ELF file format. My theory is that ar couldn't read symbol tables of your .o files and therefore failed to create an index. Fortunately, lld doesn't need a symbol table in an archive, so please try again with make LDFLAGS=-fuse-ld=lld.

It did change some stuff! We got new error messages. Progress I guess :)

icpx out/tar.o out/strerror.o out/perf.o out/main.o out/hyperloglog.o out/filepath.o out/demangle.o out/compress.o out/elf/version-matcher.o out/elf/subprocess.o out/elf/relocatable.o out/elf/passes.o out/elf/output-file.o out/elf/output-chunks.o out/elf/mapfile.o out/elf/main.o out/elf/lto.o out/elf/linker-script.o out/elf/input-sections.o out/elf/input-files.o out/elf/icf.o out/elf/glob.o out/elf/gc-sections.o out/elf/dwarf.o out/elf/cmdline.o out/elf/arch-x86-64.o out/elf/arch-riscv64.o out/elf/arch-i386.o out/elf/arch-arm64.o out/elf/arch-arm32.o out/macho/yaml.o out/macho/tapi.o out/macho/output-file.o out/macho/output-chunks.o out/macho/object-file.o out/macho/mapfile.o out/macho/main.o out/macho/input-sections.o out/macho/dumper.o out/macho/dead-strip.o out/macho/cmdline.o out/macho/arch-x86-64.o out/macho/arch-arm64.o -o mold -pthread -lz -lm -ldl -Wl,-whole-archive out/mimalloc/libmimalloc.a -Wl,-no-whole-archive out/tbb/libs/libtbb.a -Wl,-push-state -Wl,-as-needed -lrt -Wl,-pop-state  -lcrypto -fuse-ld=lld
ld.lld: error: undefined symbol: mold::MappedFile<mold::elf::Context<mold::elf::X86_64> >* mold::elf::find_library<mold::elf::X86_64>(mold::elf::Context<mold::elf::X86_64>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
>>> referenced by relocatable.cc
>>>               out/elf/relocatable.o:(void mold::elf::combine_objects<mold::elf::X86_64>(mold::elf::Context<mold::elf::X86_64>&, std::span<std::basic_string_view<char, std::char_traits<char> >, 18446744073709551615ul>))
>>> referenced by linker-script.cc
>>>               out/elf/linker-script.o:(mold::MappedFile<mold::elf::Context<mold::elf::X86_64> >* mold::elf::resolve_path<mold::elf::X86_64>(mold::elf::Context<mold::elf::X86_64>&, std::basic_string_view<char, std::char_traits<char> >))

ld.lld: error: undefined symbol: mold::MappedFile<mold::elf::Context<mold::elf::I386> >* mold::elf::find_library<mold::elf::I386>(mold::elf::Context<mold::elf::I386>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
>>> referenced by relocatable.cc
>>>               out/elf/relocatable.o:(void mold::elf::combine_objects<mold::elf::I386>(mold::elf::Context<mold::elf::I386>&, std::span<std::basic_string_view<char, std::char_traits<char> >, 18446744073709551615ul>))
>>> referenced by linker-script.cc
>>>               out/elf/linker-script.o:(mold::MappedFile<mold::elf::Context<mold::elf::I386> >* mold::elf::resolve_path<mold::elf::I386>(mold::elf::Context<mold::elf::I386>&, std::basic_string_view<char, std::char_traits<char> >))

ld.lld: error: undefined symbol: mold::MappedFile<mold::elf::Context<mold::elf::ARM64> >* mold::elf::find_library<mold::elf::ARM64>(mold::elf::Context<mold::elf::ARM64>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
>>> referenced by relocatable.cc
>>>               out/elf/relocatable.o:(void mold::elf::combine_objects<mold::elf::ARM64>(mold::elf::Context<mold::elf::ARM64>&, std::span<std::basic_string_view<char, std::char_traits<char> >, 18446744073709551615ul>))
>>> referenced by linker-script.cc
>>>               out/elf/linker-script.o:(mold::MappedFile<mold::elf::Context<mold::elf::ARM64> >* mold::elf::resolve_path<mold::elf::ARM64>(mold::elf::Context<mold::elf::ARM64>&, std::basic_string_view<char, std::char_traits<char> >))

ld.lld: error: undefined symbol: mold::MappedFile<mold::elf::Context<mold::elf::ARM32> >* mold::elf::find_library<mold::elf::ARM32>(mold::elf::Context<mold::elf::ARM32>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
>>> referenced by relocatable.cc
>>>               out/elf/relocatable.o:(void mold::elf::combine_objects<mold::elf::ARM32>(mold::elf::Context<mold::elf::ARM32>&, std::span<std::basic_string_view<char, std::char_traits<char> >, 18446744073709551615ul>))
>>> referenced by linker-script.cc
>>>               out/elf/linker-script.o:(mold::MappedFile<mold::elf::Context<mold::elf::ARM32> >* mold::elf::resolve_path<mold::elf::ARM32>(mold::elf::Context<mold::elf::ARM32>&, std::basic_string_view<char, std::char_traits<char> >))

ld.lld: error: undefined symbol: mold::MappedFile<mold::elf::Context<mold::elf::RISCV64> >* mold::elf::find_library<mold::elf::RISCV64>(mold::elf::Context<mold::elf::RISCV64>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
>>> referenced by relocatable.cc
>>>               out/elf/relocatable.o:(void mold::elf::combine_objects<mold::elf::RISCV64>(mold::elf::Context<mold::elf::RISCV64>&, std::span<std::basic_string_view<char, std::char_traits<char> >, 18446744073709551615ul>))
>>> referenced by linker-script.cc
>>>               out/elf/linker-script.o:(mold::MappedFile<mold::elf::Context<mold::elf::RISCV64> >* mold::elf::resolve_path<mold::elf::RISCV64>(mold::elf::Context<mold::elf::RISCV64>&, std::basic_string_view<char, std::char_traits<char> >))
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Thanks again!

[rui314]

As to the link error, it's odd because MappedFile is defined in a header and doesn't have to be linked from other object file. I guess I need to install icpx to debug it.

[TApplencourt]

icpx should be easy to download. It's not really a super high priority for me, I don't want you to waste to many time on this.

Thanks again!