error when linking against LLDEnzymeFlags CMake target

[samuelpmishLLNL]

When linking a library/executable against LLDEnzymeFlags:

target_link_libraries(my_library PUBLIC LLDEnzymeFlags)

The link line is producing an error message (this is from a mac build):

ld64.lld: error: unknown argument '--load-pass-plugin=/path/to/my_library/build/lib/LLDEnzyme-16.dylib'

The path in the error message does point to the right shared library, did the method of loading the plugin change? This is from LLVM/clang 16, built with

    CMAKE_ARGS -DLLVM_TARGETS_TO_BUILD:STRING=host
               -DLLVM_ENABLE_PROJECTS:STRING=clang|clang-tools-extra|lld
               -DLLVM_ENABLE_PLUGINS=ON
               -DCMAKE_BUILD_TYPE=Release
               -DLLVM_ENABLE_PLUGINS=ON
               -DLLVM_ENABLE_ASSERTIONS=ON
               -DLLVM_ENABLE_RUNTIMES:STRING=libcxx|libcxxabi
               -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_SOURCE_DIR}/build
               -DDEFAULT_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

[wsmoses]

That is strange since we do test LLDEnzymeFlags https://github.com/EnzymeAD/Enzyme/blob/main/enzyme/test/test_find_package/CMakeLists.txt

[samuelpmishLLNL]

here's the complete invocation:

/path/to/my_library/build/bin/clang++ -O3 -DNDEBUG -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names -fuse-ld=lld -Wl,-mllvm -Wl,-load=/path/to/my_library/build/lib/LLDEnzyme-16.dylib -Wl,--load-pass-plugin=/path/to/my_library/build/lib/LLDEnzyme-16.dylib CMakeFiles/differentiate_state_update.dir/tests/differentiate_state_update.cpp.o -o differentiate_state_update  lib/libgtest_main.a lib/libgtest.a -v
clang version 16.0.0
Target: arm64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /path/to/my_library/build/bin
 "/path/to/my_library/build/bin/ld64.lld" -demangle -dynamic -arch arm64 -platform_version macos 13.0.0 13.3 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o differentiate_state_update -search_paths_first -headerpad_max_install_names -mllvm -load=/path/to/my_library/build/lib/LLDEnzyme-16.dylib --load-pass-plugin=/path/to/my_library/build/lib/LLDEnzyme-16.dylib CMakeFiles/differentiate_state_update.dir/tests/differentiate_state_update.cpp.o lib/libgtest_main.a lib/libgtest.a -lc++ -lSystem

ld64.lld: error: unknown argument '--load-pass-plugin=/path/to/my_library/build/lib/LLDEnzyme-16.dylib'
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)

[tgymnich]

Is it possible that ld64 does not support LLVM plugins?

[wsmoses]

It appears the answer is no: https://github.com/llvm/llvm-project/blob/b63c6e585d8662b67b38988f0cd38a0bd6e0d38a/lld/ELF/Options.td#L765

cc @jdoerfert

[tgymnich]

ld64 != lld

[tgymnich]

I would try something like: -fuse-ld=lld or -DCMAKE_LINKER=lld

[wsmoses]

Yeah though the actual LLVM linker only has support for plugins inside the ELF subcomponent itself, and not the MachO (macOS) subcomponent, so there is no support for plugins.

Recompiling LLVM won't add support, its a feature that needs to be added to upstream LLVM.

[wsmoses]

The old pass manager pipeline exists across all versions, but the new pass manager only exists for ELF.

[tgymnich]

odd. Thought we were using system ld64. Even more odd that ld64.lld does not support plugins.

[wsmoses]

@samuelpmishLLNL can you make an issue on LLVM proper to have lld support plugins on macOS?

[samuelpmishLLNL]

This is weird, I could have sworn I was using LLDEnzyme on my mac earlier.

edit: ah, maybe I was using the old pass manager previously?

---

I would try something like: -fuse-ld=lld or -DCMAKE_LINKER=lld

-fuse-ld=lld was in the command, and it was using the lld built with LLVM.

@samuelpmishLLNL can you make an issue on LLVM proper to have lld support plugins on macOS?

Sure, I'll go make the issue but I might have to defer to you guys for clarification.

[samuelpmishLLNL]

Follow up: are you guys able to reproduce the LLDEnzyme error message on macs?