Clarify usage through linking

[kiranshila]

Hey there, I'm trying to work out how Enzyme works through linking and have been trying to compile something like

#include "math.h"
#include <stdio.h>

extern double __enzyme_autodiff(void *, double);

double dlog(double x) { return __enzyme_autodiff((void *)log, x); }

int main() {
  for (double i = 1; i < 5; i++) {
    printf("%f %f\n", log(i), dlog(i));
  }
}

with a CMakeLists.txt of

cmake_minimum_required(VERSION 3.16)
project(enzyme_test LANGUAGES C)
find_package(Enzyme REQUIRED)
add_executable(enzyme_test main.c)
target_link_libraries(enzyme_test LLDEnzymeFlags m)

But I get

ld.lld: error: <unknown>:0:0: in function main i32 (): Enzyme: failed to find fn to differentiate  %1 = tail call double @__enzyme_autodiff(ptr noundef nonnull @log, double noundef 1.000000e+00) #3 - found - ; Function Attrs: mustprogress nofree nounwind willreturn memory(write)
declare double @log(double noundef) #1

clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/enzyme_test.dir/build.make:97: enzyme_test] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/enzyme_test.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

I've tried to search around the docs and mailing list, but have come up short with the right way to do this, any help would be appreciated!

[ZuseZ4]

This slightly alternated example gives enzyme a function rather than intrinsic and works. Is it important for you to be able to differentiate intrinsics directly?

#include "math.h"
#include <stdio.h>

extern double __enzyme_autodiff(void *, double);

double mylog(double x) { return log(x);}

double dlog(double x) { return __enzyme_autodiff((void *)mylog, x); }

int main() {
  for (double i = 1; i < 5; i++) {
    printf("%f %f\n", log(i), dlog(i));
  }
}

[kiranshila]

Huh, interesting. This is more pedagogy for me rather than directly useful (and hopefully finding some blind spots in the docs), I'm helping a friend bring in Enzyme for one of her projects, and I am trying to understand how it fits together. As in, she has a big gravitational wave model as an external library that she wants to differentiate as part of parameter estimation, but was unsure on how that pieces together.

So, do calls to external code always require a wrapper function?

In the case of not an intrinsic, but still through linking, i.e., with

mylib.c

double f(double x) { return 12.0 + 31.4 * x + 11.2 * x * x; }

mylib.h

double f(double x);

main.c

#include "mylib.h"
#include <stdio.h>

extern double __enzyme_autodiff(void *, double);

double fn(double x) { return f(x); }

double dfn(double x) { return __enzyme_autodiff((void *)fn, x); }

int main() {
  for (double i = 1; i < 5; i++) {
    printf("%f %f\n", fn(i), dfn(i));
  }
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.16)
project(enzyme_test LANGUAGES C)
find_package(Enzyme REQUIRED)
add_library(mylib mylib.c)
add_executable(enzyme_test main.c)
target_link_libraries(enzyme_test PUBLIC LLDEnzymeFlags mylib)

I get

ld.lld: error: <unknown>:0:0: in function preprocess_fn double (double): Enzyme: No reverse pass found for f
 at context:   %2 = tail call double @f(double noundef %0) #3

clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/enzyme_test.dir/build.make:98: enzyme_test] Error 1
make[1]: *** [CMakeFiles/Makefile2:111: CMakeFiles/enzyme_test.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

[wsmoses]

The outermost function Enzyme differentiates presently can't be a standard math function, but a defined function (which could just directly return a standard math function).

It's definitely on our list to remove this usage limitation, but it is presently required or sin/log/etc.

This is only a restriction for sin/cos/etc, and is not an issue for external code like in your example.

The issue there seems to be that you're perhaps not using clang/llvm/lto to force the emission of all the LLVM [so we do not see a definition of f to differentiatie].

Can you do a clean and show the log of doing make -v

[kiranshila]

Makes sense. You want the version of make?

$ make -v
GNU Make 4.4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Or here's the verbose log

$ make VERBOSE=1
/nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin/cmake -S/home/kiran/Dropbox/Projects/C/enzyme_test -B/home/kiran/Dropbox/Projects/C/enzyme_test/build --check-build-system CMakeFiles/Makefile.cmake 0
/nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin/cmake -E cmake_progress_start /home/kiran/Dropbox/Projects/C/enzyme_test/build/CMakeFiles /home/kiran/Dropbox/Projects/C/enzyme_test/build//CMakeFiles/progress.marks
make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/kiran/Dropbox/Projects/C/enzyme_test/build'
make  -f CMakeFiles/mylib.dir/build.make CMakeFiles/mylib.dir/depend
make[2]: Entering directory '/home/kiran/Dropbox/Projects/C/enzyme_test/build'
cd /home/kiran/Dropbox/Projects/C/enzyme_test/build && /nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin/cmake -E cmake_depends "Unix Makefiles" /home/kiran/Dropbox/Projects/C/enzyme_test /home/kiran/Dropbox/Projects/C/enzyme_test /home/kiran/Dropbox/Projects/C/enzyme_test/build /home/kiran/Dropbox/Projects/C/enzyme_test/build /home/kiran/Dropbox/Projects/C/enzyme_test/build/CMakeFiles/mylib.dir/DependInfo.cmake "--color="
make[2]: Leaving directory '/home/kiran/Dropbox/Projects/C/enzyme_test/build'
make  -f CMakeFiles/mylib.dir/build.make CMakeFiles/mylib.dir/build
make[2]: Entering directory '/home/kiran/Dropbox/Projects/C/enzyme_test/build'
[ 25%] Building C object CMakeFiles/mylib.dir/mylib.c.o
/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/bin/clang    -MD -MT CMakeFiles/mylib.dir/mylib.c.o -MF CMakeFiles/mylib.dir/mylib.c.o.d -o CMakeFiles/mylib.dir/mylib.c.o -c /home/kiran/Dropbox/Projects/C/enzyme_test/mylib.c
[ 50%] Linking C static library libmylib.a
/nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin/cmake -P CMakeFiles/mylib.dir/cmake_clean_target.cmake
/nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin/cmake -E cmake_link_script CMakeFiles/mylib.dir/link.txt --verbose=1
/nix/store/xhkqzjmzpsrzyjfpvwgpaqb5gvqkhmyv-llvm-17.0.6/bin/llvm-ar qc libmylib.a CMakeFiles/mylib.dir/mylib.c.o
/nix/store/xhkqzjmzpsrzyjfpvwgpaqb5gvqkhmyv-llvm-17.0.6/bin/llvm-ranlib libmylib.a
make[2]: Leaving directory '/home/kiran/Dropbox/Projects/C/enzyme_test/build'
[ 50%] Built target mylib
make  -f CMakeFiles/enzyme_test.dir/build.make CMakeFiles/enzyme_test.dir/depend
make[2]: Entering directory '/home/kiran/Dropbox/Projects/C/enzyme_test/build'
cd /home/kiran/Dropbox/Projects/C/enzyme_test/build && /nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin/cmake -E cmake_depends "Unix Makefiles" /home/kiran/Dropbox/Projects/C/enzyme_test /home/kiran/Dropbox/Projects/C/enzyme_test /home/kiran/Dropbox/Projects/C/enzyme_test/build /home/kiran/Dropbox/Projects/C/enzyme_test/build /home/kiran/Dropbox/Projects/C/enzyme_test/build/CMakeFiles/enzyme_test.dir/DependInfo.cmake "--color="
make[2]: Leaving directory '/home/kiran/Dropbox/Projects/C/enzyme_test/build'
make  -f CMakeFiles/enzyme_test.dir/build.make CMakeFiles/enzyme_test.dir/build
make[2]: Entering directory '/home/kiran/Dropbox/Projects/C/enzyme_test/build'
[ 75%] Building C object CMakeFiles/enzyme_test.dir/main.c.o
/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/bin/clang   -flto -MD -MT CMakeFiles/enzyme_test.dir/main.c.o -MF CMakeFiles/enzyme_test.dir/main.c.o.d -o CMakeFiles/enzyme_test.dir/main.c.o -c /home/kiran/Dropbox/Projects/C/enzyme_test/main.c
[100%] Linking C executable enzyme_test
/nix/store/q1nssraba326p2kp6627hldd2bhg254c-cmake-3.29.2/bin/cmake -E cmake_link_script CMakeFiles/enzyme_test.dir/link.txt --verbose=1
/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/bin/clang -fuse-ld=lld -Wl,-mllvm -Wl,-load=/nix/store/1yf4i293k5bjpyqqg5nrd1c2h9s9aijy-enzyme-0.0.121/lib/LLDEnzyme-17.so -Wl,--load-pass-plugin=/nix/store/1yf4i293k5bjpyqqg5nrd1c2h9s9aijy-enzyme-0.0.121/lib/LLDEnzyme-17.so CMakeFiles/enzyme_test.dir/main.c.o -o enzyme_test  libmylib.a
ld.lld: error: <unknown>:0:0: in function preprocess_fn double (double): Enzyme: No reverse pass found for f
 at context:   %2 = tail call double @f(double noundef %0) #3

clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/enzyme_test.dir/build.make:98: enzyme_test] Error 1
make[2]: Leaving directory '/home/kiran/Dropbox/Projects/C/enzyme_test/build'
make[1]: *** [CMakeFiles/Makefile2:111: CMakeFiles/enzyme_test.dir/all] Error 2
make[1]: Leaving directory '/home/kiran/Dropbox/Projects/C/enzyme_test/build'
make: *** [Makefile:91: all] Error 2

[wsmoses]

yeah this doesn't have lto on:

[ 25%] Building C object CMakeFiles/mylib.dir/mylib.c.o
/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/bin/clang    -MD -MT CMakeFiles/mylib.dir/mylib.c.o -MF CMakeFiles/mylib.dir/mylib.c.o.d -o CMakeFiles/mylib.dir/mylib.c.o -c /home/kiran/Dropbox/Projects/C/enzyme_test/mylib.c

[wsmoses]

not sure if it'll automatically do but what if you target_link_libraries mylib with LLDEnzymeFlags

[kiranshila]

Ok I've added target_link_libraries(mylib LLDEnzymeFlags) and it seems it does LTO the lib now and seems to work!

I'd like to add some documentation or an example on CMake usage through linking like this as I'm sure others have run into this. Any advise on where that should go?

[wsmoses]

That would be great! I think probably here https://github.com/EnzymeAD/www

[kiranshila]

Sorry to come back to this, but I'm still slightly confused how this all works together.

Say I have this large external project with some ancient (non CMake) build system. All I can do is pass in some CFLAGS and LDFLAGS. Is all that's required to support LLDEnzyme is -flto and -fuse-ld=lld? Does asking clang to LTO result in a shared library that contains bitcode that LLDEnzyme can use? I thought this was the case, but I still get the No reverse pass found error when trying to link what I thought contained IR.

[kiranshila]

Things seem to work properly if I link against the .a static artifact from the upstream library, but not the .so.