check-enzyme-integration tests failures

[davidedelvento]

From the series better late than never, this is a follow up to my previous ticket #1763 (part of the time it took me to respond was that by the time I was ready, new releases of Enzyme and LLVM came out and I did not want to post "noise" with outdated bug reports, so I had to reinstall things from scratch).

The following is with Enzyme v0.0.103 and LLVM v18.1.1 which I believe are the most current ones.

The summary of the failures is

Failed Tests (21):
  Enzyme :: Integration/ForwardMode/eigen.cpp
  Enzyme :: Integration/ReverseMode/eigensumsq.cpp
  Enzyme :: Integration/ReverseMode/eigensumsqdyn-notmp.cpp
  Enzyme :: Integration/ReverseMode/eigensumsqdyn.cpp
  Enzyme :: Integration/ReverseMode/eigentensor.cpp
  Enzyme :: Integration/ReverseMode/eigentensorfull.cpp
  Enzyme :: Integration/ReverseMode/eigentensorreal.cpp
  Enzyme :: Integration/ReverseMode/simpleeigen-made-part.cpp
  Enzyme :: Integration/ReverseMode/simpleeigen-made.cpp
  Enzyme :: Integration/ReverseMode/simpleeigen.cpp
  Enzyme :: Integration/ReverseMode/simpleeigenstatic-made-odd.cpp
  Enzyme :: Integration/ReverseMode/simpleeigenstatic-made.cpp
  Enzyme :: Integration/ReverseMode/simpleeigenstatic-sum.cpp
  Enzyme :: Integration/ReverseMode/simpleeigenstatic-sumsq.cpp
  Enzyme :: Integration/ReverseMode/simpleeigenstatic-vec.cpp
  Enzyme :: Integration/ReverseMode/simpleeigenstatic.cpp
  Enzyme :: Integration/Sparse/eigen_analysis.cpp
  Enzyme :: Integration/Sparse/ringspring2Dextenddata.cpp
  Enzyme :: Integration/Sparse/ringspring3Dextenddata.cpp
  Enzyme :: Integration/Sparse/ringspring3Dextenddatarestlengthone.cpp
  Enzyme :: Integration/Sparse/ringspring3Drestlengthone.cpp

Testing Time: 34.92s
  Passed           : 107
  Expectedly Failed:   4
  Failed           :  21
FAILED: test/Integration/CMakeFiles/check-enzyme-integration

I am not familiar with ninja, so it's a bit hard for me to gather additional relevant information, but I can tell that many of the failures are segmentation faults, and at least 5 are

Integration/Sparse/ringspring3Drestlengthone.cpp:60:24: error: Enzyme: F: ; Function Attrs: mustprogress noinline nounwind uwtable
define linkonce_odr dso_local void @_Z6hess_fIdESt6vectorI6TripleIT_ESaIS3_EEmPS2_(ptr dead_on_unwind noalias writable sret(%"class.std::vector") align 8 %0, i64 noundef %1, ptr noundef %2) local_unnamed_addr #7 comdat {
  call void @llvm.memset.p0.i64(ptr noundef nonnull align 8 dereferenceable(24) %0, i8 0, i64 24, i1 false)
  %4 = icmp ne i64 %1, 0
  call void @llvm.assume(i1 %4)
  br label %5

At least one is

Enzyme-v0.0.103/enzyme/test/Integration/Sparse/ringspring2Dextenddata.cpp:57:24: error: Enzyme: Negated solution not handled: (ne ({(-13
 + (16 * %1)),+,-1}<nw><%6> /u 16), L=)
   57 | std::vector<Triple<T>> hess_f(size_t N, T* input) {
      |                        ^
clang++: ../Enzyme/FunctionUtils.cpp:7046: llvm::SmallVector<std::pair<llvm::Value*, llvm::Value*>, 1> Constraints::allSolutions(llvm::SCEVExpander&, llvm::Type*, llvm::Instruction*, c
onst ConstraintContext&, llvm::IRBuilder<>&) const: Assertion `0' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.

a few more are

Integration/ReverseMode/simpleeigenstatic-made.cpp:10:10: fatal error: 'eigen3/Eigen/Dense' file not found
   10 | #include <eigen3/Eigen/Dense>
      |          ^~~~~~~~~~~~~~~~~~~~
1 error generated.

[davidedelvento]

Following up on this after our offline conversation.

Unfortunately, ninja insists on using fully-qualified paths in its runs, basically defying all my efforts to inject the module-installed eigen3 library (which would require the use of environment-provided settings). This actually is a larger problem, in that it fails to identify and find llvm-lit and in fact build.ninja per se includes those as \llvm-lit and obviously fails with command not found. I sed that out to simply llvm-lit and that succeed.

Ninja appears to me to not honoring CPATH, and neither the use of a compiler wrapper which injects the appropriate path itself: even if I rebuild Enzyme with -DLLVM_DIR=/path/to/the/compiler/wrapper/, all the invocations use /path/to/the/llvm/binary/clang rather than /path/to/the/compiler/wrapper/clang as the build systems I am more familiar with do. Even trying to hack my way around this issue (like I have above for llvm-lit) does not work for clang itself, even if I do it on the grand scale of something like find . -name *.script | xargs sed -i.bak s/wrong_llvm_path/right_llvm_path/

[davidedelvento]

And here is the long log of those additional failures not related to eigen3 but to Sparse

errors.log

Note: there was no /tmp/lit-tmp-76hk8zk6/ from where to harvest ringspring2Dextenddata-0e81a0.cpp and ringspring2Dextenddata-0e81a0.sh as hinted in the error message. Probably deleted by the overzealous test-running scripts?

[davidedelvento]

@wsmoses making sure you see this