[Java][Gandiva][ARM v8] Exception thrown when JAVA calls libgandiva_jni.so on ARM (but successfully on x86 platform)

[zhaonaiy]

Describe the bug, including details regarding any error messages, version, and platform.

Hello, We have encountered a wierd issue about libgandiva_jni.so. Same Java program calls Arrow 9.0.0's libgandiva_jni.so (compiled successfully) with different results on ARM and x86 platform. On ARM platform, it throwed out an exception as below,

undefined symbol: _ZN4llvm3sys14DynamicLibrary19getPermanentLibraryEPKcPSs

Then we traced this problem, and found it was blocked in the method:

llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::basic_string<char, std::char_traits<char>, std::allocator<char> >)

Furthermore, we have found this method lies in "libLLVMSupport.a", but we are sure this libLLVMSupport.a file has been linked in shared lib. Below is the command we used:

_/usr/bin/c++ -fPIC  -Wno-noexcept-type  -fdiagnostics-color=always -O3 -DNDEBUG  -Wall -fno-semantic-interposition -march=armv8-a  -O3 -DNDEBUG -Wl,--version-script=/path/to/arrow-9.0.0/cpp/src/gandiva/jni/symbols.map -shared -Wl,-soname,libgandiva_jni.so.900 -o ../../../release/libgandiva_jni.so.900.0.0 CMakeFiles/gandiva_jni_objlib.dir/config_builder.cc.o CMakeFiles/gandiva_jni_objlib.dir/config_holder.cc.o CMakeFiles/gandiva_jni_objlib.dir/expression_registry_helper.cc.o CMakeFiles/gandiva_jni_objlib.dir/jni_common.cc.o CMakeFiles/gandiva_jni_objlib.dir/Types.pb.cc.o  ../../../protobuf_ep-install/lib/libprotobuf.a ../../../release/libgandiva.a ../../../release/libarrow.a ../../../protobuf_ep-install/lib/libprotobuf.a ../../../brotli_ep/src/brotli_ep-install/lib/libbrotlienc-static.a ../../../brotli_ep/src/brotli_ep-install/lib/libbrotlidec-static.a ../../../brotli_ep/src/brotli_ep-install/lib/libbrotlicommon-static.a ../../../bzip2_ep-install/lib/libbz2.a ../../../lz4_ep-prefix/src/lz4_ep/lib/liblz4.a ../../../snappy_ep/src/snappy_ep-install/lib/libsnappy.a ../../../zstd_ep-install/lib64/libzstd.a ../../../orc_ep-install/lib/liborc.a ../../../jemalloc_ep-prefix/src/jemalloc_ep/dist//lib/libjemalloc_pic.a -pthread /usr/local/llvm/lib/libLLVMMCJIT.a /usr/local/llvm/lib/libLLVMExecutionEngine.a /usr/local/llvm/lib/libLLVMOrcTargetProcess.a /usr/local/llvm/lib/libLLVMOrcShared.a /usr/local/llvm/lib/libLLVMRuntimeDyld.a /usr/local/llvm/lib/libLLVMipo.a /usr/local/llvm/lib/libLLVMFrontendOpenMP.a /usr/local/llvm/lib/libLLVMIRReader.a /usr/local/llvm/lib/libLLVMAsmParser.a /usr/local/llvm/lib/libLLVMVectorize.a /usr/local/llvm/lib/libLLVMInstrumentation.a /usr/local/llvm/lib/libLLVMLinker.a /usr/local/llvm/lib/libLLVMAArch64CodeGen.a /usr/local/llvm/lib/libLLVMAsmPrinter.a /usr/local/llvm/lib/libLLVMDebugInfoMSF.a /usr/local/llvm/lib/libLLVMGlobalISel.a /usr/local/llvm/lib/libLLVMSelectionDAG.a /usr/local/llvm/lib/libLLVMCodeGen.a /usr/local/llvm/lib/libLLVMTarget.a /usr/local/llvm/lib/libLLVMBitWriter.a /usr/local/llvm/lib/libLLVMScalarOpts.a /usr/local/llvm/lib/libLLVMAggressiveInstCombine.a /usr/local/llvm/lib/libLLVMInstCombine.a /usr/local/llvm/lib/libLLVMTransformUtils.a /usr/local/llvm/lib/libLLVMAnalysis.a /usr/local/llvm/lib/libLLVMProfileData.a /usr/local/llvm/lib/libLLVMDebugInfoDWARF.a /usr/local/llvm/lib/libLLVMObject.a /usr/local/llvm/lib/libLLVMBitReader.a /usr/local/llvm/lib/libLLVMTextAPI.a /usr/local/llvm/lib/libLLVMCFGuard.a /usr/local/llvm/lib/libLLVMCore.a /usr/local/llvm/lib/libLLVMRemarks.a /usr/local/llvm/lib/libLLVMBitstreamReader.a /usr/local/llvm/lib/libLLVMAArch64AsmParser.a /usr/local/llvm/lib/libLLVMMCParser.a /usr/local/llvm/lib/libLLVMAArch64Disassembler.a /usr/local/llvm/lib/libLLVMAArch64Desc.a /usr/local/llvm/lib/libLLVMMCDisassembler.a /usr/local/llvm/lib/libLLVMAArch64Info.a /usr/local/llvm/lib/libLLVMMC.a /usr/local/llvm/lib/libLLVMBinaryFormat.a /usr/local/llvm/lib/libLLVMDebugInfoCodeView.a /usr/local/llvm/lib/libLLVMAArch64Utils.a /usr/local/llvm/lib/libLLVMSupport.a /usr/lib64/libz.so -lrt -ldl -lpthread -lm /usr/lib64/libtinfo.so /usr/local/llvm/lib/libLLVMDemangle.a /usr/lib64/libssl.so /usr/lib64/libcrypto.so ../../../re2_ep-install/lib/libre2.a ../../../utf8proc_ep-install/lib/libutf8proc.a__

Environment details: OS: CentOS v7.6 Tools: GCC 8.3.1, LLVM 14

If I missed anything, please let me know.

Thanks,

NaiYan.

Component(s)

C++ - Gandiva

[zhaonaiy]

By the way, the command we used for Arrow 9 compilation on ARM platform is as follows, cmake .. -DARROW_GANDIVA_JAVA=ON -DARROW_GANDIVA=ON -DARROW_WITH_RE2=ON -DARROW_WITH_UTF8PROC=ON

Thanks.

[zhaonaiy]

On arrow 16.1 (libgandiva_jni.so is NOT compiled manually, but copied from https://repository.apache.org/#nexus-search;quick~arrow-gandiva) , we did the same thing, but it throwed out a different error: "GandivaException: Failed to make LLVM module due to Could not create LLJIT instance: Symbols not found: [ llvm_orc_registerEHFrameSectionWrapper ]"

Hope this helpful. Thanks!

[kou]

39695 may be related.

[zhaonaiy]

@kou ,We have tried and double checked, our issue is different from 39695. Please advise, Thanks a lot!

[kou]

What did you try?

39695 is for https://github.com/apache/arrow/issues/41935#issuecomment-2145328656 .

[kou]

we have found this method lies in "libLLVMSupport.a"

How did you find it?

Could you show the output of nm /usr/local/llvm/lib/libLLVMSupport.a | grep _ZN4llvm3sys14DynamicLibrary19getPermanentLibraryEPKcPSs?

[zhaonaiy]

@kou Below is what we have tried,

1. Find _ZN4llvm3sys14DynamicLibrary19getPermanentLibraryEPKcPSs from libgandiva_jni.so

   $ nm --undefined-only --dynamic release/libgandiva_jni.so.900.0.0 |grep ZN4llvm3sys14DynamicLibrary19getPermanentLibraryEPKcPSs
                      U _ZN4llvm3sys14DynamicLibrary19getPermanentLibraryEPKcPSs

2. Find _ZN4llvm3sys14DynamicLibrary19getPermanentLibraryEPKcPSs from libLLVMSupport.a

   $ nm /usr/local/llvm/lib/libLLVMSupport.a |grep _ZN4llvm3sys14DynamicLibrary19getPermanentLibraryEPKcPSs

   But nothing returned.

3. Find llvm::sys::DynamicLibrary::getPermanentLibrary from libgandiva_jni.so

   $ nm --undefined-only --dynamic release/libgandiva_jni.so.900.0.0 |c++filt |grep "llvm::sys::DynamicLibrary::getPermanentLibrary"
                   U llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)

4. Find llvm::sys::DynamicLibrary::getPermanentLibrary from libLLVMSupport.a

   $ nm /usr/local/llvm/lib/libLLVMSupport.a|c++filt|grep "llvm::sys::DynamicLibrary::getPermanentLibrary"
                   U llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)
   0000000000000000 T llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)

From above, llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*) CAN'T be found in libgandiva_jni.so, however llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) has been found. If they are same?

Wait for your advice. Thanks!

[kou]

Could you also try https://github.com/apache/arrow/issues/41935#issuecomment-2145328656 and https://github.com/apache/arrow/issues/39695#issuecomment-1912186637 (LD_PRELOAD)?