search

numba / llvmlite

A lightweight LLVM python binding for writing JIT compilers
https://llvmlite.pydata.org/
BSD 2-Clause "Simplified" License
1.92k stars 316 forks source link

numba runtime linking error "libllvmlite.so: undefined symbol" after supposedly successful installation of llvmlite #182

Open mbdevpl opened 8 years ago

mbdevpl commented 8 years ago

Hello!

I'm not sure if this issue belongs strictly with llvmlite, or with numba, but I think there is some problem with llvmlite installation that affects numba runtime.

I'm using Ubuntu 14.04 x64. I've installed llvm 3.7 via deb http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.7 main repo provided at http://apt.llvm.org/ and after that I created a soft link llvm-config in path to llvm-config-3.7 executable.

$ llvm-config --version
3.7.1
$ /usr/lib/llvm-3.7/bin/llvm-config --version
3.7.1

I'm using Python 3.5.1, I've installed latest llvmlite and numba:

$ pip3.5 install numba
...
Successfully installed llvmlite-0.11.0 numba-0.26.0

However, after that, whenever I import numba I get this error: site-packages/llvmlite/binding/libllvmlite.so: undefined symbol: _ZTISt14error_category

e.g. with this simple file:

$ cat numba_tests.py 
import numba

@numba.jit
def spam(eggs, ham):
    return eggs + ham

the output is:

$ python3.5 numba_tests.py
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.5/site-packages/llvmlite/binding/ffi.py", line 40, in <module>
    lib = ctypes.CDLL(os.path.join(_lib_dir, _lib_name))
  File "/home/user/.local/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/user/.local/lib/python3.5/site-packages/llvmlite/binding/libllvmlite.so: undefined symbol: _ZTISt14error_category

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "numba_tests.py", line 1, in <module>
    import numba
  File "/home/user/.local/lib/python3.5/site-packages/numba/__init__.py", line 9, in <module>
    from . import config, errors, runtests, types
  File "/home/user/.local/lib/python3.5/site-packages/numba/config.py", line 10, in <module>
    import llvmlite.binding as ll
  File "/home/user/.local/lib/python3.5/site-packages/llvmlite/binding/__init__.py", line 6, in <module>
    from .dylib import *
  File "/home/user/.local/lib/python3.5/site-packages/llvmlite/binding/dylib.py", line 4, in <module>
    from . import ffi
  File "/home/user/.local/lib/python3.5/site-packages/llvmlite/binding/ffi.py", line 45, in <module>
    lib = ctypes.CDLL(_lib_name)
  File "/home/user/.local/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libllvmlite.so: cannot open shared object file: No such file or directory

Very similar problem was seen on StackOverflow: https://stackoverflow.com/questions/31216142/numba-runtime-linking-error but the solution from there provides no help.

I tried reinstalling llvmlite while setting LLVM_CONFIG envvar explicitly, but the end result is the same. I also have other llvm versions (3.4 and 3.6) installed. Can they cause conflicts? Interestingly, when I set the LLVM_CONFIG to wrong paths, it also installs successfully:

$ LLVM_CONFIG=/usr/lib/llvm-3.6/bin/llvm-config pip3.5 install -U llvmlite
Collecting llvmlite
Installing collected packages: llvmlite
Successfully installed llvmlite-0.11.0
$ pip3.5 uninstall llvmlite
Uninstalling llvmlite-0.11.0:
...
$ LLVM_CONFIG=/usr/lib/llvm-3.4/bin/llvm-config pip3.5 install -U llvmlite
Collecting llvmlite
Installing collected packages: llvmlite
Successfully installed llvmlite-0.11.0

Output of running these:

$ /usr/lib/llvm-3.6/bin/llvm-config --version
3.6.0
$ /usr/lib/llvm-3.4/bin/llvm-config --version
bash: /usr/lib/llvm-3.4/bin/llvm-config: No such file or directory

But needless to say, runtime fails with identical messasge each time.

Any ideas on how can I troubleshoot this problem?

mbdevpl commented 8 years ago

Building from source LLVM_CONFIG=/usr/lib/llvm-3.7/bin/llvm-config python3.5 setup.py build and installation are also successful:

$ LLVM_CONFIG=/usr/lib/llvm-3.7/bin/llvm-config python3.5 setup.py bdist_wheel
...
LVM_CONFIG=/usr/lib/llvm-3.7/bin/llvm-config pip3.5 install dist/llvmlite-0.12.0.dev0+8.g30bcfc0-py3-none-any.whl
Successfully installed llvmlite-0.12.0.dev0+8.g30bcfc0

Even if 2 of the tests fail:

$ python3.5 -m llvmlite.tests
EE....
======================================================================
ERROR: llvmlite.tests.test_binding (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: llvmlite.tests.test_binding
Traceback (most recent call last):
  File "/home/user/Projects/python/llvmlite/llvmlite/binding/ffi.py", line 40, in <module>
    lib = ctypes.CDLL(os.path.join(_lib_dir, _lib_name))
  File "/home/user/.local/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/mb/Projects/python/llvmlite/llvmlite/binding/libllvmlite.so: undefined symbol: _ZTISt14error_category

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/user/.local/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/user/Projects/python/llvmlite/llvmlite/tests/test_binding.py", line 16, in <module>
    from llvmlite import binding as llvm
  File "/home/user/Projects/python/llvmlite/llvmlite/binding/__init__.py", line 6, in <module>
    from .dylib import *
  File "/home/user/Projects/python/llvmlite/llvmlite/binding/dylib.py", line 4, in <module>
    from . import ffi
  File "/home/user/Projects/python/llvmlite/llvmlite/binding/ffi.py", line 45, in <module>
    lib = ctypes.CDLL(_lib_name)
  File "/home/user/.local/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libllvmlite.so: cannot open shared object file: No such file or directory

======================================================================
ERROR: llvmlite.tests.test_ir (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: llvmlite.tests.test_ir
Traceback (most recent call last):
  File "/home/user/Projects/python/llvmlite/llvmlite/binding/ffi.py", line 40, in <module>
    lib = ctypes.CDLL(os.path.join(_lib_dir, _lib_name))
  File "/home/user/.local/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/user/Projects/python/llvmlite/llvmlite/binding/libllvmlite.so: undefined symbol: _ZTISt14error_category

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/user/.local/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/user/Projects/python/llvmlite/llvmlite/tests/test_ir.py", line 17, in <module>
    from llvmlite import binding as llvm
  File "/home/user/Projects/python/llvmlite/llvmlite/binding/__init__.py", line 6, in <module>
    from .dylib import *
  File "/home/user/Projects/python/llvmlite/llvmlite/binding/dylib.py", line 4, in <module>
    from . import ffi
  File "/home/user/Projects/python/llvmlite/llvmlite/binding/ffi.py", line 45, in <module>
    lib = ctypes.CDLL(_lib_name)
  File "/home/user/.local/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libllvmlite.so: cannot open shared object file: No such file or directory

----------------------------------------------------------------------
Ran 6 tests in 0.001s

FAILED (errors=2)
pitrou commented 8 years ago

However, after that, whenever I import numba I get this error: site-packages/llvmlite/binding/libllvmlite.so: undefined symbol: _ZTISt14error_category

This is a symptom of a mismatching libstdc++ version. Perhaps LLVM was built with a different libstdc++... Can you try building llvmlite with the clang from the LLVM 3.7 build? You should be able to do so using the CXX environment variable (see ffi/Makefile.linux for details).

mbdevpl commented 8 years ago

Ah, unfortunately I get a build error /tmp/assembly-baff2f.o: file not recognized: File format not recognized.

Full output:

$ LLVM_CONFIG=/usr/lib/llvm-3.7/bin/llvm-config CXX=/usr/lib/llvm-3.7/bin/clang++ python3.5 setup.py build
(...)
/home/user/.local/bin/python3.5 /home/user/Projects/python/llvmlite/ffi/build.py
LLVM version... 3.7.1

# static-libstdc++ avoids runtime dependencies on a
# particular libstdc++ version.
/usr/lib/llvm-3.7/bin/clang++ -static-libstdc++ -shared -I/usr/lib/llvm-3.7/include -DNDEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -fomit-frame-pointer -std=c++11 -fvisibility-inlines-hidden -fno-exceptions -fPIC -ffunction-sections -fdata-sections -Wcast-qual -fno-rtti -g -flto assembly.cpp bitcode.cpp core.cpp initfini.cpp module.cpp value.cpp executionengine.cpp transforms.cpp passmanagers.cpp targets.cpp dylib.cpp linker.cpp -o libllvmlite.so -L/usr/lib/llvm-3.7/lib -flto -Wl,--exclude-libs=ALL -lLLVMLTO -lLLVMObjCARCOpts -lLLVMLinker -lLLVMBitWriter -lLLVMIRReader -lLLVMBPFCodeGen -lLLVMBPFDesc -lLLVMBPFInfo -lLLVMBPFAsmPrinter -lLLVMAMDGPUCodeGen -lLLVMAMDGPUAsmParser -lLLVMAMDGPUUtils -lLLVMAMDGPUDesc -lLLVMAMDGPUInfo -lLLVMAMDGPUAsmPrinter -lLLVMSystemZDisassembler -lLLVMSystemZCodeGen -lLLVMSystemZAsmParser -lLLVMSystemZDesc -lLLVMSystemZInfo -lLLVMSystemZAsmPrinter -lLLVMHexagonDisassembler -lLLVMHexagonCodeGen -lLLVMHexagonDesc -lLLVMHexagonInfo -lLLVMNVPTXCodeGen -lLLVMNVPTXDesc -lLLVMNVPTXInfo -lLLVMNVPTXAsmPrinter -lLLVMCppBackendCodeGen -lLLVMCppBackendInfo -lLLVMMSP430CodeGen -lLLVMMSP430Desc -lLLVMMSP430Info -lLLVMMSP430AsmPrinter -lLLVMXCoreDisassembler -lLLVMXCoreCodeGen -lLLVMXCoreDesc -lLLVMXCoreInfo -lLLVMXCoreAsmPrinter -lLLVMMipsDisassembler -lLLVMMipsCodeGen -lLLVMMipsAsmParser -lLLVMMipsDesc -lLLVMMipsInfo -lLLVMMipsAsmPrinter -lLLVMAArch64Disassembler -lLLVMAArch64CodeGen -lLLVMAArch64AsmParser -lLLVMAArch64Desc -lLLVMAArch64Info -lLLVMAArch64AsmPrinter -lLLVMAArch64Utils -lLLVMARMDisassembler -lLLVMARMCodeGen -lLLVMARMAsmParser -lLLVMARMDesc -lLLVMARMInfo -lLLVMARMAsmPrinter -lLLVMPowerPCDisassembler -lLLVMPowerPCCodeGen -lLLVMPowerPCAsmParser -lLLVMPowerPCDesc -lLLVMPowerPCInfo -lLLVMPowerPCAsmPrinter -lLLVMSparcDisassembler -lLLVMSparcCodeGen -lLLVMSparcAsmParser -lLLVMSparcDesc -lLLVMSparcInfo -lLLVMSparcAsmPrinter -lLLVMMIRParser -lLLVMAsmParser -lLLVMLibDriver -lLLVMOption -lLLVMDebugInfoPDB -lLLVMTableGen -lLLVMOrcJIT -lLLVMLineEditor -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMMCJIT -lLLVMDebugInfoDWARF -lLLVMPasses -lLLVMipo -lLLVMVectorize -lLLVMInterpreter -lLLVMExecutionEngine -lLLVMRuntimeDyld -lLLVMCodeGen -lLLVMTarget -lLLVMScalarOpts -lLLVMProfileData -lLLVMObject -lLLVMMCParser -lLLVMBitReader -lLLVMInstCombine -lLLVMInstrumentation -lLLVMTransformUtils -lLLVMipa -lLLVMMC -lLLVMAnalysis -lLLVMCore -lLLVMSupport -lz -lpthread -lffi -ledit -ltinfo -ldl -lm
/tmp/assembly-baff2f.o: file not recognized: File format not recognized
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [libllvmlite.so] Error 1
Traceback (most recent call last):
  File "/home/user/Projects/python/llvmlite/ffi/build.py", line 152, in <module>
    main()
  File "/home/user/Projects/python/llvmlite/ffi/build.py", line 142, in main
    main_posix('linux', '.so')
  File "/home/user/Projects/python/llvmlite/ffi/build.py", line 134, in main_posix
    subprocess.check_call(['make', '-f', makefile])
  File "/home/user/.local/lib/python3.5/subprocess.py", line 584, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-f', 'Makefile.linux']' returned non-zero exit status 2
error: command '/home/user/.local/bin/python3.5' failed with exit status 1

Build succeeds when I set CXX to my default g++ 5.3.0 explicitly, and also when I try some older versions of g++ I have, e.g. 4.9. But the runtime error persists.

The missing symbol is slightly different with 4.9, I don't know if it suggests anything:

Does this suggest anything? Is there a problem with clang installation?

pitrou commented 8 years ago

Try removing -flto from the options in the Makefile?

mbdevpl commented 8 years ago

@pitrou Thank you so much for your help so far! After that the build succeeds, but I end up with the same error as in case of g++ 5.3: libllvmlite.so: undefined symbol: _ZTISt14error_category when running the tests.

pitrou commented 8 years ago

Then I can only suggest that you use conda and our pre-built binaries.

mbdevpl commented 8 years ago

Thank you very much for help with the troubleshooting! But I don't think I'm going to try conda because my aim is to use numba as well as many packages that are not available in conda...

I managed to get it working by doing additional change, I got rid of -static-libstdc++ flag. Does this setup have any immediate flaws? Assuming I'll be installing/removing/updating some packages related to libstdc++, I'll have to rebuild llvmlite, right? But without any system changes, runtime performance shouldn't be affected, yes?

Changes that solved this are available on branch issue-182 of my fork of llvmlite, specifically in this commit: https://github.com/numba/llvmlite/commit/bf8a63f6189a531b53420e2cc30c8e85bc6d422e

The issue might get resolved auto-magically when I update my Ubuntu to 16.04.

The more I look the more people I find who have or had similar issue:

pitrou commented 8 years ago

Assuming I'll be installing/removing/updating some packages related to libstdc++, I'll have to rebuild llvmlite, right? But without any system changes, runtime performance shouldn't be affected, yes?

As long as you don't need to redistribute the llvmlite binaries to different Linux distributions or versions, there is no downside I know of. We link libstdc++ statically in our own builds, so that we can build a single binary that will work from CentOS 5 up to the most modern distributions.