Open ziw-liu opened 8 months ago
We dynamically link to LLVM. Hence the library is a required dependency for us
However the runtime library is named libllvm14
, which includes the version in the name
Another package could depend on a different LLVM runtime library (for example libllvm15
) and both could be installed without clobbering each other
A little unclear what the other package is that you are using, but maybe it would be worth following up with them in a new issue to switch to the versioned LLVM library
The conflicted 'package' is not a conda package, it's the system's graphics driver, the llvmpipe OpenGL renderer from Mesa. It will just try to use whatever LLVM it can find (it will be conda-installed libllvm14 in this case) to JIT-compile shaders. This is not what an unprivileged user can hope to control, and very difficult to downgrade on a HPC system.
Is there a reason to ship a dynamically linked LLVM? Since Numba's official builds seem to be running fine on all the platforms (including arm, powerpc etc) by bundling LLVM into llvmlite.
Here is the backtrace:
#0 0x0000153a43e09c9f _ZL15AddNodeIDCustomRN4llvm16FoldingSetNodeIDEPKNS_6SDNodeE (libLLVM-14.so)
#1 0x0000153a43e09d54 _ZN4llvm10FoldingSetINS_6SDNodeEE10NodeEqualsEPKNS_14FoldingSetBaseEPNS3_4NodeERKNS_16FoldingSetNodeIDEjRS8_ (libLLVM-14.so)
#2 0x00001539deedc0ee _ZN4llvm14FoldingSetBase19FindNodeOrInsertPosERKNS_16FoldingSetNodeIDERPvRKNS0_14FoldingSetInfoE (libLLVM-15.so)
#3 0x00001539df98c692 _ZN4llvm12SelectionDAG19FindNodeOrInsertPosERKNS_16FoldingSetNodeIDERKNS_5SDLocERPv (libLLVM-15.so)
#4 0x00001539df9bd866 _ZN4llvm12SelectionDAG7getNodeEjRKNS_5SDLocENS_3EVTENS_7SDValueES5_NS_11SDNodeFlagsE (libLLVM-15.so)
#5 0x00001539df96129b _ZN4llvm19SelectionDAGBuilder18visitShuffleVectorERKNS_4UserE (libLLVM-15.so)
#6 0x00001539df986a13 _ZN4llvm19SelectionDAGBuilder5visitERKNS_11InstructionE (libLLVM-15.so)
#7 0x00001539df9f2624 _ZN4llvm16SelectionDAGISel16SelectBasicBlockENS_14ilist_iteratorINS_12ilist_detail12node_optionsINS_11InstructionELb0ELb0EvEELb0ELb1EEES6_Rb (libLLVM-15.so)
#8 0x00001539df9f3b12 _ZN4llvm16SelectionDAGISel20SelectAllBasicBlocksERKNS_8FunctionE (libLLVM-15.so)
#9 0x00001539df9f629a _ZN4llvm16SelectionDAGISel20runOnMachineFunctionERNS_15MachineFunctionE.part.0 (libLLVM-15.so)
#10 0x00001539e2690c93 _ZN12_GLOBAL__N_115X86DAGToDAGISel20runOnMachineFunctionERN4llvm15MachineFunctionE (libLLVM-15.so)
#11 0x00001539df443644 _ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE.part.0 (libLLVM-15.so)
#12 0x00001539df154031 _ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE (libLLVM-15.so)
#13 0x00001539df15416c _ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE (libLLVM-15.so)
#14 0x00001539df154aef _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE (libLLVM-15.so)
#15 0x00001539e11e87e0 _ZN4llvm5MCJIT10emitObjectEPNS_6ModuleE (libLLVM-15.so)
#16 0x00001539e11e8e9e _ZN4llvm5MCJIT21generateCodeForModuleEPNS_6ModuleE (libLLVM-15.so)
#17 0x00001539e11e4b11 _ZN4llvm5MCJIT14finalizeObjectEv (libLLVM-15.so)
#18 0x00001539e116289f LLVMGetPointerToGlobal (libLLVM-15.so)
#19 0x00001539e618eeeb llvmpipe_update_fs (swrast_dri.so)
#20 0x00001539e617f7a8 llvmpipe_update_derived (swrast_dri.so)
#21 0x00001539e615c8d8 llvmpipe_draw_vbo (swrast_dri.so)
#22 0x00001539e5cf33f5 _mesa_draw_arrays.part.11 (swrast_dri.so)
The Mesa driver gets confused between llvm 14 and 15, and segfaults.
Numba requires a specific LLVM to work correctly. Currently for 0.59 this is LLVM 14. Numba and llvmlite are then built differently by distributors:
The Numba and llvmlite form PyPi and from the Numba channel (-c numba
) on anaconda.org have LLVM bundled into llvmlite (static linking).
The Numba and llvmlite from conda-forge and from the defaults
channel on anaconda.org do depend on a libllvm
package (dynamic linking).
These setups are very unlikely to change, so I would recommend using a Numba and llvmlite combo from a distributor such that you can work around the issues you are having.
Alternatively, perhaps there is an environment variable that can be set for llvmpipe
to point that package to a suitable LLVM?
Alternatively, perhaps there is an environment variable that can be set for llvmpipe to point that package to a suitable LLVM?
I cannot find a way to point llvmpipe to a specific LLVM at runtime other than setting a global LD_LIBRARY_PATH
, which will also affect other packages.
For now the best workaround may be using the numba
channel instead.
Alternatively, perhaps there is an environment variable that can be set for llvmpipe to point that package to a suitable LLVM?
I cannot find a way to point llvmpipe to a specific LLVM at runtime other than setting a global
LD_LIBRARY_PATH
, which will also affect other packages.For now the best workaround may be using the
numba
channel instead.
Yes. I hope that works for you, do let us know how it goes!
It may be worth adding an LLVM_CONFIG
feature to llvmpipe
or asking for that feature to be implemented on that project! Shouldn't be too hard of an addition.
Solution to issue cannot be found in the documentation.
Issue
The conda-forge recipe for numba hard-pins libllvm to 14.0, which breaks OpenGL-based software when the llvmpipe renderer is used for mesa on Linux.
https://github.com/conda-forge/llvmlite-feedstock/blob/ca6e60523541905df2482e8f2ff893dce40fede8/recipe/meta.yaml#L32
However, the upstream documentation specifically explains that only llvmlite is used and any LLVM installation will be ignored:
Also, installing from upstream channel will not install libllvm, and it works with llvmpipe:
Installed packages
Environment info