ptheywood
commented
3 months ago Running readelf -d /path/to/_pyflamegpu.so for local builds on ubuntu do not appear to depend on versioned libnvrtc-builtins.so explcitly, just libnvrtc.so.12.
The version installed from the distributed wheel however depends on libnvrtc-builtins.so.12.0 explicitly.
Cmake 3.28.3 was used locally, on ubuntu 22.04, while the manylinux2014 container from January 2024 was CMake 3.28.1 and is Cent7 based.
During pyflamepgu linking libnvrtc-builtins is present in the command for both cases, but it is unversioned in both, so the source of the difference is still unclear.
mondus
Originally identified as part of #1191, although nvrtc sonames are now major version only (since 11.3, using
.so.11.2or.so.12.0),libnvrtc-builtins.sois still explicitly versioned, and is dependended upon by the genericlibnvrtc.sowhich we are linking against.In practice, this means that currently you must have the exact version of the CTK installed and available at runtime as was used to build the python wheel, rather than a "compatible" version.
This was not noticed when testing locally, as CMake causes a
RPATH/RUNPATHto be set pointing at the exact location of the shared object, so if installed in the same location it is found even ifLD_LIBRARY_PATHdoes not point to it.A workaround has been added to the google-colab notebooke in https://github.com/FLAMEGPU/FLAMEGPU2-tutorial-python/commit/47cc1c119654c79509934164228f52b4bf59713b, which installs the
nvidia-cuda-nvrtc-cu12==12.0.140python package to bring in the matching version oflibnvrtc-builtins.so, and then explicitly imports it viactypes.CDLLto ensure it is loaded.A robust verison of this fix, would be to add the exact version of
nvidia-cuda-nvrtc-cu12(ornvidia-cuda-nvrtc-cu11) required to our python packages'extra_install_requiresinsetup.py, when the packages are intended to be distributed (i.e. only in our CI, not in all local builds to avoid binary bloat).In
__init__.py, we would then need to ensure that the appropraite .so is loaded at runtime if it is not found implicilty, i.e. try and import it system wide, catch errors and if the error is appropraite explicitly load the exactlibnvrtc.soprovided by the python package.Unfortunately,
nvidia-cuda-nvrtc-cu11is only provided on pypi for11.7.99and11.8.89, so this does not work for our 11.2 wheels...Alternative options include:
libnvrtc. Bigger wheels, requires CMake >= 3.26 and CUDA >= 11.5 to do robustly.libnvrtc.soandlibnvrtc-builtins.soin the wheel (like vis dlls) and ensure that version is used if needed. Bloated wheels, but avoids pip conflicts. Related to #647 (but not exact)nvidia-cuda-nvrtc-cuXXmight still be a good idea).More detailed investigation notes can be found in #1191