search

getkeops / keops

KErnel OPerationS, on CPUs and GPUs, with autodiff and without memory overflows
https://www.kernel-operations.io
MIT License
1.03k stars 65 forks source link

pykeops.test_torch_bindings() results in Fatal Python error in PyThreadState_Get #311

Closed ilias-koulalis closed 1 year ago

ilias-koulalis commented 1 year ago

I installed pykeops using pip, following the steps in the docs.

Current thread 0x00007f84650fe740 (most recent call first): File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/keopscore/binders/nvrtc/Gpu_link_compile.py", line 67 in generate_code File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/keopscore/binders/LinkCompile.py", line 101 in get_dll_and_params File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/keopscore/get_keops_dll.py", line 124 in get_keops_dll_impl File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/keopscore/utils/Cache.py", line 27 in call File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/pykeops/common/keops_io/LoadKeOps.py", line 125 in init File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/pykeops/common/keops_io/LoadKeOps.py", line 18 in init File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/pykeops/common/keops_io/LoadKeOps_nvrtc.py", line 15 in init File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/keopscore/utils/Cache.py", line 68 in call File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/pykeops/torch/generic/generic_red.py", line 77 in forward File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/pykeops/torch/generic/generic_red.py", line 626 in call File "~/miniconda3/envs/torch2/lib/python3.10/site-packages/pykeops/torch/test_install.py", line 21 in test_torch_bindings File "", line 1 in

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special (total: 20) Aborted (core dumped)


* Expected behaviour:
According to [the docs](https://www.kernel-operations.io/keops/python/installation.html#testing-your-installation) the method should return ```pyKeOps with torch bindings is working!```

This is the environment I'm using:

packages in environment at ~/miniconda3/envs/torch2:

#

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_kmp_llvm conda-forge alsa-lib 1.2.8 h166bdaf_0 conda-forge asttokens 2.0.5 pyhd3eb1b0_0 attr 2.5.1 h166bdaf_1 conda-forge backcall 0.2.0 pyhd3eb1b0_0 binutils 2.39 hdd6e379_1 conda-forge binutils_impl_linux-64 2.39 he00db2b_1 conda-forge binutils_linux-64 2.39 h5fc0e48_13 conda-forge blas 2.116 mkl conda-forge blas-devel 3.9.0 16_linux64_mkl conda-forge brotli 1.0.9 h166bdaf_8 conda-forge brotli-bin 1.0.9 h166bdaf_8 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.19.1 hd590300_0 conda-forge c-compiler 1.5.2 h0b41bf4_0 conda-forge ca-certificates 2023.5.7 hbcca054_0 conda-forge certifi 2023.5.7 pyhd8ed1ab_0 conda-forge cftime 1.6.2 py310hde88566_1 conda-forge charset-normalizer 3.1.0 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge comm 0.1.2 py310h06a4308_0 compilers 1.5.2 ha770c72_0 conda-forge contourpy 1.0.7 py310hdf3cbec_0 conda-forge cuda-cudart 11.7.99 0 nvidia cuda-cupti 11.7.101 0 nvidia cuda-libraries 11.7.1 0 nvidia cuda-nvcc-dev_linux-64 12.0.76 ha770c72_0 conda-forge cuda-nvrtc 11.7.99 0 nvidia cuda-nvtx 11.7.91 0 nvidia cuda-runtime 11.7.1 0 nvidia cuda-version 12.0 hffde075_2 conda-forge cudatoolkit-dev 11.7.0 h1de0b5d_6 conda-forge curl 8.1.1 h409715c_0 conda-forge cxx-compiler 1.5.2 hf52228f_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge dbus 1.13.6 h5008d03_3 conda-forge debugpy 1.5.1 py310h295c915_0 decorator 5.1.1 pyhd3eb1b0_0 executing 0.8.3 pyhd3eb1b0_0 expat 2.5.0 hcb278e6_1 conda-forge ffmpeg 4.3 hf484d3e_0 pytorch fftw 3.3.10 nompi_hc118613_107 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.2 h14ed4e7_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.39.4 py310h2372a71_0 conda-forge fortran-compiler 1.5.2 hdb1a99f_0 conda-forge freetype 2.12.1 hca18f0e_1 conda-forge gcc 11.3.0 h02d0930_13 conda-forge gcc_impl_linux-64 11.3.0 hab1b70f_19 conda-forge gcc_linux-64 11.3.0 he6f903b_13 conda-forge gettext 0.21.1 h27087fc_0 conda-forge gfortran 11.3.0 ha859ce3_13 conda-forge gfortran_impl_linux-64 11.3.0 he34c6f7_19 conda-forge gfortran_linux-64 11.3.0 h3c55166_13 conda-forge glib 2.76.3 hfc55251_0 conda-forge glib-tools 2.76.3 hfc55251_0 conda-forge gmp 6.2.1 h58526e2_0 conda-forge gnutls 3.6.13 h85f3911_1 conda-forge gst-plugins-base 1.21.3 h4243ec0_1 conda-forge gstreamer 1.21.3 h25f0c4b_1 conda-forge gstreamer-orc 0.4.33 h166bdaf_0 conda-forge gxx 11.3.0 h02d0930_13 conda-forge gxx_impl_linux-64 11.3.0 hab1b70f_19 conda-forge gxx_linux-64 11.3.0 hc203a17_13 conda-forge hdf4 4.2.15 h9772cbc_5 conda-forge hdf5 1.12.2 nompi_h4df4325_101 conda-forge icu 70.1 h27087fc_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge ipykernel 6.19.2 py310h2f386ee_0 ipython 8.12.0 py310h06a4308_0 jack 1.9.22 h11f4161_0 conda-forge jedi 0.18.1 py310h06a4308_1 joblib 1.2.0 pyhd8ed1ab_0 conda-forge jpeg 9e h0b41bf4_3 conda-forge jupyter_client 8.1.0 py310h06a4308_0 jupyter_core 5.3.0 py310h06a4308_0 keopscore 2.1.2 pypi_0 pypi kernel-headers_linux-64 2.6.32 he073ed8_15 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.4 py310hbf28c38_1 conda-forge krb5 1.20.1 h81ceb04_0 conda-forge lame 3.100 h166bdaf_1003 conda-forge lcms2 2.15 hfd0df8a_0 conda-forge ld_impl_linux-64 2.39 hcc3a1bd_1 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libaec 1.0.6 hcb278e6_1 conda-forge libblas 3.9.0 16_linux64_mkl conda-forge libbrotlicommon 1.0.9 h166bdaf_8 conda-forge libbrotlidec 1.0.9 h166bdaf_8 conda-forge libbrotlienc 1.0.9 h166bdaf_8 conda-forge libcap 2.67 he9d0100_0 conda-forge libcblas 3.9.0 16_linux64_mkl conda-forge libclang 15.0.7 default_h7634d5b_2 conda-forge libclang13 15.0.7 default_h9986a30_2 conda-forge libcublas 11.10.3.66 0 nvidia libcufft 10.7.2.124 h4fbf590_0 nvidia libcufile 1.6.1.9 0 nvidia libcups 2.3.3 h36d4200_3 conda-forge libcurand 10.3.2.106 0 nvidia libcurl 8.1.1 h409715c_0 conda-forge libcusolver 11.4.0.1 0 nvidia libcusparse 11.7.4.91 0 nvidia libdb 6.2.32 h9c3ff4c_0 conda-forge libdeflate 1.17 h0b41bf4_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 h28343ad_4 conda-forge libexpat 2.5.0 hcb278e6_1 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libflac 1.4.2 h27087fc_0 conda-forge libgcc-devel_linux-64 11.3.0 h210ce93_19 conda-forge libgcc-ng 12.2.0 h65d4601_19 conda-forge libgcrypt 1.10.1 h166bdaf_0 conda-forge libgfortran-ng 12.2.0 h69a702a_19 conda-forge libgfortran5 12.2.0 h337968e_19 conda-forge libglib 2.76.3 hebfc3b9_0 conda-forge libgomp 12.2.0 h65d4601_19 conda-forge libgpg-error 1.46 h620e276_0 conda-forge libhwloc 2.9.1 hd6dc26d_0 conda-forge libiconv 1.17 h166bdaf_0 conda-forge liblapack 3.9.0 16_linux64_mkl conda-forge liblapacke 3.9.0 16_linux64_mkl conda-forge libllvm14 14.0.6 hcd5def8_2 conda-forge libllvm15 15.0.7 hadd5161_1 conda-forge libnetcdf 4.9.1 nompi_h34a3ff0_101 conda-forge libnghttp2 1.52.0 h61bc06f_0 conda-forge libnpp 11.7.4.75 0 nvidia libnsl 2.0.0 h7f98852_0 conda-forge libnvjpeg 11.8.0.2 0 nvidia libogg 1.3.4 h7f98852_1 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.39 h753d276_0 conda-forge libpq 15.3 hbcd7760_1 conda-forge libsanitizer 11.3.0 h239ccf8_19 conda-forge libsndfile 1.2.0 hb75c966_0 conda-forge libsodium 1.0.18 h7b6447c_0 libsqlite 3.42.0 h2797004_0 conda-forge libssh2 1.10.0 hf14f497_3 conda-forge libstdcxx-devel_linux-64 11.3.0 h210ce93_19 conda-forge libstdcxx-ng 12.2.0 h46fd767_19 conda-forge libsystemd0 253 h8c4010b_1 conda-forge libtiff 4.5.0 h6adf6a1_2 conda-forge libtool 2.4.7 h27087fc_0 conda-forge libudev1 253 h0b41bf4_1 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp-base 1.3.0 h0b41bf4_0 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxkbcommon 1.5.0 h79f4944_1 conda-forge libxml2 2.10.3 hca2bb57_4 conda-forge libzip 1.9.2 hc929e4a_1 conda-forge libzlib 1.2.13 h166bdaf_4 conda-forge llvm-openmp 16.0.4 h4dfa4b3_0 conda-forge llvmlite 0.40.0 py310h1b8f574_0 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge matplotlib 3.7.1 py310hff52083_0 conda-forge matplotlib-base 3.7.1 py310he60537e_0 conda-forge matplotlib-inline 0.1.6 py310h06a4308_0 mkl 2022.1.0 h84fe81f_915 conda-forge mkl-devel 2022.1.0 ha770c72_916 conda-forge mkl-include 2022.1.0 h84fe81f_915 conda-forge mpg123 1.31.3 hcb278e6_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.32 hf1915f5_2 conda-forge mysql-libs 8.0.32 hca2cd23_2 conda-forge ncurses 6.3 h27087fc_1 conda-forge nest-asyncio 1.5.6 py310h06a4308_0 netcdf4 1.6.3 nompi_py310h0feb132_100 conda-forge nettle 3.6 he412f7d_0 conda-forge nspr 4.35 h27087fc_0 conda-forge nss 3.89 he45b914_0 conda-forge numba 0.57.0 py310h0f6aa51_0 conda-forge numpy 1.24.3 py310ha4c1d20_0 conda-forge openh264 2.1.1 h780b84a_0 conda-forge openjpeg 2.5.0 hfec8fc6_2 conda-forge openssl 3.1.0 hd590300_3 conda-forge packaging 23.1 pyhd8ed1ab_0 conda-forge pandas 2.0.1 py310h7cbd5c2_1 conda-forge parso 0.8.3 pyhd3eb1b0_0 pcre2 10.40 hc3806b6_0 conda-forge pexpect 4.8.0 pyhd3eb1b0_3 pickleshare 0.7.5 pyhd3eb1b0_1003 pillow 9.4.0 py310h023d228_1 conda-forge pip 23.1.2 pyhd8ed1ab_0 conda-forge platformdirs 2.5.2 py310h06a4308_0 ply 3.11 py_1 conda-forge pooch 1.7.0 pyha770c72_3 conda-forge prompt-toolkit 3.0.36 py310h06a4308_0 psutil 5.9.0 py310h5eee18b_0 pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3eb1b0_2 pulseaudio 16.1 hcb278e6_3 conda-forge pulseaudio-client 16.1 h5195f5e_3 conda-forge pulseaudio-daemon 16.1 ha8d29e2_3 conda-forge pure_eval 0.2.2 pyhd3eb1b0_0 pybind11 2.10.4 pypi_0 pypi pygments 2.15.1 py310h06a4308_1 pykeops 2.1.2 pypi_0 pypi pynndescent 0.5.10 pyh1a96a4e_0 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyqt 5.15.7 py310hab646b1_3 conda-forge pyqt5-sip 12.11.0 py310heca2aa9_3 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.10.11 he550d4f_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.10 3_cp310 conda-forge pytorch 1.13.1 py3.10_cuda11.7_cudnn8.5.0_0 pytorch pytorch-cuda 11.7 h778d358_5 pytorch pytorch-mutex 1.0 cuda pytorch pytz 2023.3 pyhd8ed1ab_0 conda-forge pyzmq 25.0.2 py310h6a678d5_0 qt-main 5.15.6 hf6cd601_5 conda-forge readline 8.2 h8228510_1 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge scikit-learn 1.2.2 py310h41b6a48_1 conda-forge scipy 1.10.1 py310ha4c1d20_3 conda-forge setuptools 67.7.2 pyhd8ed1ab_0 conda-forge sip 6.7.9 py310hc6cd4ac_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge stack_data 0.2.0 pyhd3eb1b0_0 sysroot_linux-64 2.12 he073ed8_15 conda-forge tbb 2021.9.0 hf52228f_0 conda-forge threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge torchaudio 0.13.1 py310_cu117 pytorch torchvision 0.14.1 py310_cu117 pytorch tornado 6.3.2 py310h2372a71_0 conda-forge tqdm 4.65.0 pyhd8ed1ab_1 conda-forge traitlets 5.7.1 py310h06a4308_0 typing_extensions 4.6.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge umap-learn 0.5.3 py310hff52083_1 conda-forge unicodedata2 15.0.0 py310h5764c6d_0 conda-forge urllib3 2.0.2 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyhd3eb1b0_0 wheel 0.40.0 pyhd8ed1ab_0 conda-forge xarray 2023.5.0 pyhd8ed1ab_0 conda-forge xcb-util 0.4.0 h166bdaf_0 conda-forge xcb-util-image 0.4.0 h166bdaf_0 conda-forge xcb-util-keysyms 0.4.0 h166bdaf_0 conda-forge xcb-util-renderutil 0.3.9 h166bdaf_0 conda-forge xcb-util-wm 0.4.1 h166bdaf_0 conda-forge xkeyboard-config 2.38 h0b41bf4_0 conda-forge xorg-libxau 1.0.11 hd590300_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xz 5.2.6 h166bdaf_0 conda-forge zeromq 4.3.4 h2531618_0 zlib 1.2.13 h166bdaf_4 conda-forge zstd 1.5.2 h3eb15da_6 conda-forge

ilias-koulalis commented 1 year ago

A mismatch between cuda versions was the cause. fixed

jfkback commented 6 months ago

I had the same issue. I solved it by running:

conda install cudatoolkit -c nvidia/label/cuda-12.1.0

where the cuda version was the same one produced when I ran:

torch.version.cuda

After installing cudatoolkit I reinstalled pykeops via pip and no longer got the error.