pip install error with CUDA 12 and PyTorch

[RuiqiaoHe]

I tried to install torchsort in my linux environment with: Python 3.8.10 CUDA V12.0.140 PyTorch 1.13.0

After pip install, the error is:

  File "setup.py", line 52, in <module>
    setup(
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
    self.run_command(cmd)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/command/install.py", line 74, in run
    self.do_egg_install()
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/command/install.py", line 123, in do_egg_install
    self.run_command('bdist_egg')
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 165, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 151, in call_command
    self.run_command(cmdname)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/command/install_lib.py", line 112, in build
    self.run_command('build_ext')
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
    self.build_extensions()
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 499, in build_extensions
    _check_cuda_version(compiler_name, compiler_version)
  File "/A/anaconda2/envs/gpu/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 386, in _check_cuda_version
    raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError: 
The detected CUDA version (12.0) mismatches the version that was used to compile
PyTorch (11.7). Please make sure to use the same CUDA versions.

Could you please help me with this problem? Thank you very much.

[teddykoker]

Hi @RuiqiaoHe,

It looks like you have a mismatch between your CUDA version (12.0) and the version used to compile PyTorch (11.7).

The most recent version of PyTorch only includes up to 11.8, so you may need to downgrade your CUDA to 11.7 or 11.8 and ensure you have the PyTorch installed with the same version. You can find installation instructions here (make sure you select the right CUDA version under "compute platform").

If you must use CUDA 12, I believe you may need to build PyTorch yourself, although it looks like they haven't finalized support for CUDA 12 so you may run into more issues with this route.

[RuiqiaoHe]

Thank you for quick reply. I have downgrade my CUDA to 11.8 with conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit. However, new problem arises:

warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))
      /A/anaconda2/envs/gpu/lib/python3.8/site-packages/torch/utils/cpp_extension.py:387: UserWarning: The detected CUDA version (11.8) has a minor version mismatch with the version that was used to compile PyTorch (11.7). Most likely this shouldn't be a problem.
        warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
      /A/anaconda2/envs/gpu/lib/python3.8/site-packages/torch/utils/cpp_extension.py:397: UserWarning: There are no g++ version bounds defined for CUDA version 11.8
        warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
      building 'torchsort.isotonic_cpu' extension
      creating build/temp.linux-x86_64-cpython-38
      creating build/temp.linux-x86_64-cpython-38/torchsort
      gcc -pthread -B /A/anaconda2/envs/gpu/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/A/software/R-3.4.4/include -fPIC -I/A/anaconda2/envs/gpu/lib/python3.8/site-packages/torch/include -I/A/anaconda2/envs/gpu/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/A/anaconda2/envs/gpu/lib/python3.8/site-packages/torch/include/TH -I/A/anaconda2/envs/gpu/lib/python3.8/site-packages/torch/include/THC -I/A/anaconda2/envs/gpu/include/python3.8 -c torchsort/isotonic_cpu.cpp -o build/temp.linux-x86_64-cpython-38/torchsort/isotonic_cpu.o -fopenmp -ffast-math -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=isotonic_cpu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
      cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
      g++ -pthread -B /A/anaconda2/envs/gpu/compiler_compat -Wl,--sysroot=/ -pthread -shared -B /A/anaconda2/envs/gpu/compiler_compat -L/A/anaconda2/envs/gpu/lib -Wl,-rpath=/A/anaconda2/envs/gpu/lib -Wl,--no-as-needed -Wl,--sysroot=/ -L/A/software/R-3.4.4/lib -I/A/software/R-3.4.4/include build/temp.linux-x86_64-cpython-38/torchsort/isotonic_cpu.o -L/A/anaconda2/envs/gpu/lib/python3.8/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-cpython-38/torchsort/isotonic_cpu.cpython-38-x86_64-linux-gnu.so
      error: command '/A/anaconda2/bin/g++' failed: Exec format error
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> torchsort

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

My g++ is 4.8.5, however it seems that the version mismatch is a warning message. Could you please help me fix it?

[RuiqiaoHe]

I updated my GCC to 6.3.0 and solved this problem. Thanks a lot!