ModuleNotFoundError: No module named 'torch_int._CUDA'

[aakejiang]

Error happens as below:

>>> import torch_int
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/jiang/torch-int/torch_int/__init__.py", line 1, in <module>
    from . import nn
  File "/home/xxx/jiang/torch-int/torch_int/nn/__init__.py", line 1, in <module>
    from .linear import W8A16Linear, W8FakeA8Linear
  File "/home/xxx/jiang/torch-int/torch_int/nn/linear.py", line 2, in <module>
    from .._CUDA import (linear_a8_w8_b32_o32,
ModuleNotFoundError: No module named 'torch_int._CUDA'

Where is the _CUDA module?

[han2ni3bal]

@aakejiang Hi, I met the same issue as you did. Have you find the solution for this one?

[aakejiang]

Just install torch-int according to installation guide, except you need install it in the same conda env as SmoothQuant.

[Hukongtao]

Just install torch-int according to installation guide, except you need install it in the same conda env as SmoothQuant.

I install torch-int according to installation guide, but got an error:
Do you know how to fix this?

[Jenny20000129]

Just install torch-int according to installation guide, except you need install it in the same conda env as SmoothQuant.

I have followed installation guide, but still have the same error "ModuleNotFoundError: No module named 'torch_int._CUDA'"

[hychiang-git]

Same issue with torch2.2 + CUDA12.1

    from .._CUDA import (linear_a8_w8_b32_o32,
ModuleNotFoundError: No module named 'torch_int._CUDA'

[hychiang-git]

Hello folks, I found pip install -e . or just pip install . work well. Cheers.

[bethalianovike]

Hi @ken012git, do you successfully install torch_int and run the test? I try to run python setup.py install but encounter some error (conda with torch 2.2.1+cu118) The error message:

...
running build_ext
/home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/utils/cpp_extension.py:425: UserWarning: There are no g++ version bounds defined for CUDA version 11.8
  warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'torch_int._CUDA' extension
creating build/temp.linux-x86_64-cpython-38
creating build/temp.linux-x86_64-cpython-38/torch_int
creating build/temp.linux-x86_64-cpython-38/torch_int/kernels
gcc -pthread -B /home/miniconda3/envs/smoothquant/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Itorch_int/kernels/include -I/home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include -I/home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include/TH -I/home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/miniconda3/envs/smoothquant/include/python3.8 -c torch_int/kernels/bindings.cpp -o build/temp.linux-x86_64-cpython-38/torch_int/kernels/bindings.o -std=c++14 -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=_CUDA -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command-line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from /home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from torch_int/kernels/include/bmm.h:3,
                 from torch_int/kernels/bindings.cpp:1:
/home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include/ATen/ATen.h:4:2: error: #error C++17 or later compatible compiler is required to use ATen.
    4 | #error C++17 or later compatible compiler is required to use ATen.
      |  ^~~~~
In file included from /home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include/ATen/core/Generator.h:12,
                 from /home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include/ATen/CPUGeneratorImpl.h:3,
                 from /home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include/ATen/Context.h:3,
                 from /home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include/ATen/ATen.h:7,
                 from /home/miniconda3/envs/smoothquant/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from torch_int/kernels/include/bmm.h:3,
                 from torch_int/kernels/bindings.cpp:1:
...

When I install torch 1.12.1+cu113, the error message will show when I run python tests/test_linear_modules.py The error message:

$ python tests/test_linear_modules.py
Traceback (most recent call last):
  File "tests/test_linear_modules.py", line 2, in <module>
    from torch_int.nn.linear import W8A8B8O8LinearReLU, W8A8B8O8Linear, W8A8B32O32Linear, W8A8BFP32OFP32Linear
  File "/home/torch-int/torch_int/__init__.py", line 1, in <module>
    from . import nn
  File "/home/torch-int/torch_int/nn/__init__.py", line 1, in <module>
    from .linear import W8A16Linear, W8FakeA8Linear
  File "/home/torch-int/torch_int/nn/linear.py", line 2, in <module>
    from .._CUDA import (linear_a8_w8_b32_o32,
ModuleNotFoundError: No module named 'torch_int._CUDA'

My environment's CUDA version is 11.8 Do we must use CUDA 11.3 to run this code? Thank you!

[bethalianovike]

[solved] Solution of my problem above: edit setup.py

# from this
extra_compile_args={'cxx': ['-std=c++14', '-O3'],
                    'nvcc': ['-O3', '-std=c++14', '-U__CUDA_NO_HALF_OPERATORS__', '-U__CUDA_NO_HALF_CONVERSIONS__', '-U__CUDA_NO_HALF2_OPERATORS__', f'-DCUDA_ARCH={cuda_arch}']},

# to this (edit c++14 to c++17)
extra_compile_args={'cxx': ['-std=c++17', '-O3'],
                    'nvcc': ['-O3', '-std=c++17', '-U__CUDA_NO_HALF_OPERATORS__', '-U__CUDA_NO_HALF_CONVERSIONS__', '-U__CUDA_NO_HALF2_OPERATORS__', f'-DCUDA_ARCH={cuda_arch}']},                            

Then run:

1. setup.py install again
2. pip install -e .
3. python3 tests/test_linear_modules.py result:

   test_w8a8b8o8_linear_relu
   ic| r2: tensor(0.0002)
   test_w8a8b8o8_linear
   ic| r2: tensor(0.0002)
   test_w8a8b32o32_linear_with_scaling
   ic| r2: tensor(0.0002)
   test_w8a8bfp32ofp32_linear
   ic| r2: tensor(0.0001)