search

facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data
https://pytorch3d.org/
Other
8.8k stars 1.32k forks source link

Compiler Error on building pytorch3d._C on Linux with CUDA #1406

Open PeggyYuchunWang opened 1 year ago

PeggyYuchunWang commented 1 year ago

🐛 Bugs / Unexpected behaviors

Compiler error on pytorch3d._C when building pytorch3d from source on linux with CUDA.

Instructions To Reproduce the Issue:

The exact command(s) I ran:

pip install 'git+https://github.com/facebookresearch/pytorch3d.git'

What I observed (including the full logs):

...
copying pytorch3d/datasets/shapenet/shapenet_synset_dict_v2.json -> build/lib.linux-x86_64-cpython-310/pytorch3d/datasets/shapenet
running build_ext
/opt/conda/envs/faceverse/lib/python3.10/site-packages/torch/utils/cpp_extension.py:803: UserWarning: The detected CUDA version (11.3) has a minor version mismatch with the version that was used to compile PyTorch (11.2). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
/opt/conda/envs/faceverse/lib/python3.10/site-packages/torch/utils/cpp_extension.py:810: UserWarning: There are no g++ version bounds defined for CUDA version 11.3
warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'pytorch3d._C' extension
creating /tmp/pip-req-build-2oble90z/build/temp.linux-x86_64-cpython-310
creating /tmp/pip-req-build-2oble90z/build/temp.linux-x86_64-cpython-310/tmp
creating /tmp/pip-req-build-2oble90z/build/temp.linux-x86_64-cpython-310/tmp/pip-req-build-2oble90z
creating /tmp/pip-req-build-2oble90z/build/temp.linux-x86_64-cpython-310/tmp/pip-req-build-2oble90z/pytorch3d
creating /tmp/pip-req-build-2oble90z/build/temp.linux-x86_64-cpython-310/tmp/pip-req-build-2oble90z/pytorch3d/csrc
creating /tmp/pip-req-build-2oble90z/build/temp.linux-x86_64-cpython-310/tmp/pip-req-build-2oble90z/pytorch3d/csrc/ball_query
...
/opt/conda/envs/faceverse/lib/python3.10/site-packages/torch/include/c10/core/SymInt.h(84): warning: integer conversion resulted in a change of sign

      ninja: build stopped: subcommand failed.
      Traceback (most recent call last):
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1798, in _run_ninja_build
          subprocess.run(
        File "/opt/conda/envs/faceverse/lib/python3.10/subprocess.py", line 526, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-req-build-2oble90z/setup.py", line 144, in <module>
          setup(
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
          super().run_command(command)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
          super().run_command(command)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
          super().run_command(command)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
          _build_ext.run(self)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
          self.build_extensions()
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 755, in build_extensions
          build_ext.build_extensions(self)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 468, in build_extensions
          self._build_extensions_serial()
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 494, in _build_extensions_serial
          self.build_extension(ext)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
          _build_ext.build_extension(self, ext)
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 549, in build_extension
          objects = self.compiler.compile(
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 576, in unix_wrap_ninja_compile
          _write_ninja_file_and_compile_objects(
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1477, in _write_ninja_file_and_compile_objects
          _run_ninja_build(
        File "/opt/conda/envs/faceverse/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1814, in _run_ninja_build
          raise RuntimeError(message) from e
      RuntimeError: Error compiling objects for extension
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pytorch3d
  Running setup.py clean for pytorch3d
Failed to build pytorch3d

Full logs here: pastebin

I get a similar error when I run conda install pytorch3d -c pytorch3d. The package seems to install fine, but then when I run import pytorch3d in my code, I get an error:

>>> from pytorch3d import _C 
ImportError: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory

System Info

Version: pytorch-gpu.1-12.m100 Based on: Debian GNU/Linux 10 (buster) (GNU/Linux 4.19.0-23-cloud-amd64 x86_64\n) Google Deep Learning VM

GPU Info

peggy@deeplearning-test-t4-vm:~$ nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   57C    P0    27W /  70W |      0MiB / 15360MiB |     10%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

peggy@deeplearning-test-t4-vm:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0

Conda Environment

Conda environment:
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
alsa-lib                  1.2.8                h166bdaf_0    conda-forge
aom                       3.5.0                h27087fc_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
brotli                    1.0.9                h166bdaf_8    conda-forge
brotli-bin                1.0.9                h166bdaf_8    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
cairo                     1.16.0            ha61ee94_1014    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
coloredlogs               15.0.1             pyhd8ed1ab_3    conda-forge
contourpy                 1.0.6           py310hbf28c38_0    conda-forge
cryptography              38.0.4          py310h600f1e7_0    conda-forge
cuda                      11.6.1                        0    nvidia
cuda-cccl                 11.6.55              hf6102b2_0    nvidia
cuda-command-line-tools   11.6.2                        0    nvidia
cuda-compiler             11.6.2                        0    nvidia
cuda-cudart               11.6.55              he381448_0    nvidia
cuda-cudart-dev           11.6.55              h42ad0f4_0    nvidia
cuda-cuobjdump            11.6.124             h2eeebcb_0    nvidia
cuda-cupti                11.6.124             h86345e5_0    nvidia
cuda-cuxxfilt             11.6.124             hecbf4f6_0    nvidia
cuda-driver-dev           11.6.55                       0    nvidia
cuda-gdb                  12.0.90                       0    nvidia
cuda-libraries            11.6.1                        0    nvidia
cuda-libraries-dev        11.6.1                        0    nvidia
cuda-memcheck             11.8.86                       0    nvidia
cuda-nsight               12.0.78                       0    nvidia
cuda-nsight-compute       12.0.0                        0    nvidia
cuda-nvcc                 11.6.124             hbba6d2d_0    nvidia
cuda-nvdisasm             12.0.76                       0    nvidia
cuda-nvml-dev             11.6.55              haa9ef22_0    nvidia
cuda-nvprof               12.0.90                       0    nvidia
cuda-nvprune              11.6.124             he22ec0a_0    nvidia
cuda-nvrtc                11.6.124             h020bade_0    nvidia
cuda-nvrtc-dev            11.6.124             h249d397_0    nvidia
cuda-nvtx                 11.6.124             h0630a44_0    nvidia
cuda-nvvp                 12.0.90                       0    nvidia
cuda-runtime              11.6.1                        0    nvidia
cuda-samples              11.6.101             h8efea70_0    nvidia
cuda-sanitizer-api        12.0.90                       0    nvidia
cuda-toolkit              11.6.1                        0    nvidia
cuda-tools                11.6.1                        0    nvidia
cuda-visual-tools         11.6.1                        0    nvidia
cudatoolkit               11.6.0              hecad31d_11    conda-forge
cudnn                     8.4.1.50             hed8a83a_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
expat                     2.5.0                h27087fc_0    conda-forge
ffmpeg                    5.1.2           gpl_h8dda1f0_105    conda-forge
fftw                      3.3.10          nompi_hf0379b8_106    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.1               hc2a2eb6_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.38.0          py310h5764c6d_1    conda-forge
freeglut                  3.2.2                h9c3ff4c_1    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
fvcore                    0.1.5.post20221221    pyhd8ed1ab_0    conda-forge
gds-tools                 1.5.0.59                      0    nvidia
gettext                   0.21.1               h27087fc_0    conda-forge
glib                      2.74.1               h6239696_1    conda-forge
glib-tools                2.74.1               h6239696_1    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gmpy2                     2.1.2           py310h3ec546c_1    conda-forge
gnutls                    3.7.8                hf3e180e_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
gst-plugins-base          1.21.3               h4243ec0_1    conda-forge
gstreamer                 1.21.3               h25f0c4b_1    conda-forge
gstreamer-orc             0.4.33               h166bdaf_0    conda-forge
harfbuzz                  6.0.0                h8e241bc_0    conda-forge
hdf5                      1.12.2          nompi_h4df4325_101    conda-forge
humanfriendly             10.0            py310hff52083_4    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
iopath                    0.1.9              pyhd8ed1ab_0    conda-forge
jack                      1.9.21               h583fa2b_2    conda-forge
jasper                    2.0.33               ha77e612_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4           py310hbf28c38_1    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.14                 hfd0df8a_1    conda-forge
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libaec                    1.0.6                h9c3ff4c_0    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
libbrotlidec              1.0.9                h166bdaf_8    conda-forge
libbrotlienc              1.0.9                h166bdaf_8    conda-forge
libcap                    2.66                 ha37c62d_0    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libclang                  15.0.6          default_h2e3cab8_0    conda-forge
libclang13                15.0.6          default_h3a83d3e_0    conda-forge
libcublas                 11.9.2.110           h5e84587_0    nvidia
libcublas-dev             11.9.2.110           h5c901ab_0    nvidia
libcufft                  10.7.1.112           hf425ae0_0    nvidia
libcufft-dev              10.7.1.112           ha5ce4c0_0    nvidia
libcufile                 1.5.0.59                      0    nvidia
libcufile-dev             1.5.0.59                      0    nvidia
libcups                   2.3.3                h36d4200_3    conda-forge
libcurand                 10.3.1.50                     0    nvidia
libcurand-dev             10.3.1.50                     0    nvidia
libcurl                   7.87.0               hdc1c0ab_0    conda-forge
libcusolver               11.3.4.124           h33c3c4e_0    nvidia
libcusparse               11.7.2.124           h7538f96_0    nvidia
libcusparse-dev           11.7.2.124           hbbe9722_0    nvidia
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libdeflate                1.14                 h166bdaf_0    conda-forge
libdrm                    2.4.114              h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h28343ad_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.2                h27087fc_0    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgcrypt                 1.10.1               h166bdaf_0    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libglib                   2.74.1               h606061b_1    conda-forge
libglu                    9.0.0             he1b5a44_1001    conda-forge
libgpg-error              1.45                 hc0c96e0_0    conda-forge
libhwloc                  2.8.0                h32351e8_1    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libidn2                   2.3.4                h166bdaf_0    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
liblapacke                3.9.0           16_linux64_openblas    conda-forge
libllvm15                 15.0.6               h63197d8_0    conda-forge
libnghttp2                1.47.0               hff17c54_1    conda-forge
libnpp                    11.6.3.124           hd2722f0_0    nvidia
libnpp-dev                11.6.3.124           h3c42840_0    nvidia
libnsl                    2.0.0                h7f98852_0    conda-forge
libnvjpeg                 11.6.2.124           hd473ad6_0    nvidia
libnvjpeg-dev             11.6.2.124           hb5906b9_0    nvidia
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libopencv                 4.6.0           py310h5bd1119_9    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpciaccess              0.17                 h166bdaf_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.1                 hb675445_2    conda-forge
libprotobuf               3.21.12              h3eb15da_0    conda-forge
libsndfile                1.1.0                hcb278e6_1    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libssh2                   1.10.0               hf14f497_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libsystemd0               252                  h2a991cd_0    conda-forge
libtasn1                  4.19.0               h166bdaf_0    conda-forge
libtiff                   4.5.0                h82bc61c_0    conda-forge
libtool                   2.4.6             h9c3ff4c_1008    conda-forge
libudev1                  252                  h166bdaf_0    conda-forge
libunistring              0.9.10               h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libva                     2.16.0               h166bdaf_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libvpx                    1.11.0               h9c3ff4c_3    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.10.3               h7463322_0    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
llvm-openmp               15.0.6               he0ac6c6_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
magma                     2.5.4                hc72dce7_4    conda-forge
matplotlib                3.6.2           py310hff52083_0    conda-forge
matplotlib-base           3.6.2           py310h8d5ebf3_0    conda-forge
mkl                       2022.2.1         h84fe81f_16997    conda-forge
mpc                       1.2.1                h9f54685_0    conda-forge
mpfr                      4.1.0                h9202a9a_1    conda-forge
mpg123                    1.31.1               h27087fc_0    conda-forge
mpmath                    1.2.1              pyhd8ed1ab_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.31               h26416b9_0    conda-forge
mysql-libs                8.0.31               hbc51c84_0    conda-forge
nccl                      2.14.3.1             h0800d71_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
nettle                    3.8.1                hc379101_1    conda-forge
ninja                     1.11.0               h924138e_0    conda-forge
nsight-compute            2022.4.0.15                   0    nvidia
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.82                 he02c5a1_0    conda-forge
numpy                     1.24.0          py310h08bbf29_0    conda-forge
nvidiacub                 1.10.0                        0    bottler
onnxruntime               1.13.1          py310h00a7d45_1    conda-forge
opencv                    4.6.0           py310hff52083_9    conda-forge
openh264                  2.3.1                h27087fc_1    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.0.7                h0b41bf4_1    conda-forge
p11-kit                   0.24.1               hc5aa10d_0    conda-forge
packaging                 22.0               pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pillow                    9.2.0           py310h023d228_4    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
plotly                    5.11.0             pyhd8ed1ab_1    conda-forge
ply                       3.11                       py_1    conda-forge
portalocker               2.6.0           py310hff52083_1    conda-forge
protobuf                  4.21.12         py310heca2aa9_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pulseaudio                16.1                 h126f2b6_0    conda-forge
py-opencv                 4.6.0           py310hfdc917e_9    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 22.1.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.7          py310h29803b5_2    conda-forge
pyqt5-sip                 12.11.0         py310hd8f1fbe_2    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.8          h4a9ceb5_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-flatbuffers        22.12.6            pyhd8ed1ab_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pytorch                   1.12.1          cuda112py310he33e0d6_201    conda-forge
pytorch-cuda              11.6                 h867d48c_1    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pyyaml                    6.0             py310h5764c6d_5    conda-forge
qt-main                   5.15.6               hf6cd601_5    conda-forge
re2                       2022.06.01           h27087fc_1    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
setuptools                65.6.3             pyhd8ed1ab_0    conda-forge
sip                       6.7.5           py310hd8f1fbe_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h9b69904_2    conda-forge
svt-av1                   1.4.1                hcb278e6_0    conda-forge
sympy                     1.11.1          py310hff52083_2    conda-forge
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
tbb                       2021.7.0             h924138e_1    conda-forge
tenacity                  8.1.0              pyhd8ed1ab_0    conda-forge
termcolor                 2.1.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
torchaudio                0.12.1              py310_cu116    pytorch
torchvision               0.13.1              py310_cu116    pytorch
tornado                   6.2             py310h5764c6d_1    conda-forge
tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
unicodedata2              15.0.0          py310h5764c6d_0    conda-forge
urllib3                   1.26.13            pyhd8ed1ab_0    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
x264                      1!164.3095           h166bdaf_2    conda-forge
x265                      3.5                  h924138e_3    conda-forge
xcb-util                  0.4.0                h166bdaf_0    conda-forge
xcb-util-image            0.4.0                h166bdaf_0    conda-forge
xcb-util-keysyms          0.4.0                h166bdaf_0    conda-forge
xcb-util-renderutil       0.3.9                h166bdaf_0    conda-forge
xcb-util-wm               0.4.1                h166bdaf_0    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yacs                      0.1.8              pyhd8ed1ab_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge
bottler commented 1 year ago

This looks like #1097 and #1206. What happens if you build without setting CUB_HOME - so relying on CUDA's own CUB?

iyerkrithika21 commented 1 year ago

Uninstalling CUB with: conda uninstall -c bottler nvidiacub solved this problem. I am using CUDA 11.5