search

facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data
https://pytorch3d.org/
Other
8.81k stars 1.32k forks source link

Marching Cubes CUDA implementation generates face indices out of bounds #1751

Closed kaftanski closed 8 months ago

kaftanski commented 8 months ago

🐛 Bugs / Unexpected behaviors

I am getting an index out of bounds in pytorch3d.ops.marching_cubes.py in line: https://github.com/facebookresearch/pytorch3d/blob/a27755db416644273de8d70b770f1d2e68db720b/pytorch3d/ops/marching_cubes.py#L299 The issue occurs only in the CUDA implementation, which generates face-indices much larger than possible (e.g. 1112276992 with only 506 verts in the example below). I am unsure, when this happens, I have not been able to construct a case myself, it occurs rather randomly in my scripts. There are no issues in the CPU-Implementation.

The stacktrace is the following:

/opt/conda/conda-bld/pytorch_1704987394225/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [0,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/core/formatters.py", line 226, in catch_format_error
    r = method(self, *args, **kwargs)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/core/formatters.py", line 711, in __call__
    printer.pretty(obj)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 394, in pretty
    return self.type_pprinters[cls](obj, self, cycle)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 641, in inner
    p.pretty(x)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 394, in pretty
    return self.type_pprinters[cls](obj, self, cycle)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 641, in inner
    p.pretty(x)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 411, in pretty
    return _repr_pprint(obj, self, cycle)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/IPython/lib/pretty.py", line 779, in _repr_pprint
    output = repr(obj)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/torch/_tensor.py", line 461, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/torch/_tensor_str.py", line 677, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/torch/_tensor_str.py", line 597, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/torch/_tensor_str.py", line 349, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/home/users/kaftan/miniconda3/envs/fissure2/lib/python3.10/site-packages/torch/_tensor_str.py", line 137, in __init__
    nonzero_finite_vals = torch.masked_select(
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Instructions To Reproduce the Issue:

Failure case volume: vol.pth.zip

Code to reproduce:

import torch
from pytorch3d.ops.marching_cubes import marching_cubes
vol = torch.load('vol.pth')
verts, faces = marching_cubes(vol, isolevel=0)  # works fine
verts, faces = marching_cubes(vol.cuda(), isolevel=0)  # index out of bounds

My environment (conda list). I installed pytorch3d via pip install "git+https://github.com/facebookresearch/pytorch3d.git".

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
addict                    2.4.0                    pypi_0    pypi
argparse                  1.4.0                    pypi_0    pypi
asttokens                 2.4.1                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
batchgenerators           0.25                     pypi_0    pypi
blas                      1.0                         mkl  
blinker                   1.7.0                    pypi_0    pypi
blosc                     1.21.3               h6a678d5_0  
bottleneck                1.3.7           py310ha9d4c09_0  
brotli                    1.0.9                h5eee18b_7  
brotli-bin                1.0.9                h5eee18b_7  
brotli-python             1.0.9           py310h6a678d5_7  
brunsli                   0.1                  h2531618_0  
bzip2                     1.0.8                h5eee18b_5  
c-ares                    1.19.1               h5eee18b_0  
ca-certificates           2024.2.2             hbcca054_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cfitsio                   3.470                h5893167_7  
charls                    2.2.0                h2531618_0  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.1.7           py310h06a4308_0  
cloudpickle               2.2.1           py310h06a4308_0  
comm                      0.2.1                    pypi_0    pypi
configargparse            1.7                      pypi_0    pypi
contourpy                 1.2.0           py310hdb19cb5_0  
cuda-cudart               12.1.105                      0    nvidia
cuda-cupti                12.1.105                      0    nvidia
cuda-libraries            12.1.0                        0    nvidia
cuda-nvrtc                12.1.105                      0    nvidia
cuda-nvtx                 12.1.105                      0    nvidia
cuda-opencl               12.3.101                      0    nvidia
cuda-runtime              12.1.0                        0    nvidia
cycler                    0.11.0             pyhd3eb1b0_0  
cyrus-sasl                2.1.28               h52b45da_1  
cytoolz                   0.12.2          py310h5eee18b_0  
dash                      2.16.0                   pypi_0    pypi
dash-core-components      2.0.0                    pypi_0    pypi
dash-html-components      2.0.0                    pypi_0    pypi
dash-table                5.0.0                    pypi_0    pypi
dask-core                 2023.11.0       py310h06a4308_0  
dbus                      1.13.18              hb2f20db_0  
decorator                 5.1.1                    pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
executing                 2.0.1                    pypi_0    pypi
expat                     2.5.0                h6a678d5_0  
fastjsonschema            2.19.1                   pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
filelock                  3.13.1          py310h06a4308_0  
flask                     3.0.2                    pypi_0    pypi
fontconfig                2.14.1               h4c34cd2_2  
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.12.1               h4a9f257_0  
fsspec                    2023.10.0       py310h06a4308_0  
future                    1.0.0                    pypi_0    pypi
fvcore                    0.1.5.post20221221          pypi_0    pypi
giflib                    5.2.1                h5eee18b_3  
glib                      2.78.4               h6a678d5_0  
glib-tools                2.78.4               h6a678d5_0  
gmp                       6.2.1                h295c915_3  
gmpy2                     2.1.2           py310heeb90bb_0  
gnutls                    3.6.15               he1e5248_0  
gst-plugins-base          1.14.1               h6a678d5_1  
gstreamer                 1.14.1               h5eee18b_1  
icu                       73.1                 h6a678d5_0  
idna                      3.4             py310h06a4308_0  
igl                       2.2.1           py310h784662a_2    conda-forge
imagecodecs               2021.8.26       py310hecf7e94_1  
imageio                   2.33.1          py310h06a4308_0  
importlib-metadata        7.0.1           py310h06a4308_0  
intel-openmp              2023.1.0         hdb19cb5_46306  
iopath                    0.1.10                   pypi_0    pypi
ipython                   8.22.2                   pypi_0    pypi
ipywidgets                8.1.2                    pypi_0    pypi
itsdangerous              2.1.2                    pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
jinja2                    3.1.3           py310h06a4308_0  
joblib                    1.2.0           py310h06a4308_0  
jpeg                      9e                   h5eee18b_1  
jsonschema                4.21.1                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
jupyter-core              5.7.1                    pypi_0    pypi
jupyterlab-widgets        3.0.10                   pypi_0    pypi
jxrlib                    1.1                  h7b6447c_2  
kiwisolver                1.4.4           py310h6a678d5_0  
krb5                      1.20.1               h143b758_1  
lame                      3.100                h7b6447c_0  
lazy_loader               0.3             py310h06a4308_0  
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.38                 h1181459_1  
lerc                      3.0                  h295c915_0  
libaec                    1.0.4                he6710b0_1  
libblas                   3.9.0           1_h86c2bf4_netlib    conda-forge
libbrotlicommon           1.0.9                h5eee18b_7  
libbrotlidec              1.0.9                h5eee18b_7  
libbrotlienc              1.0.9                h5eee18b_7  
libcblas                  3.9.0           5_h92ddd45_netlib    conda-forge
libclang                  14.0.6          default_hc6dbbc7_1  
libclang13                14.0.6          default_he11475f_1  
libcublas                 12.1.0.26                     0    nvidia
libcufft                  11.0.2.4                      0    nvidia
libcufile                 1.8.1.2                       0    nvidia
libcups                   2.4.2                h2d74bed_1  
libcurand                 10.3.4.107                    0    nvidia
libcurl                   8.5.0                h251f7ec_0  
libcusolver               11.4.4.55                     0    nvidia
libcusparse               12.0.2.55                     0    nvidia
libdeflate                1.8                  h7f8727e_5  
libedit                   3.1.20230828         h5eee18b_0  
libev                     4.33                 h7f8727e_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgfortran-ng            11.2.0               h00389a5_1  
libgfortran5              11.2.0               h1234567_1  
libglib                   2.78.4               hdc74915_0  
libgomp                   13.2.0               h807b86a_5    conda-forge
libiconv                  1.16                 h7f8727e_2  
libidn2                   2.3.4                h5eee18b_0  
libjpeg-turbo             2.0.0                h9bf148f_0    pytorch
libllvm14                 14.0.6               hdb19cb5_3  
libnghttp2                1.57.0               h2d74bed_0  
libnpp                    12.0.2.50                     0    nvidia
libnvjitlink              12.1.105                      0    nvidia
libnvjpeg                 12.1.1.14                     0    nvidia
libpng                    1.6.39               h5eee18b_0  
libpq                     12.17                hdbd6064_0  
libssh2                   1.10.0               hdbd6064_2  
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libtasn1                  4.19.0               h5eee18b_0  
libtiff                   4.4.0                hecacb30_2  
libunistring              0.9.10               h27cfd23_0  
libuuid                   1.41.5               h5eee18b_0  
libwebp                   1.2.4                h11a3e52_1  
libwebp-base              1.2.4                h5eee18b_1  
libxcb                    1.15                 h7f8727e_0  
libxkbcommon              1.0.1                h5eee18b_1  
libxml2                   2.10.4               hf1b16e4_1  
libzopfli                 1.0.3                he6710b0_0  
linecache2                1.0.0                    pypi_0    pypi
llvm-openmp               14.0.6               h9e868ea_0  
locket                    1.0.0           py310h06a4308_0  
lz4-c                     1.9.4                h6a678d5_0  
markupsafe                2.1.3           py310h5eee18b_0  
matplotlib                3.8.0           py310h06a4308_0  
matplotlib-base           3.8.0           py310h1128e8f_0  
matplotlib-inline         0.1.6                    pypi_0    pypi
mkl                       2023.1.0         h213fc3f_46344  
mkl-service               2.4.0           py310h5eee18b_1  
mkl_fft                   1.3.8           py310h5eee18b_0  
mkl_random                1.2.4           py310hdb19cb5_0  
mpc                       1.1.0                h10f8cd9_1  
mpfr                      4.0.2                hb69a4c5_1  
mpmath                    1.3.0           py310h06a4308_0  
munkres                   1.1.4                      py_0  
mysql                     5.7.24               h721c034_2  
nbformat                  5.9.2                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
nest-asyncio              1.6.0                    pypi_0    pypi
nettle                    3.7.3                hbbd107a_1  
networkx                  3.1             py310h06a4308_0  
numexpr                   2.8.7           py310h85018f9_0  
numpy                     1.26.4          py310h5f9d8c6_0  
numpy-base                1.26.4          py310hb5e798b_0  
open3d                    0.18.0                   pypi_0    pypi
opencv-python             4.9.0.80                 pypi_0    pypi
openh264                  2.1.1                h4ff587b_0  
openjpeg                  2.4.0                h3ad879b_0  
openssl                   3.2.1                hd590300_0    conda-forge
packaging                 23.1            py310h06a4308_0  
pandas                    2.2.1           py310h6a678d5_0  
parso                     0.8.3                    pypi_0    pypi
partd                     1.4.1           py310h06a4308_0  
pcre2                     10.42                hebb0a14_0  
pexpect                   4.9.0                    pypi_0    pypi
pillow                    9.4.0           py310h6a678d5_0  
pip                       23.3.1          py310h06a4308_0  
platformdirs              4.2.0                    pypi_0    pypi
plotly                    5.19.0                   pypi_0    pypi
ply                       3.11            py310h06a4308_0  
plyfile                   1.0.3                    pypi_0    pypi
portalocker               2.8.2                    pypi_0    pypi
prompt-toolkit            3.0.43                   pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pyamg                     5.0.1                    pypi_0    pypi
pycpd                     2.0.0                    pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pyparsing                 3.0.9           py310h06a4308_0  
pyqt                      5.15.10         py310h6a678d5_0  
pyqt5-sip                 12.13.0         py310h5eee18b_0  
pyquaternion              0.9.9                    pypi_0    pypi
pysocks                   1.7.1           py310h06a4308_0  
python                    3.10.13              h955ad1f_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python-tzdata             2023.3             pyhd3eb1b0_0  
python_abi                3.10                    2_cp310    conda-forge
pytorch                   2.2.0           py3.10_cuda12.1_cudnn8.9.2_0    pytorch
pytorch-cuda              12.1                 ha16c6d3_5    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytorch3d                 0.7.6                    pypi_0    pypi
pytz                      2023.3.post1    py310h06a4308_0  
pywavelets                1.5.0           py310ha9d4c09_0  
pyyaml                    6.0.1           py310h5eee18b_0  
qt-main                   5.15.2              h53bd1ea_10  
readline                  8.2                  h5eee18b_0  
referencing               0.33.0                   pypi_0    pypi
requests                  2.31.0          py310h06a4308_1  
retrying                  1.3.4                    pypi_0    pypi
rpds-py                   0.18.0                   pypi_0    pypi
scikit-image              0.20.0          py310h6a678d5_0  
scikit-learn              1.3.0           py310h1128e8f_1  
scipy                     1.11.4          py310h5f9d8c6_0  
seaborn                   0.12.2          py310h06a4308_0  
setuptools                68.2.2          py310h06a4308_0  
simpleitk                 2.2.1           py310h3fd9d12_0    simpleitk
sip                       6.7.12          py310h6a678d5_0  
six                       1.16.0             pyhd3eb1b0_1  
snappy                    1.1.10               h6a678d5_1  
sqlite                    3.41.2               h5eee18b_0  
stack-data                0.6.3                    pypi_0    pypi
sympy                     1.12            py310h06a4308_0  
tabulate                  0.9.0                    pypi_0    pypi
tbb                       2021.8.0             hdb19cb5_0  
tenacity                  8.2.3                    pypi_0    pypi
termcolor                 2.4.0                    pypi_0    pypi
thop                      0.1.1-2209072238          pypi_0    pypi
threadpoolctl             2.2.0              pyh0d69192_0  
tifffile                  2021.7.2           pyhd3eb1b0_2  
tk                        8.6.12               h1ccaba5_0  
tomli                     2.0.1           py310h06a4308_0  
toolz                     0.12.0          py310h06a4308_0  
torchaudio                2.2.0               py310_cu121    pytorch
torchtriton               2.2.0                     py310    pytorch
torchvision               0.17.0              py310_cu121    pytorch
tornado                   6.3.3           py310h5eee18b_0  
tqdm                      4.66.2                   pypi_0    pypi
traceback2                1.4.0                    pypi_0    pypi
traitlets                 5.14.1                   pypi_0    pypi
trimesh                   4.1.7              pyhd8ed1ab_0    conda-forge
typing_extensions         4.9.0           py310h06a4308_1  
tzdata                    2024a                h04d1e81_0  
unittest2                 1.1.0                    pypi_0    pypi
urllib3                   2.1.0           py310h06a4308_1  
wcwidth                   0.2.13                   pypi_0    pypi
werkzeug                  3.0.1                    pypi_0    pypi
wheel                     0.41.2          py310h06a4308_0  
widgetsnbextension        4.0.10                   pypi_0    pypi
xz                        5.4.6                h5eee18b_0  
yacs                      0.1.8                    pypi_0    pypi
yaml                      0.2.5                h7b6447c_0  
zfp                       0.5.5                h295c915_6  
zipp                      3.17.0          py310h06a4308_0  
zlib                      1.2.13               h5eee18b_0  
zstd                      1.5.5                hc292b87_0
JaapSuter commented 8 months ago

I believe this to be fixed by 7566530669203769783c94024c25a39e1744e4ed, I ran your repro locally and it succeeded. Do let me know if it's still broken for you. Thanks!

kaftanski commented 8 months ago

I don't have any more issues after re-installing pytorch3d from master with your fix. Thank you so much for the very fast help!