search

TencentARC / InstantMesh

InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models
Apache License 2.0
3.03k stars 315 forks source link

RuntimeError: handle_0 INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":15, please report a bug to PyTorch.` #101

Open LeiLei-Ya opened 3 months ago

LeiLei-Ya commented 3 months ago

When I run the demo from the command line I get the following error: Traceback (most recent call last): File "/mnt/f/M.Lei_Study/Project_3D_reconstruct/InstantMesh/run.py", line 216, in <module> mesh_out = model.extract_mesh( File "/mnt/f/M.Lei_Study/Project_3D_reconstruct/InstantMesh/src/models/lrm_mesh.py", line 357, in extract_mesh mesh_v, mesh_f, sdf, deformation, v_deformed, sdf_reg_loss = self.get_geometry_prediction(planes) File "/mnt/f/M.Lei_Study/Project_3D_reconstruct/InstantMesh/src/models/lrm_mesh.py", line 165, in get_geometry_prediction sdf, deformation, sdf_reg_loss, weight = self.get_sdf_deformation_prediction(planes) File "/mnt/f/M.Lei_Study/Project_3D_reconstruct/InstantMesh/src/models/lrm_mesh.py", line 110, in get_sdf_deformation_prediction sdf, deformation, weight = torch.utils.checkpoint.checkpoint( File "/root/anaconda3/envs/instantmesh/lib/python3.10/site-packages/torch/_compile.py", line 24, in inner return torch._dynamo.disable(fn, recursive)(*args, **kwargs) File "/root/anaconda3/envs/instantmesh/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn return fn(*args, **kwargs) File "/root/anaconda3/envs/instantmesh/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner return fn(*args, **kwargs) File "/root/anaconda3/envs/instantmesh/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 458, in checkpoint ret = function(*args, **kwargs) File "/mnt/f/M.Lei_Study/Project_3D_reconstruct/InstantMesh/src/models/renderer/synthesizer_mesh.py", line 132, in get_geometry_prediction sdf, deformation, weight = self.decoder.get_geometry_prediction(sampled_features, flexicubes_indices) File "/mnt/f/M.Lei_Study/Project_3D_reconstruct/InstantMesh/src/models/renderer/synthesizer_mesh.py", line 76, in get_geometry_prediction grid_features = torch.index_select(input=sampled_features, index=flexicubes_indices.reshape(-1), dim=1) RuntimeError: handle_0 INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":15, please report a bug to PyTorch.

When I run the demo from the command line I get the following error. I am running it on wsl and checking the video card memory for no abnormalities.

The hardware is as follows: 2080TI 22G 32G RAM

Running environment is as follows: `Package Version

absl-py 2.1.0 accelerate 0.30.1 aiofiles 23.2.1 aiohttp 3.9.5 aiosignal 1.3.1 altair 5.3.0 annotated-types 0.7.0 antlr4-python3-runtime 4.9.3 anyio 4.4.0 async-timeout 4.0.3 attrs 23.2.0 bitsandbytes 0.43.1 braceexpand 0.1.7 certifi 2024.2.2 charset-normalizer 3.3.2 click 8.1.7 coloredlogs 15.0.1 contourpy 1.2.1 cycler 0.12.1 diffusers 0.20.2 dnspython 2.6.1 einops 0.8.0 email_validator 2.1.1 exceptiongroup 1.2.1 fastapi 0.111.0 fastapi-cli 0.0.4 ffmpy 0.3.2 filelock 3.14.0 flatbuffers 24.3.25 fonttools 4.52.4 frozenlist 1.4.1 fsspec 2024.5.0 gradio 3.41.2 gradio_client 0.5.0 grpcio 1.64.0 h11 0.14.0 httpcore 1.0.5 httptools 0.6.1 httpx 0.27.0 huggingface-hub 0.17.3 humanfriendly 10.0 idna 3.7 imageio 2.34.1 imageio-ffmpeg 0.4.9 importlib_metadata 7.1.0 importlib_resources 6.4.0 Jinja2 3.1.4 jsonschema 4.22.0 jsonschema-specifications 2023.12.1 kiwisolver 1.4.5 lazy_loader 0.4 lightning-utilities 0.11.2 llvmlite 0.42.0 Markdown 3.6 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.9.0 mdurl 0.1.2 mpmath 1.3.0 multidict 6.0.5 networkx 3.3 numba 0.59.1 numpy 1.26.4 nvdiffrast 0.3.1 nvidia-cublas-cu12 12.1.3.1 nvidia-cuda-cupti-cu12 12.1.105 nvidia-cuda-nvrtc-cu12 12.1.105 nvidia-cuda-runtime-cu12 12.1.105 nvidia-cudnn-cu12 8.9.2.26 nvidia-cufft-cu12 11.0.2.54 nvidia-curand-cu12 10.3.2.106 nvidia-cusolver-cu12 11.4.5.107 nvidia-cusparse-cu12 12.1.0.106 nvidia-nccl-cu12 2.18.1 nvidia-nvjitlink-cu12 12.5.40 nvidia-nvtx-cu12 12.1.105 omegaconf 2.3.0 onnxruntime 1.18.0 opencv-python-headless 4.9.0.80 orjson 3.10.3 packaging 24.0 pandas 2.2.2 pillow 10.3.0 pip 24.0 platformdirs 4.2.2 plyfile 1.0.3 pooch 1.8.1 protobuf 5.27.0 psutil 5.9.8 pydantic 2.7.1 pydantic_core 2.18.2 pydub 0.25.1 Pygments 2.18.0 PyMatting 1.1.12 PyMCubes 0.1.4 pyparsing 3.1.2 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 python-multipart 0.0.9 pytorch-lightning 2.1.0 pytz 2024.1 PyYAML 6.0.1 referencing 0.35.1 regex 2024.5.15 rembg 2.0.57 requests 2.32.2 rich 13.7.1 rpds-py 0.18.1 safetensors 0.4.3 scikit-image 0.23.2 scipy 1.13.1 semantic-version 2.10.0 setuptools 69.5.1 shellingham 1.5.4 six 1.16.0 sniffio 1.3.1 starlette 0.37.2 sympy 1.12 tensorboard 2.16.2 tensorboard-data-server 0.7.2 tifffile 2024.5.22 tokenizers 0.14.1 toolz 0.12.1 torch 2.1.0 torchaudio 2.1.0 torchmetrics 1.4.0.post0 torchvision 0.16.0 tqdm 4.66.4 transformers 4.34.1 trimesh 4.4.0 triton 2.1.0 typer 0.12.3 typing_extensions 4.12.0 tzdata 2024.1 ujson 5.10.0 urllib3 2.2.1 uvicorn 0.30.0 uvloop 0.19.0 watchfiles 0.22.0 webdataset 0.2.86 websockets 11.0.3 Werkzeug 3.0.3 wheel 0.43.0 xatlas 0.0.9 xformers 0.0.22.post7 yarl 1.9.4 zipp 3.19.0`

Sangrish-s commented 3 months ago

For reference:https://github.com/pytorch/pytorch/issues/66710

Updating torch might be worth a shot or like reference suggests send all to cuda

hopeliu20160622 commented 2 months ago

Update the torch to 2.30 can fix the issue. I have tested, it worked.