dancasas
commented
3 weeks ago Seems like a weird CUDA issue, a few ideas:
(1) are you sure your CUDA installation is correct?
(2) are you perhaps running out of GPU RAM?
(3) the script seems to be loading the texture /home/dudehu/Projects/SMPLitex/dummy_data/uv-textures-inpainted/304e9c4798a8c3967de7c74c24ef2e38_texture_inpaint-005_img2img-000_cfg2.0_09212024-131836.png, have you checked that this file exists?
(4) you seem to be using debugpy -- why? have you tried regular python?
Thesaltedfish-single
great Job！
when I use render_results.py,I got
Exception has occurred: RuntimeError CUDA error: device-side assert triggered Compile with
render.render_textures(INPUT_FOLDER)
RuntimeError: CUDA error: device-side assert triggered
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions. File "/home/dudehu/Projects/SMPLitex/scripts/utils/renderer/pytorch3d_renderer.py", line 272, in render_mesh_textured images = renderer(mesh) File "/home/dudehu/Projects/SMPLitex/scripts/utils/smpl_helpers.py", line 89, in render_360_gif current_im = render_mesh_textured( File "/home/dudehu/Projects/SMPLitex/scripts/render_results.py", line 116, in render_textures render_360_gif(self.device, self.verts, File "/home/dudehu/Projects/SMPLitex/scripts/render_results.py", line 129, inTORCH_USE_CUDA_DSAto enable device-side assertions.And when I 'export CUDA_LAUNCH_BLOCKING=1'
(env-py310-SMPLitex) dudehu@freebeat-alpaca-002:~/Projects/SMPLitex$ cd /home/dudehu/Projects/SMPLitex ; /usr/bin/env /home/dudehu/micromamba/envs/env-py310-SMPLitex/bin/python /home/dudehu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher 46457 -- /home/dudehu/Projects/SMPLitex/scripts/render_results.py Error processing line 1 of /home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/distutils-precedence.pth:
Traceback (most recent call last): File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site.py", line 195, in addpackage exec(line) File "", line 1, in
ModuleNotFoundError: No module named '_distutils_hack'
Remainder of file ignored Error processing line 1 of /home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/distutils-precedence.pth:
Traceback (most recent call last): File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site.py", line 195, in addpackage exec(line) File "", line 1, in
ModuleNotFoundError: No module named '_distutils_hack'
Remainder of file ignored /home/dudehu/Projects/SMPLitex/sample-data/SMPL/models WARNING: You are using a SMPL model, with only 10 shape coefficients.
Processing image /home/dudehu/Projects/SMPLitex/dummy_data/uv-textures-inpainted/304e9c4798a8c3967de7c74c24ef2e38_texture_inpaint-005_img2img-000_cfg2.0_09212024-131836.png /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [0,0,0] Assertion
cli.main()
File "/home/dudehu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
run()
File "/home/dudehu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
runpy.run_path(target, run_name="main")
File "/home/dudehu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
return _run_module_code(code, init_globals, run_name,
File "/home/dudehu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/dudehu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "/home/dudehu/Projects/SMPLitex/scripts/render_results.py", line 129, in
render.render_textures(INPUT_FOLDER)
File "/home/dudehu/Projects/SMPLitex/scripts/render_results.py", line 116, in render_textures
render_360_gif(self.device, self.verts,
File "/home/dudehu/Projects/SMPLitex/scripts/utils/smpl_helpers.py", line 89, in render_360_gif
current_im = render_mesh_textured(
File "/home/dudehu/Projects/SMPLitex/scripts/utils/renderer/pytorch3d_renderer.py", line 272, in render_mesh_textured
images = renderer(mesh)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, kwargs)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(args, kwargs)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/pytorch3d/renderer/mesh/renderer.py", line 63, in forward
fragments = self.rasterizer(meshes_world, kwargs)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(args, kwargs)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/pytorch3d/renderer/mesh/rasterizer.py", line 254, in forward
pix_to_face, zbuf, bary_coords, dists = rasterize_meshes(
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/pytorch3d/renderer/mesh/rasterize_meshes.py", line 147, in rasterize_meshes
face_verts = verts_packed[faces_packed]
RuntimeError: CUDA error: device-side assert triggered
Compile with
-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [1,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [2,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [3,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [4,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [5,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [6,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [7,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [8,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [9,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [10,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [11,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [12,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [13,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [14,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [15,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [16,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [17,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [18,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [19,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [20,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [21,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [22,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [23,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [24,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [25,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [26,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [27,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [28,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [29,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [30,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [76,0,0], thread: [31,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. Traceback (most recent call last): File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/home/dudehu/.vscode/extensions/ms-python.debugpy-2024.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, inTORCH_USE_CUDA_DSAto enable device-side assertions.(env-py310-SMPLitex) dudehu@freebeat-alpaca-002:~/Projects/SMPLitex$ python /home/dudehu/Projects/SMPLitex/scripts/render_results.py Error processing line 1 of /home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/distutils-precedence.pth:
Traceback (most recent call last): File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site.py", line 195, in addpackage exec(line) File "", line 1, in
ModuleNotFoundError: No module named '_distutils_hack'
Remainder of file ignored /home/dudehu/Projects/SMPLitex/sample-data/SMPL/models WARNING: You are using a SMPL model, with only 10 shape coefficients.
Processing image /home/dudehu/Projects/SMPLitex/dummy_data/uv-textures-inpainted/304e9c4798a8c3967de7c74c24ef2e38_texture_inpaint-005_img2img-000_cfg2.0_09212024-131836.png /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [96,0,0] Assertion
render.render_textures(INPUT_FOLDER)
File "/home/dudehu/Projects/SMPLitex/scripts/render_results.py", line 116, in render_textures
render_360_gif(self.device, self.verts,
File "/home/dudehu/Projects/SMPLitex/scripts/utils/smpl_helpers.py", line 89, in render_360_gif
current_im = render_mesh_textured(
File "/home/dudehu/Projects/SMPLitex/scripts/utils/renderer/pytorch3d_renderer.py", line 272, in render_mesh_textured
images = renderer(mesh)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, kwargs)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(args, kwargs)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/pytorch3d/renderer/mesh/renderer.py", line 63, in forward
fragments = self.rasterizer(meshes_world, kwargs)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(args, kwargs)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/pytorch3d/renderer/mesh/rasterizer.py", line 254, in forward
pix_to_face, zbuf, bary_coords, dists = rasterize_meshes(
File "/home/dudehu/micromamba/envs/env-py310-SMPLitex/lib/python3.10/site-packages/pytorch3d/renderer/mesh/rasterize_meshes.py", line 147, in rasterize_meshes
face_verts = verts_packed[faces_packed]
RuntimeError: CUDA error: device-side assert triggered
Compile with
-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [97,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [98,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [99,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [100,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [101,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [102,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [103,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [104,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [105,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [106,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [107,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [108,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [109,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [110,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [111,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [112,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [113,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [114,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [115,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [116,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [117,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [118,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [119,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [120,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [121,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [122,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [123,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [124,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [125,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [126,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [156,0,0], thread: [127,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [0,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [1,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [2,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [3,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [4,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [5,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [6,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [7,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [8,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [9,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [10,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [11,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [12,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [13,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [14,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [15,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [16,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [17,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [18,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [19,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [20,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [21,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [22,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [23,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [24,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [25,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [26,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [27,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [28,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [29,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [30,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. /opt/conda/conda-bld/pytorch_1704987280714/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [163,0,0], thread: [31,0,0] Assertion-sizes[i] <= index && index < sizes[i] && "index out of bounds"failed. Traceback (most recent call last): File "/home/dudehu/Projects/SMPLitex/scripts/render_results.py", line 129, inTORCH_USE_CUDA_DSAto enable device-side assertions.