CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Traceback (most recent call last):
File "/home/lecun/PHALP/phalp/trackers/PHALP.py", line 201, in track
detections = self.get_human_features(image_frame, pred_masks, pred_bbox, pred_bbox_pad, pred_scores, frame_name, pred_classes, t_, measurments, gt_tids, gt_annots, extra_data)
File "/home/lecun/PHALP/phalp/trackers/PHALP.py", line 410, in get_human_features
hmar_out = self.HMAR(masked_image_list.cuda(), **extra_args)
File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lecun/PHALP/scripts/camnuvem.py", line 119, in forward
rend_depth = self.neural_renderer(pred_verts,
File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/neural_renderer/renderer.py", line 80, in forward
return self.render_depth(vertices, faces, K, R, t, dist_coeffs, orig_size)
File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/neural_renderer/renderer.py", line 188, in render_depth
images = nr.rasterize_depth(faces, self.image_size, self.anti_aliasing)
File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/neural_renderer/rasterize.py", line 461, in rasterize_depth
return rasterize_rgbad(faces, None, image_size, anti_aliasing, near, far, eps, None, False, False, True)['depth']
File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/neural_renderer/rasterize.py", line 332, in rasterize_rgbad
depth = depth[:, list(reversed(range(depth.shape[1]))), :]
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Erro reproduction:
Download this video from CamNuvem robbery dataset: Exec: python3 scripts/demo.py video.source=354.mp4 video.output_dir=./output
It will raise an error: