Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

huawei-noah / Efficient-Computing

Efficient computing methods developed by Huawei Noah's Ark Lab

1.2k stars 210 forks source link

Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. #115

Open Tuerte opened 1 year ago

Tuerte commented 1 year ago

C:\w\b\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:93: block: [82,0,0], thread: [15,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. ERROR in training steps: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

bbqq7788 commented 6 months ago

I also encountered this problem, did you solve it?