tianweiy
commented
1 year ago I don't have much clue. Maybe the image is broken or sth related. I think you need to add a pdb at index 22158 and check this specific image to figure out.
Closed Adam1904 closed 1 year ago
tianweiy
commented
1 year ago I don't have much clue. Maybe the image is broken or sth related. I think you need to add a pdb at index 22158 and check this specific image to figure out.
Thank you for the great work. when I generate the points with another detector with the:
python virtual_gen.py --info_path data/nuScenes/infos_train_10sweeps_withvelo_filter_True.pkl MODEL.WEIGHTS checkpoint.pth
I get this error:
8%|███████▊ | 21981/271572 [00:03<00:25, 9734.81it/s]/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [10,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [11,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [12,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [13,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [14,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [15,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [16,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [17,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [18,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:111: operator(): block: [1,0,0], thread: [19,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. 8%|███████▊ | 22158/271572 [00:04<00:54, 4589.35it/s] Traceback (most recent call last): File "virtual_gen.py", line 314, in <module> main(args) File "virtual_gen.py", line 277, in main res = process_one_frame(info, predictor, data) File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context return func(*args, **kwargs) File "virtual_gen.py", line 227, in process_one_frame pred_label, score, pred_mask = postprocess(result[camera_id]) File "virtual_gen.py", line 197, in postprocess labels = labels[~empty_mask] RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.what could be the problem?