trouble accessing pixel colors of tracked locations

[pknmax]

I am trying to get colors of tracked locations, but when i use more than 250 images, i am getting error. code looks like following:

sequence = np.stack(images)
sequence = torch.from_numpy(sequence).permute(0, 3, 1, 2).float().unsqueeze(0)

queries = torch.tensor([[   0.0000, 1364.1562,  394.9671],
                            [   0.0000, 1274.5839,  339.1161],
                            [   0.0000, 1193.4419,  285.3727],
                            [   0.0000, 1114.4075,  236.8983],
                            [   0.0000, 1038.5344,  192.6390],
                            [   0.0000,  974.2531,  139.9494]])

pred_tracks, pred_visibility = model(sequence, queries=queries.unsqueeze(0))

Accessing colors:

pred_tracks_long = pred_tracks.long()
pred_tracks_long[..., 0] = pred_tracks_long[..., 0].clamp(0, sequence.shape[3] - 1)
pred_tracks_long[..., 1] = pred_tracks_long[..., 1].clamp(0, sequence.shape[4] - 1)

batch_size, seq_len, num_points, _ = pred_tracks_long.shape
frame_indices = torch.arange(seq_len).unsqueeze(0).unsqueeze(2).expand(batch_size, seq_len, num_points).to(pred_tracks_long.device)
batch_indices = torch.arange(batch_size).unsqueeze(1).unsqueeze(2).expand(batch_size, seq_len, num_points).unsqueeze(-1)
frame_indices = frame_indices.unsqueeze(-1)
height_indices = pred_tracks_long[..., 0].unsqueeze(-1).unsqueeze(0)
width_indices = pred_tracks_long[..., 1].unsqueeze(-1).unsqueeze(0)

tracked_colors = sequence[batch_indices, frame_indices, :, width_indices, height_indices].squeeze()
np.save('/HPS/ColorNeRF/work/co-tracker/ouput_colors/tracked_colors_2.npy', tracked_colors.cpu().numpy())
color = tracked_colors[i, j].cpu().numpy()

I works perfectly when it has less images but when i use more images i get following error:

/opt/conda/conda-bld/pytorch_1712608843393/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [18,0,0], thread: [124,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1712608843393/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [18,0,0], thread: [125,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1712608843393/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [18,0,0], thread: [126,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1712608843393/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [18,0,0], thread: [127,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.

color = tracked_colors[i, j].cpu().numpy()
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertion

How many frames supported by model? Please help with this. Thanks.

[nikitakaraevv]

Hi @pknmax, the online model supports thousands of frames. I don't think that's the problem here. It looks like the index value is bigger than the tensor size, that's it. For debugging, try to move everything to the cpu. Then you'll see what exactly is going on.