Thanks for your work. I am training the model on custom video dataset. While doing evaluation after one epoch, I get error
modeling/multimodal_meta_archs.py", line 619, in
torch.cat(x) for x in [segs_all, scores_all, cls_idxs_all]
RuntimeError: torch.cat(): expected a non-empty list of Tensors
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:31 (most recent call first):
After sometime, figured out that in def inference_single_video() function:
topk idxs tensor([94585022332496, 94582502706592, 94585097936080, ..., 94585022005264, 140088139025376, 64],
device='cuda:0')
pt_idxs tensor([94585098952080, 140088139025376, 64, ..., 94585021827888, 140088139025376, 64],
device='cuda:0')
are becoming very large.
Can you suggest what could be the issue?
Thank you.
Thanks for your work. I am training the model on custom video dataset. While doing evaluation after one epoch, I get error modeling/multimodal_meta_archs.py", line 619, in
torch.cat(x) for x in [segs_all, scores_all, cls_idxs_all]
RuntimeError: torch.cat(): expected a non-empty list of Tensors
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:31 (most recent call first):
After sometime, figured out that in def inference_single_video() function: topk idxs tensor([94585022332496, 94582502706592, 94585097936080, ..., 94585022005264, 140088139025376, 64], device='cuda:0') pt_idxs tensor([94585098952080, 140088139025376, 64, ..., 94585021827888, 140088139025376, 64], device='cuda:0') are becoming very large.
Can you suggest what could be the issue? Thank you.