Training at epoch 1 end up with CUDA error and Assertion `t >= 0 && t < n_classes` failed

[kct22aws]

I ran the following command:

python tools/run_net.py \ --cfg configs/Kinetics/TimeSformer_divST_8x32_224_4gpus.yaml \ DATA.PATH_TO_DATA_DIR /home/ubuntu/vit/kinetics-dataset/k400/videos_resized \ NUM_GPUS 4 \ TRAIN.BATCH_SIZE 16 \ \

During training in epoch 1, I observed the following error:

[06/30 01:54:32][INFO] train_net.py: 446: Start epoch: 1 [06/30 01:54:46][INFO] distributed.py: 995: Reducer buckets have been rebuilt in this iteration. [06/30 01:54:59][INFO] logging.py: 95: json_stats: {"_type": "train_iter", "dt": 1.46034, "dt_data": 0.00346, "dt_net": 1.45688, "epoch": "1/15", "eta": "7:33:55", "gpu_mem": "7.68G", "iter": "10/1244", "loss": 6.05343, "lr": 0.00500, "top1_err": 100.00000, "top5_err": 100.00000} [06/30 01:55:14][INFO] logging.py: 95: json_stats: {"_type": "train_iter", "dt": 1.50371, "dt_data": 0.00334, "dt_net": 1.50036, "epoch": "1/15", "eta": "7:47:09", "gpu_mem": "7.68G", "iter": "20/1244", "loss": 6.16927, "lr": 0.00500, "top1_err": 100.00000, "top5_err": 100.00000} ../aten/src/ATen/native/cuda/Loss.cu:271: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [2,0,0] Assertion t >= 0 && t < n_classes failed. terminate called after throwing an instance of 'c10::CUDAError' what(): CUDA error: device-side assert triggered

Follow by a lengthy exceptions being raised: Exception raised from createEvent at ../aten/src/ATen/cuda/CUDAEvent.h:166 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f10084d2612 in /home/ubuntu/anaconda3/envs/timesformer/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: + 0xea8e4a (0x7f1009892e4a in /home/ubuntu/anaconda3/envs/timesformer/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so) frame #2: + 0x33a968 (0x7f1051d51968 in /home/ubuntu/anaconda3/envs/timesformer/lib/python3.7/site-packages/torch/lib/libtorch_python.so)

.....

In my instance I have four Tesla T4 GPU with Driver Version: 510.47.03 CUDA Version: 11.6

What does the error I see above means, and how do I fix it?

[xksteven]

Does it work with smaller batch size?

[SJTUPanda]

Data label in trian.csv should be started from 0. :)