train_dino.py giving error while training

[wathoresanket]

Epoch: [0] [ 60/253] eta: 0:01:35 lr: 0.000100 class_error: 92.45 loss: 29.6756 (36.5747) loss_ce_dn: 0.5717 (0.7828) loss_bbox_dn: 0.5367 (0.7324) loss_giou_dn: 1.1702 (1.3407) loss_ce: 0.7866 (0.9046) loss_bbox: 0.3596 (0.5975) loss_giou: 0.8695 (1.1540) loss_ce_0: 0.9594 (1.0261) loss_bbox_0: 0.3572 (0.5406) loss_giou_0: 0.8869 (1.1022) loss_ce_dn_0: 0.9315 (1.0496) loss_bbox_dn_0: 0.6607 (0.7547) loss_giou_dn_0: 1.2972 (1.3791) loss_ce_1: 0.8646 (0.9691) loss_bbox_1: 0.3485 (0.5864) loss_giou_1: 0.8861 (1.1392) loss_ce_dn_1: 0.7994 (0.9569) loss_bbox_dn_1: 0.6106 (0.7719) loss_giou_dn_1: 1.2202 (1.3694) loss_ce_2: 0.8457 (0.9557) loss_bbox_2: 0.3640 (0.5954) loss_giou_2: 0.8715 (1.1489) loss_ce_dn_2: 0.6891 (0.8808) loss_bbox_dn_2: 0.5469 (0.7455) loss_giou_dn_2: 1.1897 (1.3435) loss_ce_3: 0.8072 (0.9344) loss_bbox_3: 0.3573 (0.5966) loss_giou_3: 0.8757 (1.1504) loss_ce_dn_3: 0.6398 (0.8429) loss_bbox_dn_3: 0.5330 (0.7346) loss_giou_dn_3: 1.1670 (1.3379) loss_ce_4: 0.7871 (0.9193) loss_bbox_4: 0.3597 (0.5972) loss_giou_4: 0.8697 (1.1529) loss_ce_dn_4: 0.5905 (0.8046) loss_bbox_dn_4: 0.5354 (0.7327) loss_giou_dn_4: 1.1680 (1.3397) loss_ce_interm: 0.9274 (1.0068) loss_bbox_interm: 0.3579 (0.5187) loss_giou_interm: 0.8853 (1.0788) loss_ce_dn_unscaled: 0.5717 (0.7828) loss_bbox_dn_unscaled: 0.1073 (0.1465) loss_giou_dn_unscaled: 0.5851 (0.6704) loss_xy_dn_unscaled: 0.0488 (0.0524) loss_hw_dn_unscaled: 0.0556 (0.0941) cardinality_error_dn_unscaled: 154.0000 (158.1148) loss_ce_unscaled: 0.7866 (0.9046) class_error_unscaled: 95.9184 (96.0776) loss_bbox_unscaled: 0.0719 (0.1195) loss_giou_unscaled: 0.4347 (0.5770) loss_xy_unscaled: 0.0261 (0.0320) loss_hw_unscaled: 0.0458 (0.0875) cardinality_error_unscaled: 874.0000 (875.9508) loss_ce_0_unscaled: 0.9594 (1.0261) loss_bbox_0_unscaled: 0.0714 (0.1081) loss_giou_0_unscaled: 0.4434 (0.5511) loss_xy_0_unscaled: 0.0257 (0.0332) loss_hw_0_unscaled: 0.0458 (0.0750) cardinality_error_0_unscaled: 874.0000 (875.9508) loss_ce_dn_0_unscaled: 0.9315 (1.0496) loss_bbox_dn_0_unscaled: 0.1321 (0.1509) loss_giou_dn_0_unscaled: 0.6486 (0.6896) loss_xy_dn_0_unscaled: 0.0478 (0.0516) loss_hw_dn_0_unscaled: 0.0852 (0.0993) cardinality_error_dn_0_unscaled: 154.0000 (156.0328) loss_ce_1_unscaled: 0.8646 (0.9691) loss_bbox_1_unscaled: 0.0697 (0.1173) loss_giou_1_unscaled: 0.4430 (0.5696) loss_xy_1_unscaled: 0.0259 (0.0323) loss_hw_1_unscaled: 0.0446 (0.0850) cardinality_error_1_unscaled: 874.0000 (875.9508) loss_ce_dn_1_unscaled: 0.7994 (0.9569) loss_bbox_dn_1_unscaled: 0.1221 (0.1544) loss_giou_dn_1_unscaled: 0.6101 (0.6847) loss_xy_dn_1_unscaled: 0.0480 (0.0520) loss_hw_dn_1_unscaled: 0.0751 (0.1024) cardinality_error_dn_1_unscaled: 154.0000 (158.1475) loss_ce_2_unscaled: 0.8457 (0.9557) loss_bbox_2_unscaled: 0.0728 (0.1191) loss_giou_2_unscaled: 0.4357 (0.5744) loss_xy_2_unscaled: 0.0253 (0.0317) loss_hw_2_unscaled: 0.0472 (0.0874) cardinality_error_2_unscaled: 874.0000 (862.2459) loss_ce_dn_2_unscaled: 0.6891 (0.8808) loss_bbox_dn_2_unscaled: 0.1094 (0.1491) loss_giou_dn_2_unscaled: 0.5948 (0.6718) loss_xy_dn_2_unscaled: 0.0487 (0.0524) loss_hw_dn_2_unscaled: 0.0585 (0.0968) cardinality_error_dn_2_unscaled: 154.0000 (158.1311) loss_ce_3_unscaled: 0.8072 (0.9344) loss_bbox_3_unscaled: 0.0715 (0.1193) loss_giou_3_unscaled: 0.4379 (0.5752) loss_xy_3_unscaled: 0.0257 (0.0319) loss_hw_3_unscaled: 0.0452 (0.0874) cardinality_error_3_unscaled: 874.0000 (875.9508) loss_ce_dn_3_unscaled: 0.6398 (0.8429) loss_bbox_dn_3_unscaled: 0.1066 (0.1469) loss_giou_dn_3_unscaled: 0.5835 (0.6689) loss_xy_dn_3_unscaled: 0.0487 (0.0523) loss_hw_dn_3_unscaled: 0.0553 (0.0946) cardinality_error_dn_3_unscaled: 154.0000 (158.1475) loss_ce_4_unscaled: 0.7871 (0.9193) loss_bbox_4_unscaled: 0.0719 (0.1194) loss_giou_4_unscaled: 0.4349 (0.5764) loss_xy_4_unscaled: 0.0258 (0.0319) loss_hw_4_unscaled: 0.0454 (0.0875) cardinality_error_4_unscaled: 874.0000 (875.9426) loss_ce_dn_4_unscaled: 0.5905 (0.8046) loss_bbox_dn_4_unscaled: 0.1071 (0.1465) loss_giou_dn_4_unscaled: 0.5840 (0.6699) loss_xy_dn_4_unscaled: 0.0488 (0.0524) loss_hw_dn_4_unscaled: 0.0558 (0.0941) cardinality_error_dn_4_unscaled: 154.0000 (157.0574) loss_ce_interm_unscaled: 0.9274 (1.0068) loss_bbox_interm_unscaled: 0.0716 (0.1037) loss_giou_interm_unscaled: 0.4426 (0.5394) loss_xy_interm_unscaled: 0.0259 (0.0335) loss_hw_interm_unscaled: 0.0464 (0.0702) cardinality_error_interm_unscaled: 874.0000 (875.9508) time: 0.3474 data: 0.0070 max mem: 8505 error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device ßerror in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_im2col_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device error in ms_deformable_col2im_cuda: no kernel image is available for execution on the device