How to reproduce the results?

[jih0-kim]

I am experiencing issues with achieving full reproducibility when running the code. Despite setting the seed to 0 and enabling deterministic=True in train.py, I am noticing that the results are not entirely identical across runs. For example, the loss values differ between Case 1 and Case 2 I’ve attached below. Could you please advise if there are additional steps required to ensure exact reproducibility?

• Case 1

  2024-11-01 12:53:55,736 - mmdet - INFO - Epoch [1][50/39434]    lr: 2.000e-05, eta: 8 days, 22:42:09, time: 1.634, data_time: 0.062, memory: 23917, enc_loss_cls: 0.0484, enc_loss_bbox: 5.0002, enc_loss_iou: 2.8541, loss_cls: 0.2409, loss_bbox: 4.9876, loss_iou: 2.8599, d0.loss_cls: 0.0612, d0.loss_bbox: 5.0394, d0.loss_iou: 2.8824, d1.loss_cls: 0.0841, d1.loss_bbox: 5.0198, d1.loss_iou: 2.8660, d2.loss_cls: 0.1196, d2.loss_bbox: 5.0018, d2.loss_iou: 2.8597, d3.loss_cls: 0.1376, d3.loss_bbox: 4.9940, d3.loss_iou: 2.8586, d4.loss_cls: 0.1867, d4.loss_bbox: 4.9884, d4.loss_iou: 2.8601, dn_loss_cls: 0.2034, dn_loss_bbox: 0.3142, dn_loss_iou: 1.3980, d0.dn_loss_cls: 0.1830, d0.dn_loss_bbox: 0.3145, d0.dn_loss_iou: 1.3695, d1.dn_loss_cls: 0.1905, d1.dn_loss_bbox: 0.3141, d1.dn_loss_iou: 1.3739, d2.dn_loss_cls: 0.2055, d2.dn_loss_bbox: 0.3140, d2.dn_loss_iou: 1.3790, d3.dn_loss_cls: 0.2172, d3.dn_loss_bbox: 0.3139, d3.dn_loss_iou: 1.3851, d4.dn_loss_cls: 0.2190, d4.dn_loss_bbox: 0.3140, d4.dn_loss_iou: 1.3911, loss_rpn_cls: 3.9324, loss_rpn_bbox: 0.5757, loss_cls0: 3.5153, acc0: 94.7695, loss_bbox0: 0.6281, loss_cls1: 11.7755, loss_bbox1: 13.8711, loss_centerness1: 8.1840, loss_cls_aux0: 1.1247, loss_bbox_aux0: 0.1301, loss_iou_aux0: 0.3912, d0.loss_cls_aux0: 1.0640, d0.loss_bbox_aux0: 0.1252, d0.loss_iou_aux0: 0.2333, d1.loss_cls_aux0: 1.0463, d1.loss_bbox_aux0: 0.1259, d1.loss_iou_aux0: 0.2663, d2.loss_cls_aux0: 1.0924, d2.loss_bbox_aux0: 0.1269, d2.loss_iou_aux0: 0.2987, d3.loss_cls_aux0: 1.0782, d3.loss_bbox_aux0: 0.1280, d3.loss_iou_aux0: 0.3361, d4.loss_cls_aux0: 1.1526, d4.loss_bbox_aux0: 0.1291, d4.loss_iou_aux0: 0.3623, loss_cls_aux1: 0.2029, loss_bbox_aux1: 0.2665, loss_iou_aux1: 1.2015, d0.loss_cls_aux1: 0.1843, d0.loss_bbox_aux1: 0.2671, d0.loss_iou_aux1: 1.1995, d1.loss_cls_aux1: 0.1886, d1.loss_bbox_aux1: 0.2667, d1.loss_iou_aux1: 1.1994, d2.loss_cls_aux1: 0.1972, d2.loss_bbox_aux1: 0.2666, d2.loss_iou_aux1: 1.1997, d3.loss_cls_aux1: 0.2020, d3.loss_bbox_aux1: 0.2665, d3.loss_iou_aux1: 1.2001, d4.loss_cls_aux1: 0.2103, d4.loss_bbox_aux1: 0.2665, d4.loss_iou_aux1: 1.2008, loss: 129.0304, grad_norm: 412.4246

• Case 2

  2024-11-01 12:55:57,369 - mmdet - INFO - Epoch [1][50/39434]    lr: 2.000e-05, eta: 9 days, 0:30:35, time: 1.647, data_time: 0.062, memory: 23917, enc_loss_cls: 0.0356, enc_loss_bbox: 4.5404, enc_loss_iou: 2.9436, loss_cls: 0.2533, loss_bbox: 4.5286, loss_iou: 2.9522, d0.loss_cls: 0.0624, d0.loss_bbox: 4.5806, d0.loss_iou: 2.9746, d1.loss_cls: 0.0896, d1.loss_bbox: 4.5612, d1.loss_iou: 2.9639, d2.loss_cls: 0.1209, d2.loss_bbox: 4.5446, d2.loss_iou: 2.9599, d3.loss_cls: 0.1514, d3.loss_bbox: 4.5365, d3.loss_iou: 2.9563, d4.loss_cls: 0.2030, d4.loss_bbox: 4.5310, d4.loss_iou: 2.9551, dn_loss_cls: 0.2045, dn_loss_bbox: 0.3148, dn_loss_iou: 1.3984, d0.dn_loss_cls: 0.1835, d0.dn_loss_bbox: 0.3145, d0.dn_loss_iou: 1.3694, d1.dn_loss_cls: 0.1867, d1.dn_loss_bbox: 0.3142, d1.dn_loss_iou: 1.3728, d2.dn_loss_cls: 0.1941, d2.dn_loss_bbox: 0.3142, d2.dn_loss_iou: 1.3794, d3.dn_loss_cls: 0.2104, d3.dn_loss_bbox: 0.3142, d3.dn_loss_iou: 1.3860, d4.dn_loss_cls: 0.2160, d4.dn_loss_bbox: 0.3145, d4.dn_loss_iou: 1.3920, loss_rpn_cls: 3.9968, loss_rpn_bbox: 0.5839, loss_cls0: 3.3748, acc0: 94.7988, loss_bbox0: 0.6066, loss_cls1: 11.9814, loss_bbox1: 13.8896, loss_centerness1: 8.1629, loss_cls_aux0: 1.1132, loss_bbox_aux0: 0.1340, loss_iou_aux0: 0.3945, d0.loss_cls_aux0: 1.0627, d0.loss_bbox_aux0: 0.1284, d0.loss_iou_aux0: 0.2344, d1.loss_cls_aux0: 1.0492, d1.loss_bbox_aux0: 0.1293, d1.loss_iou_aux0: 0.2643, d2.loss_cls_aux0: 1.0861, d2.loss_bbox_aux0: 0.1304, d2.loss_iou_aux0: 0.2991, d3.loss_cls_aux0: 1.0802, d3.loss_bbox_aux0: 0.1317, d3.loss_iou_aux0: 0.3375, d4.loss_cls_aux0: 1.1517, d4.loss_bbox_aux0: 0.1328, d4.loss_iou_aux0: 0.3664, loss_cls_aux1: 0.2019, loss_bbox_aux1: 0.2665, loss_iou_aux1: 1.2024, d0.loss_cls_aux1: 0.1837, d0.loss_bbox_aux1: 0.2671, d0.loss_iou_aux1: 1.1997, d1.loss_cls_aux1: 0.1877, d1.loss_bbox_aux1: 0.2668, d1.loss_iou_aux1: 1.1995, d2.loss_cls_aux1: 0.1927, d2.loss_bbox_aux1: 0.2666, d2.loss_iou_aux1: 1.1997, d3.loss_cls_aux1: 0.1969, d3.loss_bbox_aux1: 0.2664, d3.loss_iou_aux1: 1.2001, d4.loss_cls_aux1: 0.2099, d4.loss_bbox_aux1: 0.2664, d4.loss_iou_aux1: 1.2011, loss: 126.6219, grad_norm: 414.0944

[TempleX98]

Atomic operations in CUDA are unordered and random when parallelized. Therefore, even with a fixed seed and deterministic settings, training results can still vary.