search

xiemk / SPML-LAC

16 stars 0 forks source link

Loss is NAN. #4

Open boyiZheng99 opened 3 weeks ago

boyiZheng99 commented 3 weeks ago

07/03 00:00:54.724]: Epoch: [12/40][1000/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.522e-05 Loss nan ( nan) Mem 15346 [07/03 00:01:24.369]: Epoch: [12/40][1100/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.512e-05 Loss nan ( nan) Mem 15346 [07/03 00:01:53.892]: Epoch: [12/40][1200/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.502e-05 Loss nan ( nan) Mem 15346 [07/03 00:02:23.294]: Epoch: [12/40][1300/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.491e-05 Loss nan ( nan) Mem 15346 [07/03 00:02:52.894]: Epoch: [12/40][1400/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.481e-05 Loss nan ( nan) Mem 15346 [07/03 00:03:22.470]: Epoch: [12/40][1500/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.470e-05 Loss nan ( nan) Mem 15346 [07/03 00:03:51.948]: Epoch: [12/40][1600/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.459e-05 Loss nan ( nan) Mem 15346 [07/03 00:04:21.439]: Epoch: [12/40][1700/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.448e-05 Loss nan ( nan) Mem 15346 [07/03 00:04:51.830]: Epoch: [12/40][1800/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.437e-05 Loss nan ( nan) Mem 15346 [07/03 00:05:21.308]: Epoch: [12/40][1900/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.426e-05 Loss nan ( nan) Mem 15346 [07/03 00:05:50.798]: Epoch: [12/40][2000/2052] L_an nan ( nan) L_plc nan ( nan) LR 9.415e-05 Loss nan ( nan) Mem 15346 [07/03 00:06:06.870]: Test: [ 0/513] Time 0.822 (0.822) Mem 15346 [07/03 00:06:13.675]: Test: [100/513] Time 0.068 (0.076) Mem 15346 [07/03 00:06:20.474]: Test: [200/513] Time 0.068 (0.072) Mem 15346 [07/03 00:06:27.685]: Test: [300/513] Time 0.068 (0.072) Mem 15346 [07/03 00:06:34.480]: Test: [400/513] Time 0.068 (0.071) Mem 15346 [07/03 00:06:41.277]: Test: [500/513] Time 0.068 (0.070) Mem 15346 Traceback (most recent call last): File "/projappl/project_2009557/SOTAs_method/SPML-LAC/script/../first_stage.py", line 446, in main() File "/projappl/project_2009557/SOTAs_method/SPML-LAC/script/../first_stage.py", line 129, in main return main_worker(args, logger) File "/projappl/project_2009557/SOTAs_method/SPML-LAC/script/../first_stage.py", line 244, in main_worker mAP = validate(val_loader, model, args, logger) File "/MAHTI_TYKKY_MXuVtFD/miniconda/envs/env1/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/projappl/project_2009557/SOTAs_method/SPML-LAC/script/../first_stage.py", line 437, in validate mAP = function_mAP(torch.cat(targets_list).numpy(), torch.cat(outputs_sm_list).numpy()) File "/projappl/project_2009557/SOTAs_method/SPML-LAC/lib/utils/helper.py", line 39, in function_mAP average_precision_list.append(compute_avg_precision(targs[:, j], preds[:, j])) File "/projappl/project_2009557/SOTAs_method/SPML-LAC/lib/utils/helper.py", line 54, in compute_avg_precision check_inputs(targs,preds) File "/projappl/project_2009557/SOTAs_method/SPML-LAC/lib/utils/helper.py", line 74, in check_inputs assert (np.max(preds) <= 1.0) and (np.min(preds) >= 0.0) AssertionError

Hello，Excuse me, why do I encounter the problem of loss becoming nan during training?

xiemk commented 2 weeks ago

Hello~ Which dataset do you use? Do you still face the NAN problem when repeating experiments?

boyiZheng99 commented 2 weeks ago

Hello, I tried the coco2014(same as the original paper) and my dataset, all face this issue. I do not change any protocol reported in paper.

BRs, Boyi

发件人: xiemk @.> 发送时间: 2024年7月10日 11:24 收件人: xiemk/SPML-LAC @.> 抄送: Boyi Zheng @.>; Author @.> 主题: Re: [xiemk/SPML-LAC] Loss is NAN. (Issue #4)

Hello~ Which dataset do you use? Do you still face the NAN problem when repeating experiments?

― Reply to this email directly, view it on GitHubhttps://github.com/xiemk/SPML-LAC/issues/4#issuecomment-2219470392, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGVAPYBVKQMXSR4BJMII6P3ZLSSQTAVCNFSM6AAAAABKJAR5YGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJZGQ3TAMZZGI. You are receiving this because you authored the thread.Message ID: @.***>

xiemk commented 2 weeks ago

I show our training logs on COCO at the second stage.

[05/03 23:15:56.145]: Epoch: [0/40][   0/2052]  Lx 0.023 (0.023)  Lu 0.016 (0.016)  Lc 3.963 (3.963)  LR 4.000e-06  Loss 4.002 (4.002)  Mem 13387
[05/03 23:17:01.437]: Epoch: [0/40][ 100/2052]  Lx 0.032 (0.029)  Lu 0.025 (0.023)  Lc 5.924 (6.061)  LR 4.009e-06  Loss 5.981 (6.113)  Mem 15453
[05/03 23:18:05.749]: Epoch: [0/40][ 200/2052]  Lx 0.023 (0.028)  Lu 0.017 (0.022)  Lc 5.731 (5.941)  LR 4.036e-06  Loss 5.771 (5.991)  Mem 15453
[05/03 23:19:10.310]: Epoch: [0/40][ 300/2052]  Lx 0.024 (0.028)  Lu 0.018 (0.023)  Lc 5.637 (5.851)  LR 4.080e-06  Loss 5.678 (5.901)  Mem 15453
[05/03 23:20:15.317]: Epoch: [0/40][ 400/2052]  Lx 0.034 (0.028)  Lu 0.029 (0.022)  Lc 5.546 (5.783)  LR 4.141e-06  Loss 5.609 (5.833)  Mem 15453
[05/03 23:21:19.089]: Epoch: [0/40][ 500/2052]  Lx 0.023 (0.028)  Lu 0.020 (0.022)  Lc 5.486 (5.730)  LR 4.220e-06  Loss 5.530 (5.780)  Mem 15453
[05/03 23:22:22.687]: Epoch: [0/40][ 600/2052]  Lx 0.030 (0.028)  Lu 0.023 (0.023)  Lc 5.449 (5.686)  LR 4.317e-06  Loss 5.502 (5.737)  Mem 15453
[05/03 23:23:26.304]: Epoch: [0/40][ 700/2052]  Lx 0.031 (0.028)  Lu 0.028 (0.023)  Lc 5.397 (5.650)  LR 4.431e-06  Loss 5.456 (5.700)  Mem 15453
[05/03 23:24:29.266]: Epoch: [0/40][ 800/2052]  Lx 0.027 (0.028)  Lu 0.026 (0.023)  Lc 5.387 (5.621)  LR 4.563e-06  Loss 5.440 (5.671)  Mem 15453
[05/03 23:25:32.382]: Epoch: [0/40][ 900/2052]  Lx 0.020 (0.028)  Lu 0.014 (0.022)  Lc 5.374 (5.595)  LR 4.712e-06  Loss 5.408 (5.645)  Mem 15453
[05/03 23:26:35.650]: Epoch: [0/40][1000/2052]  Lx 0.023 (0.028)  Lu 0.017 (0.023)  Lc 5.340 (5.572)  LR 4.878e-06  Loss 5.381 (5.622)  Mem 15453
[05/03 23:27:39.125]: Epoch: [0/40][1100/2052]  Lx 0.029 (0.027)  Lu 0.029 (0.023)  Lc 5.363 (5.552)  LR 5.062e-06  Loss 5.421 (5.602)  Mem 15453
[05/03 23:28:42.204]: Epoch: [0/40][1200/2052]  Lx 0.029 (0.027)  Lu 0.019 (0.023)  Lc 5.313 (5.535)  LR 5.262e-06  Loss 5.361 (5.585)  Mem 15453
[05/03 23:29:45.384]: Epoch: [0/40][1300/2052]  Lx 0.029 (0.027)  Lu 0.022 (0.022)  Lc 5.330 (5.520)  LR 5.480e-06  Loss 5.381 (5.570)  Mem 15453
[05/03 23:30:48.917]: Epoch: [0/40][1400/2052]  Lx 0.028 (0.027)  Lu 0.026 (0.022)  Lc 5.317 (5.506)  LR 5.715e-06  Loss 5.371 (5.556)  Mem 15453
[05/03 23:31:52.274]: Epoch: [0/40][1500/2052]  Lx 0.025 (0.027)  Lu 0.019 (0.022)  Lc 5.331 (5.494)  LR 5.967e-06  Loss 5.375 (5.544)  Mem 15453
[05/03 23:32:55.537]: Epoch: [0/40][1600/2052]  Lx 0.028 (0.027)  Lu 0.023 (0.022)  Lc 5.301 (5.482)  LR 6.236e-06  Loss 5.352 (5.532)  Mem 15453
[05/03 23:33:58.578]: Epoch: [0/40][1700/2052]  Lx 0.027 (0.027)  Lu 0.021 (0.022)  Lc 5.284 (5.471)  LR 6.521e-06  Loss 5.332 (5.521)  Mem 15453
[05/03 23:35:01.804]: Epoch: [0/40][1800/2052]  Lx 0.031 (0.027)  Lu 0.024 (0.022)  Lc 5.270 (5.461)  LR 6.823e-06  Loss 5.325 (5.511)  Mem 15453
[05/03 23:36:05.233]: Epoch: [0/40][1900/2052]  Lx 0.033 (0.027)  Lu 0.029 (0.022)  Lc 5.278 (5.452)  LR 7.142e-06  Loss 5.340 (5.501)  Mem 15453
[05/03 23:37:08.245]: Epoch: [0/40][2000/2052]  Lx 0.031 (0.027)  Lu 0.026 (0.022)  Lc 5.281 (5.443)  LR 7.477e-06  Loss 5.337 (5.493)  Mem 15453
[05/03 23:37:43.215]: Test: [  0/513]  Time 2.428 (2.428)  Mem 15453
[05/03 23:37:56.799]: Test: [100/513]  Time 0.135 (0.159)  Mem 15453
[05/03 23:38:10.427]: Test: [200/513]  Time 0.135 (0.147)  Mem 15453
[05/03 23:38:23.979]: Test: [300/513]  Time 0.135 (0.143)  Mem 15453
[05/03 23:38:37.618]: Test: [400/513]  Time 0.151 (0.142)  Mem 15453
[05/03 23:38:51.235]: Test: [500/513]  Time 0.135 (0.141)  Mem 15453
[05/03 23:38:53.548]:   mAP: 75.15849566274714
[05/03 23:38:55.444]: Test: [  0/513]  Time 1.893 (1.893)  Mem 15453
[05/03 23:39:09.038]: Test: [100/513]  Time 0.135 (0.153)  Mem 15453
[05/03 23:39:22.583]: Test: [200/513]  Time 0.135 (0.144)  Mem 15453
[05/03 23:39:36.181]: Test: [300/513]  Time 0.134 (0.142)  Mem 15453
[05/03 23:39:49.771]: Test: [400/513]  Time 0.135 (0.140)  Mem 15453
[05/03 23:40:03.333]: Test: [500/513]  Time 0.135 (0.139)  Mem 15453
[05/03 23:40:05.669]:   mAP: 75.02602728240072
[05/03 23:40:05.670]: => Test Epoch: [ 0/40]  ETA 15:45:36  TT 0:24:14 (0:24:14)  mAP 75.15850  mAP_ema 75.02603
[05/03 23:40:07.339]: Test: [   0/1255]  Time 1.664 (1.664)  Mem 15453
[05/03 23:40:21.074]: Test: [ 100/1255]  Time 0.135 (0.152)  Mem 15453
[05/03 23:40:34.650]: Test: [ 200/1255]  Time 0.135 (0.144)  Mem 15453
[05/03 23:40:48.228]: Test: [ 300/1255]  Time 0.135 (0.141)  Mem 15453
[05/03 23:41:01.764]: Test: [ 400/1255]  Time 0.135 (0.140)  Mem 15453
[05/03 23:41:15.306]: Test: [ 500/1255]  Time 0.135 (0.139)  Mem 15453
[05/03 23:41:28.885]: Test: [ 600/1255]  Time 0.135 (0.138)  Mem 15453
[05/03 23:41:42.505]: Test: [ 700/1255]  Time 0.135 (0.138)  Mem 15453
[05/03 23:41:56.096]: Test: [ 800/1255]  Time 0.135 (0.138)  Mem 15453
[05/03 23:42:09.766]: Test: [ 900/1255]  Time 0.135 (0.138)  Mem 15453
[05/03 23:42:23.358]: Test: [1000/1255]  Time 0.142 (0.138)  Mem 15453
[05/03 23:42:36.910]: Test: [1100/1255]  Time 0.135 (0.137)  Mem 15453
[05/03 23:42:50.556]: Test: [1200/1255]  Time 0.135 (0.137)  Mem 15453
[05/03 23:42:59.178]:   mAP: 75.03725031148134
[05/03 23:42:59.181]: 0 | Set best mAP 75.15849566274714 in ep 0
[05/03 23:42:59.181]:    | best regular mAP 75.15849566274714 in ep 0
[05/03 23:42:59.181]:    | best test mAP 75.03725031148134 
[05/03 23:43:04.795]: lr:7.654221892462539e-06
[05/03 23:43:08.689]: Epoch: [1/40][   0/2052]  Lx 0.025 (0.025)  Lu 0.019 (0.019)  Lc 5.272 (5.272)  LR 7.658e-06  Loss 5.316 (5.316)  Mem 15453
[05/03 23:44:11.555]: Epoch: [1/40][ 100/2052]  Lx 0.029 (0.027)  Lu 0.025 (0.021)  Lc 5.256 (5.258)  LR 8.018e-06  Loss 5.311 (5.306)  Mem 15454
[05/03 23:45:14.235]: Epoch: [1/40][ 200/2052]  Lx 0.022 (0.026)  Lu 0.019 (0.022)  Lc 5.263 (5.260)  LR 8.394e-06  Loss 5.303 (5.308)  Mem 15454
[05/03 23:46:16.797]: Epoch: [1/40][ 300/2052]  Lx 0.027 (0.026)  Lu 0.019 (0.022)  Lc 5.224 (5.257)  LR 8.785e-06  Loss 5.270 (5.305)  Mem 15454
[05/03 23:47:20.876]: Epoch: [1/40][ 400/2052]  Lx 0.034 (0.027)  Lu 0.026 (0.022)  Lc 5.228 (5.255)  LR 9.193e-06  Loss 5.288 (5.304)  Mem 15454
[05/03 23:48:25.032]: Epoch: [1/40][ 500/2052]  Lx 0.029 (0.027)  Lu 0.025 (0.022)  Lc 5.245 (5.252)  LR 9.617e-06  Loss 5.300 (5.300)  Mem 15454
[05/03 23:49:28.542]: Epoch: [1/40][ 600/2052]  Lx 0.023 (0.027)  Lu 0.014 (0.022)  Lc 5.233 (5.248)  LR 1.006e-05  Loss 5.269 (5.297)  Mem 15454
[05/03 23:50:32.317]: Epoch: [1/40][ 700/2052]  Lx 0.028 (0.027)  Lu 0.019 (0.022)  Lc 5.229 (5.246)  LR 1.051e-05  Loss 5.276 (5.295)  Mem 15454
[05/03 23:51:36.126]: Epoch: [1/40][ 800/2052]  Lx 0.026 (0.027)  Lu 0.022 (0.022)  Lc 5.223 (5.245)  LR 1.098e-05  Loss 5.270 (5.294)  Mem 15454
[05/03 23:52:40.102]: Epoch: [1/40][ 900/2052]  Lx 0.025 (0.027)  Lu 0.019 (0.022)  Lc 5.243 (5.243)  LR 1.146e-05  Loss 5.287 (5.292)  Mem 15454
[05/03 23:53:43.725]: Epoch: [1/40][1000/2052]  Lx 0.029 (0.027)  Lu 0.024 (0.022)  Lc 5.201 (5.240)  LR 1.196e-05  Loss 5.254 (5.289)  Mem 15454
[05/03 23:54:47.654]: Epoch: [1/40][1100/2052]  Lx 0.024 (0.027)  Lu 0.018 (0.022)  Lc 5.207 (5.238)  LR 1.248e-05  Loss 5.250 (5.287)  Mem 15454
[05/03 23:55:51.102]: Epoch: [1/40][1200/2052]  Lx 0.025 (0.027)  Lu 0.022 (0.022)  Lc 5.219 (5.235)  LR 1.301e-05  Loss 5.265 (5.284)  Mem 15454
[05/03 23:56:54.934]: Epoch: [1/40][1300/2052]  Lx 0.031 (0.027)  Lu 0.024 (0.022)  Lc 5.238 (5.233)  LR 1.355e-05  Loss 5.293 (5.282)  Mem 15454
[05/03 23:57:58.057]: Epoch: [1/40][1400/2052]  Lx 0.027 (0.027)  Lu 0.023 (0.022)  Lc 5.206 (5.231)  LR 1.411e-05  Loss 5.255 (5.280)  Mem 15454
[05/03 23:59:01.600]: Epoch: [1/40][1500/2052]  Lx 0.034 (0.027)  Lu 0.030 (0.022)  Lc 5.199 (5.228)  LR 1.468e-05  Loss 5.263 (5.277)  Mem 15454
[05/04 00:00:05.417]: Epoch: [1/40][1600/2052]  Lx 0.025 (0.027)  Lu 0.020 (0.022)  Lc 5.194 (5.226)  LR 1.526e-05  Loss 5.238 (5.275)  Mem 15454
[05/04 00:01:09.445]: Epoch: [1/40][1700/2052]  Lx 0.023 (0.027)  Lu 0.021 (0.022)  Lc 5.205 (5.225)  LR 1.586e-05  Loss 5.249 (5.274)  Mem 15454
[05/04 00:02:13.430]: Epoch: [1/40][1800/2052]  Lx 0.030 (0.027)  Lu 0.033 (0.022)  Lc 5.202 (5.223)  LR 1.647e-05  Loss 5.265 (5.272)  Mem 15454
[05/04 00:03:17.466]: Epoch: [1/40][1900/2052]  Lx 0.031 (0.027)  Lu 0.021 (0.022)  Lc 5.188 (5.222)  LR 1.709e-05  Loss 5.240 (5.271)  Mem 15454
[05/04 00:04:20.421]: Epoch: [1/40][2000/2052]  Lx 0.028 (0.027)  Lu 0.021 (0.022)  Lc 5.197 (5.220)  LR 1.773e-05  Loss 5.247 (5.270)  Mem 15454
[05/04 00:04:54.733]: Test: [  0/513]  Time 2.102 (2.102)  Mem 15454
[05/04 00:05:08.388]: Test: [100/513]  Time 0.135 (0.156)  Mem 15454
[05/04 00:05:22.091]: Test: [200/513]  Time 0.135 (0.147)  Mem 15454
[05/04 00:05:35.680]: Test: [300/513]  Time 0.135 (0.143)  Mem 15454
[05/04 00:05:49.210]: Test: [400/513]  Time 0.135 (0.141)  Mem 15454
[05/04 00:06:02.862]: Test: [500/513]  Time 0.135 (0.140)  Mem 15454
[05/04 00:06:05.204]:   mAP: 74.35406867306094
[05/04 00:06:07.080]: Test: [  0/513]  Time 1.873 (1.873)  Mem 15454
[05/04 00:06:20.645]: Test: [100/513]  Time 0.135 (0.153)  Mem 15454
[05/04 00:06:34.244]: Test: [200/513]  Time 0.135 (0.144)  Mem 15454
[05/04 00:06:47.813]: Test: [300/513]  Time 0.135 (0.142)  Mem 15454
[05/04 00:07:01.413]: Test: [400/513]  Time 0.135 (0.140)  Mem 15454
[05/04 00:07:14.969]: Test: [500/513]  Time 0.142 (0.139)  Mem 15454
[05/04 00:07:17.288]:   mAP: 75.51458632384087
[05/04 00:07:17.290]: => Test Epoch: [ 1/40]  ETA 16:17:21  TT 0:27:11 (0:51:26)  mAP 74.35407  mAP_ema 75.51459
[05/04 00:07:19.107]: Test: [   0/1255]  Time 1.812 (1.812)  Mem 15454
[05/04 00:07:32.691]: Test: [ 100/1255]  Time 0.135 (0.152)  Mem 15454
[05/04 00:07:46.323]: Test: [ 200/1255]  Time 0.135 (0.144)  Mem 15454
[05/04 00:07:59.933]: Test: [ 300/1255]  Time 0.135 (0.142)  Mem 15454
[05/04 00:08:13.566]: Test: [ 400/1255]  Time 0.135 (0.140)  Mem 15454
[05/04 00:08:27.175]: Test: [ 500/1255]  Time 0.136 (0.139)  Mem 15454
[05/04 00:08:40.705]: Test: [ 600/1255]  Time 0.135 (0.139)  Mem 15454
[05/04 00:08:54.261]: Test: [ 700/1255]  Time 0.134 (0.138)  Mem 15454
[05/04 00:09:07.834]: Test: [ 800/1255]  Time 0.152 (0.138)  Mem 15454
[05/04 00:09:21.474]: Test: [ 900/1255]  Time 0.136 (0.138)  Mem 15454
[05/04 00:09:35.062]: Test: [1000/1255]  Time 0.135 (0.138)  Mem 15454
[05/04 00:09:48.634]: Test: [1100/1255]  Time 0.135 (0.137)  Mem 15454
[05/04 00:10:02.292]: Test: [1200/1255]  Time 0.138 (0.137)  Mem 15454
[05/04 00:10:11.004]:   mAP: 75.301907198553
[05/04 00:10:11.008]: 1 | Set best mAP 75.51458632384087 in ep 1
[05/04 00:10:11.008]:    | best regular mAP 75.15849566274714 in ep 0
[05/04 00:10:11.008]:    | best test mAP 75.301907198553 
[05/04 00:10:20.809]: lr:1.806049850154381e-05
[05/04 00:10:24.488]: Epoch: [2/40][   0/2052]  Lx 0.019 (0.019)  Lu 0.012 (0.012)  Lc 5.180 (5.180)  LR 1.807e-05  Loss 5.211 (5.211)  Mem 15454
[05/04 00:11:26.884]: Epoch: [2/40][ 100/2052]  Lx 0.026 (0.026)  Lu 0.022 (0.022)  Lc 5.177 (5.194)  LR 1.872e-05  Loss 5.225 (5.242)  Mem 15454
[05/04 00:12:29.377]: Epoch: [2/40][ 200/2052]  Lx 0.026 (0.026)  Lu 0.021 (0.022)  Lc 5.158 (5.182)  LR 1.939e-05  Loss 5.205 (5.231)  Mem 15454
[05/04 00:13:32.077]: Epoch: [2/40][ 300/2052]  Lx 0.026 (0.026)  Lu 0.021 (0.022)  Lc 5.227 (5.181)  LR 2.007e-05  Loss 5.273 (5.230)  Mem 15454
[05/04 00:14:34.587]: Epoch: [2/40][ 400/2052]  Lx 0.026 (0.026)  Lu 0.020 (0.023)  Lc 5.185 (5.181)  LR 2.076e-05  Loss 5.231 (5.230)  Mem 15454
[05/04 00:15:37.299]: Epoch: [2/40][ 500/2052]  Lx 0.023 (0.027)  Lu 0.021 (0.023)  Lc 5.197 (5.179)  LR 2.147e-05  Loss 5.242 (5.229)  Mem 15454
[05/04 00:16:39.861]: Epoch: [2/40][ 600/2052]  Lx 0.031 (0.027)  Lu 0.022 (0.023)  Lc 5.186 (5.179)  LR 2.218e-05  Loss 5.239 (5.228)  Mem 15454
[05/04 00:17:42.788]: Epoch: [2/40][ 700/2052]  Lx 0.023 (0.027)  Lu 0.018 (0.023)  Lc 5.157 (5.179)  LR 2.291e-05  Loss 5.198 (5.229)  Mem 15454
[05/04 00:18:46.063]: Epoch: [2/40][ 800/2052]  Lx 0.026 (0.027)  Lu 0.021 (0.023)  Lc 5.162 (5.179)  LR 2.364e-05  Loss 5.209 (5.228)  Mem 15454
[05/04 00:19:48.782]: Epoch: [2/40][ 900/2052]  Lx 0.030 (0.027)  Lu 0.026 (0.023)  Lc 5.149 (5.177)  LR 2.439e-05  Loss 5.205 (5.227)  Mem 15454
[05/04 00:20:51.446]: Epoch: [2/40][1000/2052]  Lx 0.027 (0.027)  Lu 0.019 (0.023)  Lc 5.176 (5.176)  LR 2.514e-05  Loss 5.222 (5.226)  Mem 15454
[05/04 00:21:54.157]: Epoch: [2/40][1100/2052]  Lx 0.030 (0.027)  Lu 0.022 (0.023)  Lc 5.160 (5.175)  LR 2.591e-05  Loss 5.212 (5.226)  Mem 15454
[05/04 00:22:56.348]: Epoch: [2/40][1200/2052]  Lx 0.027 (0.027)  Lu 0.019 (0.023)  Lc 5.177 (5.175)  LR 2.669e-05  Loss 5.224 (5.225)  Mem 15454
[05/04 00:23:58.919]: Epoch: [2/40][1300/2052]  Lx 0.025 (0.027)  Lu 0.022 (0.023)  Lc 5.162 (5.174)  LR 2.747e-05  Loss 5.209 (5.225)  Mem 15454
[05/04 00:25:02.117]: Epoch: [2/40][1400/2052]  Lx 0.031 (0.027)  Lu 0.030 (0.023)  Lc 5.161 (5.173)  LR 2.827e-05  Loss 5.222 (5.224)  Mem 15454
[05/04 00:26:04.837]: Epoch: [2/40][1500/2052]  Lx 0.027 (0.027)  Lu 0.025 (0.023)  Lc 5.163 (5.173)  LR 2.907e-05  Loss 5.215 (5.223)  Mem 15454
[05/04 00:27:07.419]: Epoch: [2/40][1600/2052]  Lx 0.032 (0.027)  Lu 0.031 (0.023)  Lc 5.178 (5.173)  LR 2.988e-05  Loss 5.241 (5.224)  Mem 15454
[05/04 00:28:10.357]: Epoch: [2/40][1700/2052]  Lx 0.029 (0.027)  Lu 0.020 (0.023)  Lc 5.171 (5.172)  LR 3.070e-05  Loss 5.220 (5.223)  Mem 15454
[05/04 00:29:13.095]: Epoch: [2/40][1800/2052]  Lx 0.026 (0.028)  Lu 0.019 (0.023)  Lc 5.180 (5.171)  LR 3.153e-05  Loss 5.225 (5.222)  Mem 15454
[05/04 00:30:15.820]: Epoch: [2/40][1900/2052]  Lx 0.031 (0.028)  Lu 0.029 (0.023)  Lc 5.144 (5.171)  LR 3.236e-05  Loss 5.204 (5.222)  Mem 15454
[05/04 00:31:18.463]: Epoch: [2/40][2000/2052]  Lx 0.035 (0.028)  Lu 0.033 (0.024)  Lc 5.164 (5.171)  LR 3.320e-05  Loss 5.232 (5.222)  Mem 15454
[05/04 00:31:52.119]: Test: [  0/513]  Time 1.743 (1.743)  Mem 15454
[05/04 00:32:05.857]: Test: [100/513]  Time 0.136 (0.153)  Mem 15454
[05/04 00:32:19.407]: Test: [200/513]  Time 0.135 (0.144)  Mem 15454
[05/04 00:32:32.961]: Test: [300/513]  Time 0.135 (0.141)  Mem 15454
[05/04 00:32:46.549]: Test: [400/513]  Time 0.142 (0.140)  Mem 15454
[05/04 00:33:00.196]: Test: [500/513]  Time 0.135 (0.139)  Mem 15454
[05/04 00:33:02.512]:   mAP: 71.8178161928602
[05/04 00:33:04.448]: Test: [  0/513]  Time 1.934 (1.934)  Mem 15454
[05/04 00:33:18.044]: Test: [100/513]  Time 0.135 (0.154)  Mem 15454
[05/04 00:33:31.607]: Test: [200/513]  Time 0.135 (0.145)  Mem 15454
[05/04 00:33:45.154]: Test: [300/513]  Time 0.135 (0.142)  Mem 15454
[05/04 00:33:58.684]: Test: [400/513]  Time 0.135 (0.140)  Mem 15454
[05/04 00:34:12.305]: Test: [500/513]  Time 0.135 (0.139)  Mem 15454
[05/04 00:34:14.598]:   mAP: 75.78817195951413
[05/04 00:34:14.599]: => Test Epoch: [ 2/40]  ETA 16:06:52  TT 0:26:57 (1:18:23)  mAP 71.81782  mAP_ema 75.78817
[05/04 00:34:16.539]: Test: [   0/1255]  Time 1.936 (1.936)  Mem 15454
[05/04 00:34:30.123]: Test: [ 100/1255]  Time 0.135 (0.154)  Mem 15454
[05/04 00:34:43.713]: Test: [ 200/1255]  Time 0.135 (0.145)  Mem 15454
[05/04 00:34:57.251]: Test: [ 300/1255]  Time 0.135 (0.142)  Mem 15454
[05/04 00:35:10.917]: Test: [ 400/1255]  Time 0.135 (0.140)  Mem 15454
[05/04 00:35:24.511]: Test: [ 500/1255]  Time 0.135 (0.140)  Mem 15454
[05/04 00:35:38.185]: Test: [ 600/1255]  Time 0.135 (0.139)  Mem 15454
[05/04 00:35:51.785]: Test: [ 700/1255]  Time 0.135 (0.139)  Mem 15454
[05/04 00:36:05.455]: Test: [ 800/1255]  Time 0.135 (0.138)  Mem 15454
[05/04 00:36:19.084]: Test: [ 900/1255]  Time 0.135 (0.138)  Mem 15454
[05/04 00:36:32.784]: Test: [1000/1255]  Time 0.135 (0.138)  Mem 15454
[05/04 00:36:46.412]: Test: [1100/1255]  Time 0.135 (0.138)  Mem 15454
[05/04 00:37:00.090]: Test: [1200/1255]  Time 0.135 (0.138)  Mem 15454
[05/04 00:37:08.685]:   mAP: 75.5696857912416
[05/04 00:37:08.688]: 2 | Set best mAP 75.78817195951413 in ep 2
[05/04 00:37:08.688]:    | best regular mAP 75.15849566274714 in ep 0
[05/04 00:37:08.688]:    | best test mAP 75.5696857912416 
[05/04 00:37:18.486]: lr:3.363437800208025e-05
[05/04 00:37:22.000]: Epoch: [3/40][   0/2052]  Lx 0.026 (0.026)  Lu 0.023 (0.023)  Lc 5.143 (5.143)  LR 3.364e-05  Loss 5.192 (5.192)  Mem 15454
[05/04 00:38:26.025]: Epoch: [3/40][ 100/2052]  Lx 0.030 (0.028)  Lu 0.025 (0.023)  Lc 5.124 (5.154)  LR 3.449e-05  Loss 5.179 (5.205)  Mem 15454
[05/04 00:39:29.748]: Epoch: [3/40][ 200/2052]  Lx 0.031 (0.027)  Lu 0.026 (0.024)  Lc 5.140 (5.144)  LR 3.535e-05  Loss 5.198 (5.195)  Mem 15454
[05/04 00:40:32.636]: Epoch: [3/40][ 300/2052]  Lx 0.024 (0.027)  Lu 0.027 (0.024)  Lc 5.128 (5.145)  LR 3.622e-05  Loss 5.179 (5.196)  Mem 15454
[05/04 00:41:35.610]: Epoch: [3/40][ 400/2052]  Lx 0.030 (0.027)  Lu 0.025 (0.024)  Lc 5.141 (5.145)  LR 3.709e-05  Loss 5.195 (5.196)  Mem 15454
[05/04 00:42:38.840]: Epoch: [3/40][ 500/2052]  Lx 0.037 (0.028)  Lu 0.031 (0.024)  Lc 5.157 (5.147)  LR 3.796e-05  Loss 5.225 (5.199)  Mem 15454
[05/04 00:43:41.826]: Epoch: [3/40][ 600/2052]  Lx 0.035 (0.028)  Lu 0.030 (0.024)  Lc 5.134 (5.149)  LR 3.885e-05  Loss 5.199 (5.200)  Mem 15454
[05/04 00:44:45.048]: Epoch: [3/40][ 700/2052]  Lx 0.026 (0.028)  Lu 0.022 (0.024)  Lc 5.151 (5.149)  LR 3.973e-05  Loss 5.199 (5.200)  Mem 15454
[05/04 00:45:47.447]: Epoch: [3/40][ 800/2052]  Lx 0.026 (0.028)  Lu 0.027 (0.024)  Lc 5.148 (5.149)  LR 4.062e-05  Loss 5.201 (5.201)  Mem 15454
[05/04 00:46:49.831]: Epoch: [3/40][ 900/2052]  Lx 0.029 (0.028)  Lu 0.026 (0.024)  Lc 5.180 (5.148)  LR 4.152e-05  Loss 5.235 (5.200)  Mem 15454
[05/04 00:47:53.054]: Epoch: [3/40][1000/2052]  Lx 0.024 (0.028)  Lu 0.021 (0.024)  Lc 5.143 (5.148)  LR 4.241e-05  Loss 5.188 (5.200)  Mem 15454
[05/04 00:48:56.334]: Epoch: [3/40][1100/2052]  Lx 0.026 (0.028)  Lu 0.018 (0.024)  Lc 5.160 (5.148)  LR 4.332e-05  Loss 5.205 (5.200)  Mem 15454
[05/04 00:49:59.085]: Epoch: [3/40][1200/2052]  Lx 0.031 (0.028)  Lu 0.025 (0.024)  Lc 5.136 (5.148)  LR 4.422e-05  Loss 5.192 (5.200)  Mem 15454
[05/04 00:51:02.574]: Epoch: [3/40][1300/2052]  Lx 0.026 (0.028)  Lu 0.026 (0.024)  Lc 5.125 (5.147)  LR 4.513e-05  Loss 5.176 (5.199)  Mem 15454
[05/04 00:52:05.991]: Epoch: [3/40][1400/2052]  Lx 0.026 (0.028)  Lu 0.023 (0.024)  Lc 5.133 (5.146)  LR 4.604e-05  Loss 5.181 (5.199)  Mem 15454
[05/04 00:53:09.454]: Epoch: [3/40][1500/2052]  Lx 0.024 (0.028)  Lu 0.027 (0.024)  Lc 5.156 (5.145)  LR 4.695e-05  Loss 5.206 (5.198)  Mem 15454
[05/04 00:54:12.946]: Epoch: [3/40][1600/2052]  Lx 0.026 (0.028)  Lu 0.020 (0.024)  Lc 5.161 (5.145)  LR 4.787e-05  Loss 5.207 (5.198)  Mem 15454
[05/04 00:55:16.441]: Epoch: [3/40][1700/2052]  Lx 0.024 (0.028)  Lu 0.021 (0.024)  Lc 5.158 (5.145)  LR 4.878e-05  Loss 5.203 (5.198)  Mem 15454
[05/04 00:56:19.607]: Epoch: [3/40][1800/2052]  Lx 0.036 (0.029)  Lu 0.037 (0.025)  Lc 5.142 (5.145)  LR 4.970e-05  Loss 5.216 (5.198)  Mem 15454
[05/04 00:57:23.287]: Epoch: [3/40][1900/2052]  Lx 0.040 (0.029)  Lu 0.036 (0.025)  Lc 5.103 (5.144)  LR 5.062e-05  Loss 5.180 (5.198)  Mem 15454
[05/04 00:58:26.476]: Epoch: [3/40][2000/2052]  Lx 0.033 (0.029)  Lu 0.029 (0.025)  Lc 5.126 (5.144)  LR 5.154e-05  Loss 5.188 (5.197)  Mem 15454
[05/04 00:59:00.424]: Test: [  0/513]  Time 1.864 (1.864)  Mem 15454
[05/04 00:59:13.983]: Test: [100/513]  Time 0.135 (0.153)  Mem 15454
[05/04 00:59:27.545]: Test: [200/513]  Time 0.135 (0.144)  Mem 15454
[05/04 00:59:41.056]: Test: [300/513]  Time 0.135 (0.141)  Mem 15454
[05/04 00:59:54.651]: Test: [400/513]  Time 0.138 (0.140)  Mem 15454
[05/04 01:00:08.224]: Test: [500/513]  Time 0.135 (0.139)  Mem 15454
[05/04 01:00:10.552]:   mAP: 69.4959218594899
[05/04 01:00:12.642]: Test: [  0/513]  Time 2.087 (2.087)  Mem 15454
[05/04 01:00:26.186]: Test: [100/513]  Time 0.135 (0.155)  Mem 15454
[05/04 01:00:39.744]: Test: [200/513]  Time 0.135 (0.145)  Mem 15454
[05/04 01:00:53.329]: Test: [300/513]  Time 0.135 (0.142)  Mem 15454
[05/04 01:01:06.860]: Test: [400/513]  Time 0.135 (0.140)  Mem 15454
[05/04 01:01:20.430]: Test: [500/513]  Time 0.135 (0.139)  Mem 15454
[05/04 01:01:22.743]:   mAP: 75.76406991738833
[05/04 01:01:22.745]: => Test Epoch: [ 3/40]  ETA 15:49:46  TT 0:27:08 (1:45:31)  mAP 69.49592  mAP_ema 75.76407
[05/04 01:01:22.748]: 3 | Set best mAP 75.78817195951413 in ep 2
[05/04 01:01:22.748]:    | best regular mAP 75.15849566274714 in ep 0
[05/04 01:01:22.748]:    | best test mAP 75.5696857912416 
[05/04 01:01:22.850]: lr:5.20045932515121e-05
[05/04 01:01:26.734]: Epoch: [4/40][   0/2052]  Lx 0.030 (0.030)  Lu 0.034 (0.034)  Lc 5.147 (5.147)  LR 5.201e-05  Loss 5.211 (5.211)  Mem 15454
[05/04 01:02:31.680]: Epoch: [4/40][ 100/2052]  Lx 0.033 (0.028)  Lu 0.022 (0.025)  Lc 5.136 (5.132)  LR 5.293e-05  Loss 5.191 (5.185)  Mem 15454
[05/04 01:03:35.712]: Epoch: [4/40][ 200/2052]  Lx 0.028 (0.028)  Lu 0.028 (0.025)  Lc 5.138 (5.133)  LR 5.385e-05  Loss 5.194 (5.186)  Mem 15454
[05/04 01:04:39.914]: Epoch: [4/40][ 300/2052]  Lx 0.026 (0.028)  Lu 0.027 (0.025)  Lc 5.141 (5.133)  LR 5.477e-05  Loss 5.193 (5.186)  Mem 15454
[05/04 01:05:45.599]: Epoch: [4/40][ 400/2052]  Lx 0.028 (0.028)  Lu 0.031 (0.025)  Lc 5.110 (5.134)  LR 5.568e-05  Loss 5.169 (5.187)  Mem 15454
[05/04 01:06:50.102]: Epoch: [4/40][ 500/2052]  Lx 0.031 (0.028)  Lu 0.027 (0.025)  Lc 5.112 (5.133)  LR 5.660e-05  Loss 5.169 (5.186)  Mem 15454
[05/04 01:07:54.509]: Epoch: [4/40][ 600/2052]  Lx 0.027 (0.028)  Lu 0.023 (0.025)  Lc 5.141 (5.130)  LR 5.751e-05  Loss 5.192 (5.184)  Mem 15454
[05/04 01:08:58.491]: Epoch: [4/40][ 700/2052]  Lx 0.031 (0.028)  Lu 0.028 (0.025)  Lc 5.133 (5.132)  LR 5.842e-05  Loss 5.192 (5.185)  Mem 15454
[05/04 01:10:01.650]: Epoch: [4/40][ 800/2052]  Lx 0.025 (0.029)  Lu 0.021 (0.025)  Lc 5.143 (5.131)  LR 5.933e-05  Loss 5.189 (5.185)  Mem 15454
[05/04 01:11:04.788]: Epoch: [4/40][ 900/2052]  Lx 0.027 (0.029)  Lu 0.020 (0.025)  Lc 5.137 (5.130)  LR 6.024e-05  Loss 5.184 (5.184)  Mem 15454
[05/04 01:12:07.661]: Epoch: [4/40][1000/2052]  Lx 0.034 (0.029)  Lu 0.029 (0.026)  Lc 5.169 (5.130)  LR 6.114e-05  Loss 5.232 (5.185)  Mem 15454
[05/04 01:13:11.231]: Epoch: [4/40][1100/2052]  Lx 0.027 (0.029)  Lu 0.019 (0.026)  Lc 5.177 (5.131)  LR 6.204e-05  Loss 5.223 (5.186)  Mem 15454
[05/04 01:14:14.255]: Epoch: [4/40][1200/2052]  Lx 0.036 (0.029)  Lu 0.033 (0.026)  Lc 5.126 (5.131)  LR 6.294e-05  Loss 5.195 (5.186)  Mem 15454
[05/04 01:15:17.298]: Epoch: [4/40][1300/2052]  Lx 0.027 (0.029)  Lu 0.025 (0.026)  Lc 5.138 (5.131)  LR 6.383e-05  Loss 5.190 (5.186)  Mem 15454
[05/04 01:16:20.207]: Epoch: [4/40][1400/2052]  Lx 0.033 (0.029)  Lu 0.025 (0.026)  Lc 5.121 (5.131)  LR 6.472e-05  Loss 5.179 (5.186)  Mem 15454
[05/04 01:17:23.161]: Epoch: [4/40][1500/2052]  Lx 0.030 (0.029)  Lu 0.025 (0.026)  Lc 5.127 (5.131)  LR 6.560e-05  Loss 5.181 (5.186)  Mem 15454
[05/04 01:18:26.004]: Epoch: [4/40][1600/2052]  Lx 0.028 (0.030)  Lu 0.027 (0.026)  Lc 5.145 (5.130)  LR 6.648e-05  Loss 5.200 (5.186)  Mem 15454
[05/04 01:19:30.095]: Epoch: [4/40][1700/2052]  Lx 0.032 (0.030)  Lu 0.031 (0.026)  Lc 5.122 (5.130)  LR 6.736e-05  Loss 5.185 (5.186)  Mem 15454
[05/04 01:20:34.031]: Epoch: [4/40][1800/2052]  Lx 0.037 (0.030)  Lu 0.034 (0.026)  Lc 5.147 (5.130)  LR 6.822e-05  Loss 5.218 (5.186)  Mem 15454
[05/04 01:21:38.146]: Epoch: [4/40][1900/2052]  Lx 0.027 (0.030)  Lu 0.029 (0.026)  Lc 5.143 (5.130)  LR 6.909e-05  Loss 5.198 (5.186)  Mem 15454
[05/04 01:22:42.396]: Epoch: [4/40][2000/2052]  Lx 0.032 (0.030)  Lu 0.030 (0.026)  Lc 5.146 (5.130)  LR 6.994e-05  Loss 5.208 (5.186)  Mem 15454
[05/04 01:23:16.875]: Test: [  0/513]  Time 2.300 (2.300)  Mem 15454
[05/04 01:23:30.487]: Test: [100/513]  Time 0.135 (0.158)  Mem 15454
[05/04 01:23:44.070]: Test: [200/513]  Time 0.134 (0.147)  Mem 15454
[05/04 01:23:57.678]: Test: [300/513]  Time 0.135 (0.143)  Mem 15454
[05/04 01:24:11.264]: Test: [400/513]  Time 0.135 (0.141)  Mem 15454
[05/04 01:24:24.845]: Test: [500/513]  Time 0.134 (0.140)  Mem 15454
[05/04 01:24:27.171]:   mAP: 68.28755698685023
[05/04 01:24:29.014]: Test: [  0/513]  Time 1.840 (1.840)  Mem 15454
[05/04 01:24:42.584]: Test: [100/513]  Time 0.135 (0.153)  Mem 15454
[05/04 01:24:56.214]: Test: [200/513]  Time 0.135 (0.144)  Mem 15454
[05/04 01:25:09.848]: Test: [300/513]  Time 0.135 (0.142)  Mem 15454
[05/04 01:25:23.397]: Test: [400/513]  Time 0.135 (0.140)  Mem 15454
[05/04 01:25:37.093]: Test: [500/513]  Time 0.135 (0.140)  Mem 15454
[05/04 01:25:39.437]:   mAP: 75.39993469837314
[05/04 01:25:39.439]: => Test Epoch: [ 4/40]  ETA 15:08:39  TT 0:24:16 (2:09:48)  mAP 68.28756  mAP_ema 75.39993
[05/04 01:25:39.442]: 4 | Set best mAP 75.78817195951413 in ep 2
[05/04 01:25:39.442]:    | best regular mAP 75.15849566274714 in ep 0
[05/04 01:25:39.442]:    | best test mAP 75.5696857912416 
[05/04 01:25:39.548]: lr:7.037410913593423e-05
[05/04 01:25:43.061]: Epoch: [5/40][   0/2052]  Lx 0.030 (0.030)  Lu 0.022 (0.022)  Lc 5.147 (5.147)  LR 7.038e-05  Loss 5.198 (5.198)  Mem 15454
[05/04 01:26:47.397]: Epoch: [5/40][ 100/2052]  Lx 0.027 (0.029)  Lu 0.024 (0.026)  Lc 5.111 (5.141)  LR 7.123e-05  Loss 5.162 (5.196)  Mem 15454
[05/04 01:27:50.218]: Epoch: [5/40][ 200/2052]  Lx 0.027 (0.029)  Lu 0.024 (0.027)  Lc 5.128 (5.134)  LR 7.207e-05  Loss 5.179 (5.190)  Mem 15454
[05/04 01:28:53.208]: Epoch: [5/40][ 300/2052]  Lx 0.025 (0.029)  Lu 0.023 (0.027)  Lc 5.139 (5.128)  LR 7.290e-05  Loss 5.188 (5.184)  Mem 15454
[05/04 01:29:55.540]: Epoch: [5/40][ 400/2052]  Lx 0.034 (0.029)  Lu 0.033 (0.026)  Lc 5.105 (5.128)  LR 7.372e-05  Loss 5.172 (5.184)  Mem 15454
[05/04 01:30:58.251]: Epoch: [5/40][ 500/2052]  Lx 0.028 (0.030)  Lu 0.028 (0.026)  Lc 5.154 (5.128)  LR 7.454e-05  Loss 5.211 (5.184)  Mem 15454
[05/04 01:32:01.074]: Epoch: [5/40][ 600/2052]  Lx 0.044 (0.029)  Lu 0.038 (0.026)  Lc 5.118 (5.128)  LR 7.534e-05  Loss 5.200 (5.184)  Mem 15454
[05/04 01:33:04.004]: Epoch: [5/40][ 700/2052]  Lx 0.032 (0.030)  Lu 0.028 (0.026)  Lc 5.135 (5.127)  LR 7.614e-05  Loss 5.195 (5.183)  Mem 15454
[05/04 01:34:07.011]: Epoch: [5/40][ 800/2052]  Lx 0.031 (0.030)  Lu 0.026 (0.027)  Lc 5.113 (5.126)  LR 7.693e-05  Loss 5.170 (5.183)  Mem 15454
[05/04 01:35:09.956]: Epoch: [5/40][ 900/2052]  Lx 0.037 (0.030)  Lu 0.029 (0.027)  Lc 5.127 (5.126)  LR 7.771e-05  Loss 5.193 (5.183)  Mem 15454
[05/04 01:36:13.144]: Epoch: [5/40][1000/2052]  Lx 0.040 (0.030)  Lu 0.041 (0.027)  Lc 5.124 (5.127)  LR 7.848e-05  Loss 5.206 (5.184)  Mem 15454
[05/04 01:37:16.269]: Epoch: [5/40][1100/2052]  Lx 0.030 (0.030)  Lu 0.026 (0.027)  Lc 5.162 (5.127)  LR 7.924e-05  Loss 5.219 (5.184)  Mem 15454
[05/04 01:38:19.000]: Epoch: [5/40][1200/2052]  Lx 0.036 (0.030)  Lu 0.041 (0.027)  Lc 5.121 (5.127)  LR 7.999e-05  Loss 5.198 (5.184)  Mem 15454
[05/04 01:39:21.975]: Epoch: [5/40][1300/2052]  Lx 0.036 (0.030)  Lu 0.030 (0.027)  Lc 5.126 (5.126)  LR 8.074e-05  Loss 5.192 (5.184)  Mem 15454
[05/04 01:40:24.987]: Epoch: [5/40][1400/2052]  Lx 0.024 (0.030)  Lu 0.021 (0.027)  Lc 5.155 (5.126)  LR 8.147e-05  Loss 5.199 (5.184)  Mem 15454
[05/04 01:41:28.296]: Epoch: [5/40][1500/2052]  Lx 0.044 (0.031)  Lu 0.042 (0.027)  Lc 5.141 (5.127)  LR 8.219e-05  Loss 5.227 (5.185)  Mem 15454
[05/04 01:42:31.429]: Epoch: [5/40][1600/2052]  Lx 0.032 (0.031)  Lu 0.032 (0.027)  Lc 5.121 (5.127)  LR 8.289e-05  Loss 5.185 (5.185)  Mem 15454
[05/04 01:43:35.329]: Epoch: [5/40][1700/2052]  Lx 0.039 (0.031)  Lu 0.034 (0.027)  Lc 5.150 (5.127)  LR 8.359e-05  Loss 5.223 (5.185)  Mem 15454
[05/04 01:44:38.694]: Epoch: [5/40][1800/2052]  Lx 0.034 (0.031)  Lu 0.033 (0.028)  Lc 5.152 (5.127)  LR 8.428e-05  Loss 5.220 (5.186)  Mem 15454
[05/04 01:45:41.719]: Epoch: [5/40][1900/2052]  Lx 0.035 (0.031)  Lu 0.029 (0.028)  Lc 5.140 (5.128)  LR 8.495e-05  Loss 5.203 (5.187)  Mem 15454
[05/04 01:46:44.845]: Epoch: [5/40][2000/2052]  Lx 0.039 (0.031)  Lu 0.037 (0.028)  Lc 5.136 (5.130)  LR 8.561e-05  Loss 5.213 (5.189)  Mem 15454
[05/04 01:47:19.497]: Test: [  0/513]  Time 2.384 (2.384)  Mem 15454
[05/04 01:47:33.124]: Test: [100/513]  Time 0.135 (0.159)  Mem 15454
[05/04 01:47:46.669]: Test: [200/513]  Time 0.135 (0.147)  Mem 15454
[05/04 01:48:00.301]: Test: [300/513]  Time 0.135 (0.143)  Mem 15454
[05/04 01:48:13.895]: Test: [400/513]  Time 0.135 (0.142)  Mem 15454
[05/04 01:48:27.490]: Test: [500/513]  Time 0.135 (0.140)  Mem 15454
[05/04 01:48:29.820]:   mAP: 65.31530706116828
[05/04 01:48:31.851]: Test: [  0/513]  Time 2.029 (2.029)  Mem 15454
[05/04 01:48:45.541]: Test: [100/513]  Time 0.135 (0.156)  Mem 15454
[05/04 01:48:59.147]: Test: [200/513]  Time 0.135 (0.146)  Mem 15454
[05/04 01:49:12.776]: Test: [300/513]  Time 0.135 (0.143)  Mem 15454
[05/04 01:49:26.318]: Test: [400/513]  Time 0.135 (0.141)  Mem 15454
[05/04 01:49:39.895]: Test: [500/513]  Time 0.135 (0.140)  Mem 15454
[05/04 01:49:42.271]:   mAP: 74.58010425938035
[05/04 01:49:42.273]: => Test Epoch: [ 5/40]  ETA 14:31:51  TT 0:24:02 (2:33:51)  mAP 65.31531  mAP_ema 74.58010
[05/04 01:49:42.277]: 5 | Set best mAP 75.78817195951413 in ep 2
[05/04 01:49:42.277]:    | best regular mAP 75.15849566274714 in ep 0
[05/04 01:49:42.277]:    | best test mAP 75.5696857912416 
[05/04 01:49:42.277]: epoch - best_epoch = 3, stop!