search

visionml / pytracking

Visual tracking library based on PyTorch.
GNU General Public License v3.0
3.26k stars 608 forks source link

TaMOs and LaSOT dataset #434

Open yhifny opened 5 months ago

yhifny commented 5 months ago

Hi,

We downloaded the LaSOT dataset and tried running TaMOs but got an error. We suspect there is a preprocessing step that we missed in our setup. Can you please share your insights about our problem?

Starting training
Training:  tamos  tamos_swin_base
1

load pretrain weight from /home/hlwn032u1/data/sot/networks/swin_base_patch4_window12_384_22k.pth 

using lin_sine as positional encoding
/home/hlwn032u1/data/sot/ltr/checkpoints/checkpoints ltr/tamos/tamos_swin_base TaMOsActor TaMOsNet
No matching checkpoint file found
[train: 1, 10 / 40000] FPS: 0.5 (0.7)  ,  mIoU_trafo: 0.11744  ,  mIoU_lowres: 0.12181  ,  mIoU_highres: 0.08804  ,  Loss/weighted_GIoU_highres: 0.92397  ,  mIoU_pred_center_trafo: 0.00000  ,  mIoU_pred_center_lowres: 0.00000  ,  mIoU_pred_center_highres: 0.03388  ,  Loss/total: 344.87133  ,  Loss/GIoU: 2.72229  ,  Loss/weighted_GIoU: 2.72229  ,  Loss/clf_loss_test: 3.42149  ,  Loss/weighted_clf_loss_test: 342.14904
[train: 1, 20 / 40000] FPS: 0.6 (0.7)  ,  mIoU_trafo: 0.10210  ,  mIoU_lowres: 0.11153  ,  mIoU_highres: 0.08445  ,  Loss/weighted_GIoU_highres: 0.96119  ,  mIoU_pred_center_trafo: 0.01375  ,  mIoU_pred_center_lowres: 0.00000  ,  mIoU_pred_center_highres: 0.03472  ,  Loss/total: 215.87651  ,  Loss/GIoU: 2.88734  ,  Loss/weighted_GIoU: 2.88734  ,  Loss/clf_loss_test: 2.12989  ,  Loss/weighted_clf_loss_test: 212.98917
[train: 1, 30 / 40000] FPS: 0.6 (0.7)  ,  mIoU_trafo: 0.11525  ,  mIoU_lowres: 0.13144  ,  mIoU_highres: 0.08382  ,  Loss/weighted_GIoU_highres: 0.94687  ,  mIoU_pred_center_trafo: 0.00917  ,  mIoU_pred_center_lowres: 0.00000  ,  mIoU_pred_center_highres: 0.02428  ,  Loss/total: 167.59891  ,  Loss/GIoU: 2.80602  ,  Loss/weighted_GIoU: 2.80602  ,  Loss/clf_loss_test: 1.64793  ,  Loss/weighted_clf_loss_test: 164.79289
[train: 1, 40 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.14153  ,  mIoU_lowres: 0.16058  ,  mIoU_highres: 0.10449  ,  Loss/weighted_GIoU_highres: 0.92667  ,  mIoU_pred_center_trafo: 0.00688  ,  mIoU_pred_center_lowres: 0.00796  ,  mIoU_pred_center_highres: 0.03925  ,  Loss/total: 136.65977  ,  Loss/GIoU: 2.72356  ,  Loss/weighted_GIoU: 2.72356  ,  Loss/clf_loss_test: 1.33936  ,  Loss/weighted_clf_loss_test: 133.93621
[train: 1, 50 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.14736  ,  mIoU_lowres: 0.16909  ,  mIoU_highres: 0.11693  ,  Loss/weighted_GIoU_highres: 0.91834  ,  mIoU_pred_center_trafo: 0.00730  ,  mIoU_pred_center_lowres: 0.00975  ,  mIoU_pred_center_highres: 0.04326  ,  Loss/total: 156.87429  ,  Loss/GIoU: 2.70095  ,  Loss/weighted_GIoU: 2.70095  ,  Loss/clf_loss_test: 1.54173  ,  Loss/weighted_clf_loss_test: 154.17334
[train: 1, 60 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.18547  ,  mIoU_lowres: 0.19778  ,  mIoU_highres: 0.15032  ,  Loss/weighted_GIoU_highres: 0.88010  ,  mIoU_pred_center_trafo: 0.00877  ,  mIoU_pred_center_lowres: 0.01533  ,  mIoU_pred_center_highres: 0.04295  ,  Loss/total: 176.70240  ,  Loss/GIoU: 2.58258  ,  Loss/weighted_GIoU: 2.58258  ,  Loss/clf_loss_test: 1.74120  ,  Loss/weighted_clf_loss_test: 174.11982
[train: 1, 70 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.18920  ,  mIoU_lowres: 0.20022  ,  mIoU_highres: 0.16142  ,  Loss/weighted_GIoU_highres: 0.87214  ,  mIoU_pred_center_trafo: 0.00836  ,  mIoU_pred_center_lowres: 0.01314  ,  mIoU_pred_center_highres: 0.04651  ,  Loss/total: 157.44407  ,  Loss/GIoU: 2.57309  ,  Loss/weighted_GIoU: 2.57309  ,  Loss/clf_loss_test: 1.54871  ,  Loss/weighted_clf_loss_test: 154.87098
[train: 1, 80 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.19302  ,  mIoU_lowres: 0.20383  ,  mIoU_highres: 0.17141  ,  Loss/weighted_GIoU_highres: 0.86026  ,  mIoU_pred_center_trafo: 0.01167  ,  mIoU_pred_center_lowres: 0.01150  ,  mIoU_pred_center_highres: 0.04984  ,  Loss/total: 140.48050  ,  Loss/GIoU: 2.54931  ,  Loss/weighted_GIoU: 2.54931  ,  Loss/clf_loss_test: 1.37931  ,  Loss/weighted_clf_loss_test: 137.93120
[train: 1, 90 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.21248  ,  mIoU_lowres: 0.22378  ,  mIoU_highres: 0.19850  ,  Loss/weighted_GIoU_highres: 0.83098  ,  mIoU_pred_center_trafo: 0.01037  ,  mIoU_pred_center_lowres: 0.01022  ,  mIoU_pred_center_highres: 0.04980  ,  Loss/total: 127.72065  ,  Loss/GIoU: 2.47417  ,  Loss/weighted_GIoU: 2.47417  ,  Loss/clf_loss_test: 1.25246  ,  Loss/weighted_clf_loss_test: 125.24648
[train: 1, 100 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.21693  ,  mIoU_lowres: 0.22658  ,  mIoU_highres: 0.20177  ,  Loss/weighted_GIoU_highres: 0.82483  ,  mIoU_pred_center_trafo: 0.01076  ,  mIoU_pred_center_lowres: 0.00920  ,  mIoU_pred_center_highres: 0.04601  ,  Loss/total: 121.59675  ,  Loss/GIoU: 2.45419  ,  Loss/weighted_GIoU: 2.45419  ,  Loss/clf_loss_test: 1.19143  ,  Loss/weighted_clf_loss_test: 119.14256
[train: 1, 110 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.21536  ,  mIoU_lowres: 0.22423  ,  mIoU_highres: 0.19978  ,  Loss/weighted_GIoU_highres: 0.82467  ,  mIoU_pred_center_trafo: 0.00978  ,  mIoU_pred_center_lowres: 0.00836  ,  mIoU_pred_center_highres: 0.04618  ,  Loss/total: 119.99652  ,  Loss/GIoU: 2.45523  ,  Loss/weighted_GIoU: 2.45523  ,  Loss/clf_loss_test: 1.17541  ,  Loss/weighted_clf_loss_test: 117.54129
[train: 1, 120 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.21265  ,  mIoU_lowres: 0.22053  ,  mIoU_highres: 0.19696  ,  Loss/weighted_GIoU_highres: 0.82599  ,  mIoU_pred_center_trafo: 0.00981  ,  mIoU_pred_center_lowres: 0.00766  ,  mIoU_pred_center_highres: 0.04447  ,  Loss/total: 122.49688  ,  Loss/GIoU: 2.45924  ,  Loss/weighted_GIoU: 2.45924  ,  Loss/clf_loss_test: 1.20038  ,  Loss/weighted_clf_loss_test: 120.03764
[train: 1, 130 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.21956  ,  mIoU_lowres: 0.22818  ,  mIoU_highres: 0.20570  ,  Loss/weighted_GIoU_highres: 0.81725  ,  mIoU_pred_center_trafo: 0.01182  ,  mIoU_pred_center_lowres: 0.00708  ,  mIoU_pred_center_highres: 0.04499  ,  Loss/total: 117.52087  ,  Loss/GIoU: 2.43504  ,  Loss/weighted_GIoU: 2.43504  ,  Loss/clf_loss_test: 1.15086  ,  Loss/weighted_clf_loss_test: 115.08583
[train: 1, 140 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.22215  ,  mIoU_lowres: 0.23025  ,  mIoU_highres: 0.20849  ,  Loss/weighted_GIoU_highres: 0.81512  ,  mIoU_pred_center_trafo: 0.01097  ,  mIoU_pred_center_lowres: 0.00657  ,  mIoU_pred_center_highres: 0.04709  ,  Loss/total: 120.25060  ,  Loss/GIoU: 2.42646  ,  Loss/weighted_GIoU: 2.42646  ,  Loss/clf_loss_test: 1.17824  ,  Loss/weighted_clf_loss_test: 117.82413
[train: 1, 150 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.22341  ,  mIoU_lowres: 0.23127  ,  mIoU_highres: 0.21045  ,  Loss/weighted_GIoU_highres: 0.81602  ,  mIoU_pred_center_trafo: 0.01218  ,  mIoU_pred_center_lowres: 0.00752  ,  mIoU_pred_center_highres: 0.04658  ,  Loss/total: 115.36762  ,  Loss/GIoU: 2.42994  ,  Loss/weighted_GIoU: 2.42994  ,  Loss/clf_loss_test: 1.12938  ,  Loss/weighted_clf_loss_test: 112.93769
[train: 1, 160 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.23187  ,  mIoU_lowres: 0.23909  ,  mIoU_highres: 0.21929  ,  Loss/weighted_GIoU_highres: 0.80626  ,  mIoU_pred_center_trafo: 0.01142  ,  mIoU_pred_center_lowres: 0.00705  ,  mIoU_pred_center_highres: 0.04817  ,  Loss/total: 109.95185  ,  Loss/GIoU: 2.40073  ,  Loss/weighted_GIoU: 2.40073  ,  Loss/clf_loss_test: 1.07551  ,  Loss/weighted_clf_loss_test: 107.55111
[train: 1, 170 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.23421  ,  mIoU_lowres: 0.24134  ,  mIoU_highres: 0.22205  ,  Loss/weighted_GIoU_highres: 0.80456  ,  mIoU_pred_center_trafo: 0.01270  ,  mIoU_pred_center_lowres: 0.00664  ,  mIoU_pred_center_highres: 0.05224  ,  Loss/total: 111.69269  ,  Loss/GIoU: 2.39585  ,  Loss/weighted_GIoU: 2.39585  ,  Loss/clf_loss_test: 1.09297  ,  Loss/weighted_clf_loss_test: 109.29683
[train: 1, 180 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.23746  ,  mIoU_lowres: 0.24447  ,  mIoU_highres: 0.22650  ,  Loss/weighted_GIoU_highres: 0.80147  ,  mIoU_pred_center_trafo: 0.01417  ,  mIoU_pred_center_lowres: 0.00627  ,  mIoU_pred_center_highres: 0.05119  ,  Loss/total: 106.57240  ,  Loss/GIoU: 2.38865  ,  Loss/weighted_GIoU: 2.38865  ,  Loss/clf_loss_test: 1.04184  ,  Loss/weighted_clf_loss_test: 104.18375
[train: 1, 190 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24700  ,  mIoU_lowres: 0.25331  ,  mIoU_highres: 0.23555  ,  Loss/weighted_GIoU_highres: 0.79175  ,  mIoU_pred_center_trafo: 0.01343  ,  mIoU_pred_center_lowres: 0.00738  ,  mIoU_pred_center_highres: 0.05130  ,  Loss/total: 107.72461  ,  Loss/GIoU: 2.35877  ,  Loss/weighted_GIoU: 2.35877  ,  Loss/clf_loss_test: 1.05366  ,  Loss/weighted_clf_loss_test: 105.36584
[train: 1, 200 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24425  ,  mIoU_lowres: 0.25039  ,  mIoU_highres: 0.23412  ,  Loss/weighted_GIoU_highres: 0.79468  ,  mIoU_pred_center_trafo: 0.01887  ,  mIoU_pred_center_lowres: 0.00933  ,  mIoU_pred_center_highres: 0.05156  ,  Loss/total: 111.15322  ,  Loss/GIoU: 2.36984  ,  Loss/weighted_GIoU: 2.36984  ,  Loss/clf_loss_test: 1.08783  ,  Loss/weighted_clf_loss_test: 108.78338
[train: 1, 210 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.23863  ,  mIoU_lowres: 0.24440  ,  mIoU_highres: 0.22907  ,  Loss/weighted_GIoU_highres: 0.80119  ,  mIoU_pred_center_trafo: 0.01859  ,  mIoU_pred_center_lowres: 0.00889  ,  mIoU_pred_center_highres: 0.04911  ,  Loss/total: 107.38565  ,  Loss/GIoU: 2.38845  ,  Loss/weighted_GIoU: 2.38845  ,  Loss/clf_loss_test: 1.04997  ,  Loss/weighted_clf_loss_test: 104.99720
[train: 1, 220 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.23370  ,  mIoU_lowres: 0.23879  ,  mIoU_highres: 0.22380  ,  Loss/weighted_GIoU_highres: 0.80525  ,  mIoU_pred_center_trafo: 0.01895  ,  mIoU_pred_center_lowres: 0.00848  ,  mIoU_pred_center_highres: 0.04827  ,  Loss/total: 103.38182  ,  Loss/GIoU: 2.40039  ,  Loss/weighted_GIoU: 2.40039  ,  Loss/clf_loss_test: 1.00981  ,  Loss/weighted_clf_loss_test: 100.98143
[train: 1, 230 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.23433  ,  mIoU_lowres: 0.23933  ,  mIoU_highres: 0.22520  ,  Loss/weighted_GIoU_highres: 0.80346  ,  mIoU_pred_center_trafo: 0.02021  ,  mIoU_pred_center_lowres: 0.01039  ,  mIoU_pred_center_highres: 0.04966  ,  Loss/total: 99.82412  ,  Loss/GIoU: 2.39609  ,  Loss/weighted_GIoU: 2.39609  ,  Loss/clf_loss_test: 0.97428  ,  Loss/weighted_clf_loss_test: 97.42802
[train: 1, 240 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24162  ,  mIoU_lowres: 0.24609  ,  mIoU_highres: 0.23175  ,  Loss/weighted_GIoU_highres: 0.79951  ,  mIoU_pred_center_trafo: 0.02674  ,  mIoU_pred_center_lowres: 0.01485  ,  mIoU_pred_center_highres: 0.05349  ,  Loss/total: 97.74737  ,  Loss/GIoU: 2.38328  ,  Loss/weighted_GIoU: 2.38328  ,  Loss/clf_loss_test: 0.95364  ,  Loss/weighted_clf_loss_test: 95.36409
[train: 1, 250 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.23893  ,  mIoU_lowres: 0.24330  ,  mIoU_highres: 0.22969  ,  Loss/weighted_GIoU_highres: 0.80117  ,  mIoU_pred_center_trafo: 0.02910  ,  mIoU_pred_center_lowres: 0.01684  ,  mIoU_pred_center_highres: 0.05438  ,  Loss/total: 94.88330  ,  Loss/GIoU: 2.38910  ,  Loss/weighted_GIoU: 2.38910  ,  Loss/clf_loss_test: 0.92494  ,  Loss/weighted_clf_loss_test: 92.49421
[train: 1, 260 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24335  ,  mIoU_lowres: 0.24695  ,  mIoU_highres: 0.23312  ,  Loss/weighted_GIoU_highres: 0.79802  ,  mIoU_pred_center_trafo: 0.03361  ,  mIoU_pred_center_lowres: 0.02450  ,  mIoU_pred_center_highres: 0.05652  ,  Loss/total: 91.85940  ,  Loss/GIoU: 2.37816  ,  Loss/weighted_GIoU: 2.37816  ,  Loss/clf_loss_test: 0.89481  ,  Loss/weighted_clf_loss_test: 89.48123
[train: 1, 270 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24343  ,  mIoU_lowres: 0.24686  ,  mIoU_highres: 0.23371  ,  Loss/weighted_GIoU_highres: 0.79836  ,  mIoU_pred_center_trafo: 0.03435  ,  mIoU_pred_center_lowres: 0.02796  ,  mIoU_pred_center_highres: 0.05970  ,  Loss/total: 88.93356  ,  Loss/GIoU: 2.37938  ,  Loss/weighted_GIoU: 2.37938  ,  Loss/clf_loss_test: 0.86554  ,  Loss/weighted_clf_loss_test: 86.55418
[train: 1, 280 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24158  ,  mIoU_lowres: 0.24509  ,  mIoU_highres: 0.23193  ,  Loss/weighted_GIoU_highres: 0.79914  ,  mIoU_pred_center_trafo: 0.03508  ,  mIoU_pred_center_lowres: 0.03170  ,  mIoU_pred_center_highres: 0.06002  ,  Loss/total: 86.20370  ,  Loss/GIoU: 2.38142  ,  Loss/weighted_GIoU: 2.38142  ,  Loss/clf_loss_test: 0.83822  ,  Loss/weighted_clf_loss_test: 83.82228
[train: 1, 290 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24428  ,  mIoU_lowres: 0.24798  ,  mIoU_highres: 0.23507  ,  Loss/weighted_GIoU_highres: 0.79538  ,  mIoU_pred_center_trafo: 0.03768  ,  mIoU_pred_center_lowres: 0.03216  ,  mIoU_pred_center_highres: 0.05962  ,  Loss/total: 83.63693  ,  Loss/GIoU: 2.37065  ,  Loss/weighted_GIoU: 2.37065  ,  Loss/clf_loss_test: 0.81266  ,  Loss/weighted_clf_loss_test: 81.26628
[train: 1, 300 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24368  ,  mIoU_lowres: 0.24777  ,  mIoU_highres: 0.23542  ,  Loss/weighted_GIoU_highres: 0.79410  ,  mIoU_pred_center_trafo: 0.04254  ,  mIoU_pred_center_lowres: 0.03109  ,  mIoU_pred_center_highres: 0.05952  ,  Loss/total: 82.26040  ,  Loss/GIoU: 2.36817  ,  Loss/weighted_GIoU: 2.36817  ,  Loss/clf_loss_test: 0.79892  ,  Loss/weighted_clf_loss_test: 79.89222
[train: 1, 310 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24554  ,  mIoU_lowres: 0.24925  ,  mIoU_highres: 0.23775  ,  Loss/weighted_GIoU_highres: 0.79161  ,  mIoU_pred_center_trafo: 0.04527  ,  mIoU_pred_center_lowres: 0.03178  ,  mIoU_pred_center_highres: 0.06329  ,  Loss/total: 80.26747  ,  Loss/GIoU: 2.36201  ,  Loss/weighted_GIoU: 2.36201  ,  Loss/clf_loss_test: 0.77905  ,  Loss/weighted_clf_loss_test: 77.90546
[train: 1, 320 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24392  ,  mIoU_lowres: 0.24729  ,  mIoU_highres: 0.23603  ,  Loss/weighted_GIoU_highres: 0.79251  ,  mIoU_pred_center_trafo: 0.04645  ,  mIoU_pred_center_lowres: 0.03242  ,  mIoU_pred_center_highres: 0.06303  ,  Loss/total: 78.13199  ,  Loss/GIoU: 2.36469  ,  Loss/weighted_GIoU: 2.36469  ,  Loss/clf_loss_test: 0.75767  ,  Loss/weighted_clf_loss_test: 75.76730
[train: 1, 330 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24487  ,  mIoU_lowres: 0.24839  ,  mIoU_highres: 0.23754  ,  Loss/weighted_GIoU_highres: 0.79150  ,  mIoU_pred_center_trafo: 0.04850  ,  mIoU_pred_center_lowres: 0.03307  ,  mIoU_pred_center_highres: 0.06614  ,  Loss/total: 76.16906  ,  Loss/GIoU: 2.36185  ,  Loss/weighted_GIoU: 2.36185  ,  Loss/clf_loss_test: 0.73807  ,  Loss/weighted_clf_loss_test: 73.80721
[train: 1, 340 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24419  ,  mIoU_lowres: 0.24767  ,  mIoU_highres: 0.23641  ,  Loss/weighted_GIoU_highres: 0.79286  ,  mIoU_pred_center_trafo: 0.05082  ,  mIoU_pred_center_lowres: 0.03624  ,  mIoU_pred_center_highres: 0.06693  ,  Loss/total: 74.29162  ,  Loss/GIoU: 2.36433  ,  Loss/weighted_GIoU: 2.36433  ,  Loss/clf_loss_test: 0.71927  ,  Loss/weighted_clf_loss_test: 71.92729
[train: 1, 350 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24591  ,  mIoU_lowres: 0.24952  ,  mIoU_highres: 0.23845  ,  Loss/weighted_GIoU_highres: 0.79041  ,  mIoU_pred_center_trafo: 0.05419  ,  mIoU_pred_center_lowres: 0.03790  ,  mIoU_pred_center_highres: 0.06978  ,  Loss/total: 72.43766  ,  Loss/GIoU: 2.35758  ,  Loss/weighted_GIoU: 2.35758  ,  Loss/clf_loss_test: 0.70080  ,  Loss/weighted_clf_loss_test: 70.08008
[train: 1, 360 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.24911  ,  mIoU_lowres: 0.25262  ,  mIoU_highres: 0.24263  ,  Loss/weighted_GIoU_highres: 0.78665  ,  mIoU_pred_center_trafo: 0.05825  ,  mIoU_pred_center_lowres: 0.03898  ,  mIoU_pred_center_highres: 0.07092  ,  Loss/total: 70.87857  ,  Loss/GIoU: 2.34851  ,  Loss/weighted_GIoU: 2.34851  ,  Loss/clf_loss_test: 0.68530  ,  Loss/weighted_clf_loss_test: 68.53006
[train: 1, 370 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25037  ,  mIoU_lowres: 0.25345  ,  mIoU_highres: 0.24391  ,  Loss/weighted_GIoU_highres: 0.78622  ,  mIoU_pred_center_trafo: 0.06182  ,  mIoU_pred_center_lowres: 0.04206  ,  mIoU_pred_center_highres: 0.07219  ,  Loss/total: 69.31629  ,  Loss/GIoU: 2.34759  ,  Loss/weighted_GIoU: 2.34759  ,  Loss/clf_loss_test: 0.66969  ,  Loss/weighted_clf_loss_test: 66.96870
[train: 1, 380 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25139  ,  mIoU_lowres: 0.25437  ,  mIoU_highres: 0.24503  ,  Loss/weighted_GIoU_highres: 0.78461  ,  mIoU_pred_center_trafo: 0.06335  ,  mIoU_pred_center_lowres: 0.04465  ,  mIoU_pred_center_highres: 0.07391  ,  Loss/total: 67.71425  ,  Loss/GIoU: 2.34305  ,  Loss/weighted_GIoU: 2.34305  ,  Loss/clf_loss_test: 0.65371  ,  Loss/weighted_clf_loss_test: 65.37120
[train: 1, 390 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25193  ,  mIoU_lowres: 0.25394  ,  mIoU_highres: 0.24520  ,  Loss/weighted_GIoU_highres: 0.78391  ,  mIoU_pred_center_trafo: 0.06639  ,  mIoU_pred_center_lowres: 0.04612  ,  mIoU_pred_center_highres: 0.07391  ,  Loss/total: 66.79338  ,  Loss/GIoU: 2.34097  ,  Loss/weighted_GIoU: 2.34097  ,  Loss/clf_loss_test: 0.64452  ,  Loss/weighted_clf_loss_test: 64.45241
[train: 1, 400 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25293  ,  mIoU_lowres: 0.25387  ,  mIoU_highres: 0.24515  ,  Loss/weighted_GIoU_highres: 0.78426  ,  mIoU_pred_center_trafo: 0.07044  ,  mIoU_pred_center_lowres: 0.05081  ,  mIoU_pred_center_highres: 0.07665  ,  Loss/total: 65.35149  ,  Loss/GIoU: 2.34087  ,  Loss/weighted_GIoU: 2.34087  ,  Loss/clf_loss_test: 0.63011  ,  Loss/weighted_clf_loss_test: 63.01062
[train: 1, 410 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25331  ,  mIoU_lowres: 0.25060  ,  mIoU_highres: 0.24213  ,  Loss/weighted_GIoU_highres: 0.78708  ,  mIoU_pred_center_trafo: 0.07152  ,  mIoU_pred_center_lowres: 0.04993  ,  mIoU_pred_center_highres: 0.07539  ,  Loss/total: 64.02453  ,  Loss/GIoU: 2.34615  ,  Loss/weighted_GIoU: 2.34615  ,  Loss/clf_loss_test: 0.61678  ,  Loss/weighted_clf_loss_test: 61.67838
[train: 1, 420 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25500  ,  mIoU_lowres: 0.24891  ,  mIoU_highres: 0.24081  ,  Loss/weighted_GIoU_highres: 0.78813  ,  mIoU_pred_center_trafo: 0.07381  ,  mIoU_pred_center_lowres: 0.04929  ,  mIoU_pred_center_highres: 0.07397  ,  Loss/total: 62.81524  ,  Loss/GIoU: 2.34622  ,  Loss/weighted_GIoU: 2.34622  ,  Loss/clf_loss_test: 0.60469  ,  Loss/weighted_clf_loss_test: 60.46902
[train: 1, 430 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25448  ,  mIoU_lowres: 0.24662  ,  mIoU_highres: 0.23755  ,  Loss/weighted_GIoU_highres: 0.79176  ,  mIoU_pred_center_trafo: 0.07433  ,  mIoU_pred_center_lowres: 0.04914  ,  mIoU_pred_center_highres: 0.07243  ,  Loss/total: 61.93120  ,  Loss/GIoU: 2.35323  ,  Loss/weighted_GIoU: 2.35323  ,  Loss/clf_loss_test: 0.59578  ,  Loss/weighted_clf_loss_test: 59.57797
[train: 1, 440 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25466  ,  mIoU_lowres: 0.24518  ,  mIoU_highres: 0.23600  ,  Loss/weighted_GIoU_highres: 0.79302  ,  mIoU_pred_center_trafo: 0.07499  ,  mIoU_pred_center_lowres: 0.04823  ,  mIoU_pred_center_highres: 0.07202  ,  Loss/total: 60.94997  ,  Loss/GIoU: 2.35507  ,  Loss/weighted_GIoU: 2.35507  ,  Loss/clf_loss_test: 0.58595  ,  Loss/weighted_clf_loss_test: 58.59490
[train: 1, 450 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25518  ,  mIoU_lowres: 0.24566  ,  mIoU_highres: 0.23679  ,  Loss/weighted_GIoU_highres: 0.79214  ,  mIoU_pred_center_trafo: 0.07641  ,  mIoU_pred_center_lowres: 0.04874  ,  mIoU_pred_center_highres: 0.07319  ,  Loss/total: 59.89920  ,  Loss/GIoU: 2.35267  ,  Loss/weighted_GIoU: 2.35267  ,  Loss/clf_loss_test: 0.57547  ,  Loss/weighted_clf_loss_test: 57.54653
[train: 1, 460 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25614  ,  mIoU_lowres: 0.24672  ,  mIoU_highres: 0.23801  ,  Loss/weighted_GIoU_highres: 0.79056  ,  mIoU_pred_center_trafo: 0.07942  ,  mIoU_pred_center_lowres: 0.04878  ,  mIoU_pred_center_highres: 0.07359  ,  Loss/total: 59.02047  ,  Loss/GIoU: 2.34814  ,  Loss/weighted_GIoU: 2.34814  ,  Loss/clf_loss_test: 0.56672  ,  Loss/weighted_clf_loss_test: 56.67233
[train: 1, 470 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25512  ,  mIoU_lowres: 0.24587  ,  mIoU_highres: 0.23690  ,  Loss/weighted_GIoU_highres: 0.79189  ,  mIoU_pred_center_trafo: 0.07916  ,  mIoU_pred_center_lowres: 0.05001  ,  mIoU_pred_center_highres: 0.07346  ,  Loss/total: 57.96891  ,  Loss/GIoU: 2.35190  ,  Loss/weighted_GIoU: 2.35190  ,  Loss/clf_loss_test: 0.55617  ,  Loss/weighted_clf_loss_test: 55.61702
[train: 1, 480 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25856  ,  mIoU_lowres: 0.24966  ,  mIoU_highres: 0.24103  ,  Loss/weighted_GIoU_highres: 0.78789  ,  mIoU_pred_center_trafo: 0.08347  ,  mIoU_pred_center_lowres: 0.05361  ,  mIoU_pred_center_highres: 0.07690  ,  Loss/total: 57.06046  ,  Loss/GIoU: 2.34100  ,  Loss/weighted_GIoU: 2.34100  ,  Loss/clf_loss_test: 0.54719  ,  Loss/weighted_clf_loss_test: 54.71946
[train: 1, 490 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25993  ,  mIoU_lowres: 0.25129  ,  mIoU_highres: 0.24290  ,  Loss/weighted_GIoU_highres: 0.78635  ,  mIoU_pred_center_trafo: 0.08647  ,  mIoU_pred_center_lowres: 0.05846  ,  mIoU_pred_center_highres: 0.07839  ,  Loss/total: 56.08338  ,  Loss/GIoU: 2.33693  ,  Loss/weighted_GIoU: 2.33693  ,  Loss/clf_loss_test: 0.53746  ,  Loss/weighted_clf_loss_test: 53.74645
[train: 1, 500 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26078  ,  mIoU_lowres: 0.25267  ,  mIoU_highres: 0.24455  ,  Loss/weighted_GIoU_highres: 0.78443  ,  mIoU_pred_center_trafo: 0.08848  ,  mIoU_pred_center_lowres: 0.05871  ,  mIoU_pred_center_highres: 0.08029  ,  Loss/total: 55.20101  ,  Loss/GIoU: 2.33222  ,  Loss/weighted_GIoU: 2.33222  ,  Loss/clf_loss_test: 0.52869  ,  Loss/weighted_clf_loss_test: 52.86879
[train: 1, 510 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25877  ,  mIoU_lowres: 0.25001  ,  mIoU_highres: 0.24227  ,  Loss/weighted_GIoU_highres: 0.78645  ,  mIoU_pred_center_trafo: 0.08833  ,  mIoU_pred_center_lowres: 0.05904  ,  mIoU_pred_center_highres: 0.08030  ,  Loss/total: 54.52456  ,  Loss/GIoU: 2.33836  ,  Loss/weighted_GIoU: 2.33836  ,  Loss/clf_loss_test: 0.52186  ,  Loss/weighted_clf_loss_test: 52.18620
[train: 1, 520 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26158  ,  mIoU_lowres: 0.25286  ,  mIoU_highres: 0.24564  ,  Loss/weighted_GIoU_highres: 0.78257  ,  mIoU_pred_center_trafo: 0.09253  ,  mIoU_pred_center_lowres: 0.06217  ,  mIoU_pred_center_highres: 0.08308  ,  Loss/total: 53.72526  ,  Loss/GIoU: 2.32772  ,  Loss/weighted_GIoU: 2.32772  ,  Loss/clf_loss_test: 0.51398  ,  Loss/weighted_clf_loss_test: 51.39755
[train: 1, 530 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26113  ,  mIoU_lowres: 0.25227  ,  mIoU_highres: 0.24512  ,  Loss/weighted_GIoU_highres: 0.78257  ,  mIoU_pred_center_trafo: 0.09292  ,  mIoU_pred_center_lowres: 0.06268  ,  mIoU_pred_center_highres: 0.08319  ,  Loss/total: 52.87110  ,  Loss/GIoU: 2.32765  ,  Loss/weighted_GIoU: 2.32765  ,  Loss/clf_loss_test: 0.50543  ,  Loss/weighted_clf_loss_test: 50.54345
[train: 1, 540 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26193  ,  mIoU_lowres: 0.25332  ,  mIoU_highres: 0.24648  ,  Loss/weighted_GIoU_highres: 0.78119  ,  mIoU_pred_center_trafo: 0.09357  ,  mIoU_pred_center_lowres: 0.06465  ,  mIoU_pred_center_highres: 0.08538  ,  Loss/total: 52.08543  ,  Loss/GIoU: 2.32465  ,  Loss/weighted_GIoU: 2.32465  ,  Loss/clf_loss_test: 0.49761  ,  Loss/weighted_clf_loss_test: 49.76078
[train: 1, 550 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25990  ,  mIoU_lowres: 0.25150  ,  mIoU_highres: 0.24463  ,  Loss/weighted_GIoU_highres: 0.78274  ,  mIoU_pred_center_trafo: 0.09409  ,  mIoU_pred_center_lowres: 0.06598  ,  mIoU_pred_center_highres: 0.08432  ,  Loss/total: 51.37396  ,  Loss/GIoU: 2.32951  ,  Loss/weighted_GIoU: 2.32951  ,  Loss/clf_loss_test: 0.49044  ,  Loss/weighted_clf_loss_test: 49.04444
[train: 1, 560 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25943  ,  mIoU_lowres: 0.25110  ,  mIoU_highres: 0.24457  ,  Loss/weighted_GIoU_highres: 0.78248  ,  mIoU_pred_center_trafo: 0.09298  ,  mIoU_pred_center_lowres: 0.06699  ,  mIoU_pred_center_highres: 0.08528  ,  Loss/total: 50.61197  ,  Loss/GIoU: 2.32954  ,  Loss/weighted_GIoU: 2.32954  ,  Loss/clf_loss_test: 0.48282  ,  Loss/weighted_clf_loss_test: 48.28243
[train: 1, 570 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26144  ,  mIoU_lowres: 0.25350  ,  mIoU_highres: 0.24668  ,  Loss/weighted_GIoU_highres: 0.78025  ,  mIoU_pred_center_trafo: 0.09499  ,  mIoU_pred_center_lowres: 0.06920  ,  mIoU_pred_center_highres: 0.08757  ,  Loss/total: 49.89517  ,  Loss/GIoU: 2.32253  ,  Loss/weighted_GIoU: 2.32253  ,  Loss/clf_loss_test: 0.47573  ,  Loss/weighted_clf_loss_test: 47.57264
[train: 1, 580 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26102  ,  mIoU_lowres: 0.25339  ,  mIoU_highres: 0.24653  ,  Loss/weighted_GIoU_highres: 0.78012  ,  mIoU_pred_center_trafo: 0.09566  ,  mIoU_pred_center_lowres: 0.06991  ,  mIoU_pred_center_highres: 0.08826  ,  Loss/total: 49.30671  ,  Loss/GIoU: 2.32227  ,  Loss/weighted_GIoU: 2.32227  ,  Loss/clf_loss_test: 0.46984  ,  Loss/weighted_clf_loss_test: 46.98445
[train: 1, 590 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26163  ,  mIoU_lowres: 0.25344  ,  mIoU_highres: 0.24713  ,  Loss/weighted_GIoU_highres: 0.77968  ,  mIoU_pred_center_trafo: 0.09723  ,  mIoU_pred_center_lowres: 0.07191  ,  mIoU_pred_center_highres: 0.09023  ,  Loss/total: 48.64940  ,  Loss/GIoU: 2.32125  ,  Loss/weighted_GIoU: 2.32125  ,  Loss/clf_loss_test: 0.46328  ,  Loss/weighted_clf_loss_test: 46.32815
[train: 1, 600 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26010  ,  mIoU_lowres: 0.25160  ,  mIoU_highres: 0.24565  ,  Loss/weighted_GIoU_highres: 0.78076  ,  mIoU_pred_center_trafo: 0.09701  ,  mIoU_pred_center_lowres: 0.07278  ,  mIoU_pred_center_highres: 0.09073  ,  Loss/total: 48.00181  ,  Loss/GIoU: 2.32485  ,  Loss/weighted_GIoU: 2.32485  ,  Loss/clf_loss_test: 0.45677  ,  Loss/weighted_clf_loss_test: 45.67696
[train: 1, 610 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.25924  ,  mIoU_lowres: 0.25084  ,  mIoU_highres: 0.24510  ,  Loss/weighted_GIoU_highres: 0.78090  ,  mIoU_pred_center_trafo: 0.09810  ,  mIoU_pred_center_lowres: 0.07460  ,  mIoU_pred_center_highres: 0.09200  ,  Loss/total: 47.39739  ,  Loss/GIoU: 2.32571  ,  Loss/weighted_GIoU: 2.32571  ,  Loss/clf_loss_test: 0.45072  ,  Loss/weighted_clf_loss_test: 45.07167
[train: 1, 620 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26020  ,  mIoU_lowres: 0.25185  ,  mIoU_highres: 0.24609  ,  Loss/weighted_GIoU_highres: 0.77999  ,  mIoU_pred_center_trafo: 0.10108  ,  mIoU_pred_center_lowres: 0.07802  ,  mIoU_pred_center_highres: 0.09343  ,  Loss/total: 46.74782  ,  Loss/GIoU: 2.32286  ,  Loss/weighted_GIoU: 2.32286  ,  Loss/clf_loss_test: 0.44425  ,  Loss/weighted_clf_loss_test: 44.42496
[train: 1, 630 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26034  ,  mIoU_lowres: 0.25208  ,  mIoU_highres: 0.24648  ,  Loss/weighted_GIoU_highres: 0.77988  ,  mIoU_pred_center_trafo: 0.10257  ,  mIoU_pred_center_lowres: 0.08006  ,  mIoU_pred_center_highres: 0.09513  ,  Loss/total: 46.13166  ,  Loss/GIoU: 2.32285  ,  Loss/weighted_GIoU: 2.32285  ,  Loss/clf_loss_test: 0.43809  ,  Loss/weighted_clf_loss_test: 43.80881
[train: 1, 640 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26145  ,  mIoU_lowres: 0.25342  ,  mIoU_highres: 0.24786  ,  Loss/weighted_GIoU_highres: 0.77843  ,  mIoU_pred_center_trafo: 0.10413  ,  mIoU_pred_center_lowres: 0.08182  ,  mIoU_pred_center_highres: 0.09649  ,  Loss/total: 45.56195  ,  Loss/GIoU: 2.31877  ,  Loss/weighted_GIoU: 2.31877  ,  Loss/clf_loss_test: 0.43243  ,  Loss/weighted_clf_loss_test: 43.24318
[train: 1, 650 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26130  ,  mIoU_lowres: 0.25365  ,  mIoU_highres: 0.24795  ,  Loss/weighted_GIoU_highres: 0.77847  ,  mIoU_pred_center_trafo: 0.10529  ,  mIoU_pred_center_lowres: 0.08354  ,  mIoU_pred_center_highres: 0.09771  ,  Loss/total: 44.97078  ,  Loss/GIoU: 2.31903  ,  Loss/weighted_GIoU: 2.31903  ,  Loss/clf_loss_test: 0.42652  ,  Loss/weighted_clf_loss_test: 42.65174
[train: 1, 660 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26272  ,  mIoU_lowres: 0.25512  ,  mIoU_highres: 0.24955  ,  Loss/weighted_GIoU_highres: 0.77653  ,  mIoU_pred_center_trafo: 0.10763  ,  mIoU_pred_center_lowres: 0.08650  ,  mIoU_pred_center_highres: 0.09959  ,  Loss/total: 44.37985  ,  Loss/GIoU: 2.31354  ,  Loss/weighted_GIoU: 2.31354  ,  Loss/clf_loss_test: 0.42066  ,  Loss/weighted_clf_loss_test: 42.06631
[train: 1, 670 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26312  ,  mIoU_lowres: 0.25590  ,  mIoU_highres: 0.25038  ,  Loss/weighted_GIoU_highres: 0.77545  ,  mIoU_pred_center_trafo: 0.10955  ,  mIoU_pred_center_lowres: 0.08794  ,  mIoU_pred_center_highres: 0.10075  ,  Loss/total: 44.06690  ,  Loss/GIoU: 2.31069  ,  Loss/weighted_GIoU: 2.31069  ,  Loss/clf_loss_test: 0.41756  ,  Loss/weighted_clf_loss_test: 41.75621
[train: 1, 680 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26294  ,  mIoU_lowres: 0.25549  ,  mIoU_highres: 0.24986  ,  Loss/weighted_GIoU_highres: 0.77560  ,  mIoU_pred_center_trafo: 0.10959  ,  mIoU_pred_center_lowres: 0.08892  ,  mIoU_pred_center_highres: 0.10111  ,  Loss/total: 43.61825  ,  Loss/GIoU: 2.31071  ,  Loss/weighted_GIoU: 2.31071  ,  Loss/clf_loss_test: 0.41308  ,  Loss/weighted_clf_loss_test: 41.30754
[train: 1, 690 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26567  ,  mIoU_lowres: 0.25826  ,  mIoU_highres: 0.25252  ,  Loss/weighted_GIoU_highres: 0.77287  ,  mIoU_pred_center_trafo: 0.11118  ,  mIoU_pred_center_lowres: 0.09153  ,  mIoU_pred_center_highres: 0.10371  ,  Loss/total: 43.08884  ,  Loss/GIoU: 2.30242  ,  Loss/weighted_GIoU: 2.30242  ,  Loss/clf_loss_test: 0.40786  ,  Loss/weighted_clf_loss_test: 40.78642
[train: 1, 700 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26555  ,  mIoU_lowres: 0.25839  ,  mIoU_highres: 0.25266  ,  Loss/weighted_GIoU_highres: 0.77357  ,  mIoU_pred_center_trafo: 0.11197  ,  mIoU_pred_center_lowres: 0.09271  ,  mIoU_pred_center_highres: 0.10478  ,  Loss/total: 42.91777  ,  Loss/GIoU: 2.30475  ,  Loss/weighted_GIoU: 2.30475  ,  Loss/clf_loss_test: 0.40613  ,  Loss/weighted_clf_loss_test: 40.61302
[train: 1, 710 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26582  ,  mIoU_lowres: 0.25876  ,  mIoU_highres: 0.25309  ,  Loss/weighted_GIoU_highres: 0.77354  ,  mIoU_pred_center_trafo: 0.11255  ,  mIoU_pred_center_lowres: 0.09401  ,  mIoU_pred_center_highres: 0.10556  ,  Loss/total: 42.45166  ,  Loss/GIoU: 2.30481  ,  Loss/weighted_GIoU: 2.30481  ,  Loss/clf_loss_test: 0.40147  ,  Loss/weighted_clf_loss_test: 40.14685
[train: 1, 720 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26533  ,  mIoU_lowres: 0.25837  ,  mIoU_highres: 0.25275  ,  Loss/weighted_GIoU_highres: 0.77353  ,  mIoU_pred_center_trafo: 0.11325  ,  mIoU_pred_center_lowres: 0.09527  ,  mIoU_pred_center_highres: 0.10652  ,  Loss/total: 42.03647  ,  Loss/GIoU: 2.30498  ,  Loss/weighted_GIoU: 2.30498  ,  Loss/clf_loss_test: 0.39731  ,  Loss/weighted_clf_loss_test: 39.73149
[train: 1, 730 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26445  ,  mIoU_lowres: 0.25756  ,  mIoU_highres: 0.25203  ,  Loss/weighted_GIoU_highres: 0.77392  ,  mIoU_pred_center_trafo: 0.11385  ,  mIoU_pred_center_lowres: 0.09537  ,  mIoU_pred_center_highres: 0.10683  ,  Loss/total: 41.59381  ,  Loss/GIoU: 2.30640  ,  Loss/weighted_GIoU: 2.30640  ,  Loss/clf_loss_test: 0.39287  ,  Loss/weighted_clf_loss_test: 39.28742
[train: 1, 740 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26396  ,  mIoU_lowres: 0.25722  ,  mIoU_highres: 0.25172  ,  Loss/weighted_GIoU_highres: 0.77396  ,  mIoU_pred_center_trafo: 0.11426  ,  mIoU_pred_center_lowres: 0.09540  ,  mIoU_pred_center_highres: 0.10677  ,  Loss/total: 41.14929  ,  Loss/GIoU: 2.30668  ,  Loss/weighted_GIoU: 2.30668  ,  Loss/clf_loss_test: 0.38843  ,  Loss/weighted_clf_loss_test: 38.84261
[train: 1, 750 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26404  ,  mIoU_lowres: 0.25738  ,  mIoU_highres: 0.25193  ,  Loss/weighted_GIoU_highres: 0.77379  ,  mIoU_pred_center_trafo: 0.11506  ,  mIoU_pred_center_lowres: 0.09689  ,  mIoU_pred_center_highres: 0.10673  ,  Loss/total: 40.73255  ,  Loss/GIoU: 2.30633  ,  Loss/weighted_GIoU: 2.30633  ,  Loss/clf_loss_test: 0.38426  ,  Loss/weighted_clf_loss_test: 38.42623
[train: 1, 760 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26257  ,  mIoU_lowres: 0.25590  ,  mIoU_highres: 0.25057  ,  Loss/weighted_GIoU_highres: 0.77507  ,  mIoU_pred_center_trafo: 0.11412  ,  mIoU_pred_center_lowres: 0.09715  ,  mIoU_pred_center_highres: 0.10588  ,  Loss/total: 40.30441  ,  Loss/GIoU: 2.31036  ,  Loss/weighted_GIoU: 2.31036  ,  Loss/clf_loss_test: 0.37994  ,  Loss/weighted_clf_loss_test: 37.99405
[train: 1, 770 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26431  ,  mIoU_lowres: 0.25752  ,  mIoU_highres: 0.25234  ,  Loss/weighted_GIoU_highres: 0.77300  ,  mIoU_pred_center_trafo: 0.11456  ,  mIoU_pred_center_lowres: 0.09758  ,  mIoU_pred_center_highres: 0.10716  ,  Loss/total: 39.91761  ,  Loss/GIoU: 2.30429  ,  Loss/weighted_GIoU: 2.30429  ,  Loss/clf_loss_test: 0.37613  ,  Loss/weighted_clf_loss_test: 37.61332
[train: 1, 780 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26404  ,  mIoU_lowres: 0.25730  ,  mIoU_highres: 0.25209  ,  Loss/weighted_GIoU_highres: 0.77295  ,  mIoU_pred_center_trafo: 0.11448  ,  mIoU_pred_center_lowres: 0.09814  ,  mIoU_pred_center_highres: 0.10721  ,  Loss/total: 39.52253  ,  Loss/GIoU: 2.30416  ,  Loss/weighted_GIoU: 2.30416  ,  Loss/clf_loss_test: 0.37218  ,  Loss/weighted_clf_loss_test: 37.21837
[train: 1, 790 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26374  ,  mIoU_lowres: 0.25705  ,  mIoU_highres: 0.25192  ,  Loss/weighted_GIoU_highres: 0.77308  ,  mIoU_pred_center_trafo: 0.11495  ,  mIoU_pred_center_lowres: 0.09903  ,  mIoU_pred_center_highres: 0.10732  ,  Loss/total: 39.12574  ,  Loss/GIoU: 2.30475  ,  Loss/weighted_GIoU: 2.30475  ,  Loss/clf_loss_test: 0.36821  ,  Loss/weighted_clf_loss_test: 36.82098
[train: 1, 800 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26399  ,  mIoU_lowres: 0.25730  ,  mIoU_highres: 0.25223  ,  Loss/weighted_GIoU_highres: 0.77300  ,  mIoU_pred_center_trafo: 0.11508  ,  mIoU_pred_center_lowres: 0.09914  ,  mIoU_pred_center_highres: 0.10667  ,  Loss/total: 38.75566  ,  Loss/GIoU: 2.30454  ,  Loss/weighted_GIoU: 2.30454  ,  Loss/clf_loss_test: 0.36451  ,  Loss/weighted_clf_loss_test: 36.45112
[train: 1, 810 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26356  ,  mIoU_lowres: 0.25697  ,  mIoU_highres: 0.25184  ,  Loss/weighted_GIoU_highres: 0.77321  ,  mIoU_pred_center_trafo: 0.11506  ,  mIoU_pred_center_lowres: 0.09944  ,  mIoU_pred_center_highres: 0.10654  ,  Loss/total: 38.37291  ,  Loss/GIoU: 2.30512  ,  Loss/weighted_GIoU: 2.30512  ,  Loss/clf_loss_test: 0.36068  ,  Loss/weighted_clf_loss_test: 36.06779
[train: 1, 820 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26415  ,  mIoU_lowres: 0.25764  ,  mIoU_highres: 0.25255  ,  Loss/weighted_GIoU_highres: 0.77269  ,  mIoU_pred_center_trafo: 0.11549  ,  mIoU_pred_center_lowres: 0.10006  ,  mIoU_pred_center_highres: 0.10680  ,  Loss/total: 37.99079  ,  Loss/GIoU: 2.30379  ,  Loss/weighted_GIoU: 2.30379  ,  Loss/clf_loss_test: 0.35687  ,  Loss/weighted_clf_loss_test: 35.68700
[train: 1, 830 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26311  ,  mIoU_lowres: 0.25680  ,  mIoU_highres: 0.25173  ,  Loss/weighted_GIoU_highres: 0.77358  ,  mIoU_pred_center_trafo: 0.11492  ,  mIoU_pred_center_lowres: 0.09987  ,  mIoU_pred_center_highres: 0.10615  ,  Loss/total: 37.64711  ,  Loss/GIoU: 2.30665  ,  Loss/weighted_GIoU: 2.30665  ,  Loss/clf_loss_test: 0.35340  ,  Loss/weighted_clf_loss_test: 35.34046
[train: 1, 840 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26371  ,  mIoU_lowres: 0.25763  ,  mIoU_highres: 0.25258  ,  Loss/weighted_GIoU_highres: 0.77269  ,  mIoU_pred_center_trafo: 0.11670  ,  mIoU_pred_center_lowres: 0.10152  ,  mIoU_pred_center_highres: 0.10750  ,  Loss/total: 37.29447  ,  Loss/GIoU: 2.30423  ,  Loss/weighted_GIoU: 2.30423  ,  Loss/clf_loss_test: 0.34990  ,  Loss/weighted_clf_loss_test: 34.99023
[train: 1, 850 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26468  ,  mIoU_lowres: 0.25833  ,  mIoU_highres: 0.25344  ,  Loss/weighted_GIoU_highres: 0.77197  ,  mIoU_pred_center_trafo: 0.11820  ,  mIoU_pred_center_lowres: 0.10309  ,  mIoU_pred_center_highres: 0.10861  ,  Loss/total: 36.97688  ,  Loss/GIoU: 2.30216  ,  Loss/weighted_GIoU: 2.30216  ,  Loss/clf_loss_test: 0.34675  ,  Loss/weighted_clf_loss_test: 34.67472
[train: 1, 860 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26427  ,  mIoU_lowres: 0.25776  ,  mIoU_highres: 0.25303  ,  Loss/weighted_GIoU_highres: 0.77215  ,  mIoU_pred_center_trafo: 0.11765  ,  mIoU_pred_center_lowres: 0.10310  ,  mIoU_pred_center_highres: 0.10852  ,  Loss/total: 36.69973  ,  Loss/GIoU: 2.30275  ,  Loss/weighted_GIoU: 2.30275  ,  Loss/clf_loss_test: 0.34397  ,  Loss/weighted_clf_loss_test: 34.39698
[train: 1, 870 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26416  ,  mIoU_lowres: 0.25760  ,  mIoU_highres: 0.25307  ,  Loss/weighted_GIoU_highres: 0.77255  ,  mIoU_pred_center_trafo: 0.11831  ,  mIoU_pred_center_lowres: 0.10323  ,  mIoU_pred_center_highres: 0.10801  ,  Loss/total: 36.41303  ,  Loss/GIoU: 2.30443  ,  Loss/weighted_GIoU: 2.30443  ,  Loss/clf_loss_test: 0.34109  ,  Loss/weighted_clf_loss_test: 34.10860
[train: 1, 880 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26357  ,  mIoU_lowres: 0.25679  ,  mIoU_highres: 0.25251  ,  Loss/weighted_GIoU_highres: 0.77290  ,  mIoU_pred_center_trafo: 0.11760  ,  mIoU_pred_center_lowres: 0.10313  ,  mIoU_pred_center_highres: 0.10781  ,  Loss/total: 36.10505  ,  Loss/GIoU: 2.30576  ,  Loss/weighted_GIoU: 2.30576  ,  Loss/clf_loss_test: 0.33799  ,  Loss/weighted_clf_loss_test: 33.79929
[train: 1, 890 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26390  ,  mIoU_lowres: 0.25724  ,  mIoU_highres: 0.25288  ,  Loss/weighted_GIoU_highres: 0.77246  ,  mIoU_pred_center_trafo: 0.11734  ,  mIoU_pred_center_lowres: 0.10347  ,  mIoU_pred_center_highres: 0.10845  ,  Loss/total: 35.78839  ,  Loss/GIoU: 2.30445  ,  Loss/weighted_GIoU: 2.30445  ,  Loss/clf_loss_test: 0.33484  ,  Loss/weighted_clf_loss_test: 33.48394
[train: 1, 900 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26386  ,  mIoU_lowres: 0.25724  ,  mIoU_highres: 0.25304  ,  Loss/weighted_GIoU_highres: 0.77235  ,  mIoU_pred_center_trafo: 0.11804  ,  mIoU_pred_center_lowres: 0.10465  ,  mIoU_pred_center_highres: 0.10926  ,  Loss/total: 35.47378  ,  Loss/GIoU: 2.30450  ,  Loss/weighted_GIoU: 2.30450  ,  Loss/clf_loss_test: 0.33169  ,  Loss/weighted_clf_loss_test: 33.16928
[train: 1, 910 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26514  ,  mIoU_lowres: 0.25857  ,  mIoU_highres: 0.25446  ,  Loss/weighted_GIoU_highres: 0.77103  ,  mIoU_pred_center_trafo: 0.11911  ,  mIoU_pred_center_lowres: 0.10583  ,  mIoU_pred_center_highres: 0.10975  ,  Loss/total: 35.18083  ,  Loss/GIoU: 2.30064  ,  Loss/weighted_GIoU: 2.30064  ,  Loss/clf_loss_test: 0.32880  ,  Loss/weighted_clf_loss_test: 32.88019
[train: 1, 920 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26628  ,  mIoU_lowres: 0.25982  ,  mIoU_highres: 0.25576  ,  Loss/weighted_GIoU_highres: 0.76998  ,  mIoU_pred_center_trafo: 0.11923  ,  mIoU_pred_center_lowres: 0.10657  ,  mIoU_pred_center_highres: 0.11054  ,  Loss/total: 34.88253  ,  Loss/GIoU: 2.29768  ,  Loss/weighted_GIoU: 2.29768  ,  Loss/clf_loss_test: 0.32585  ,  Loss/weighted_clf_loss_test: 32.58484
[train: 1, 930 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26666  ,  mIoU_lowres: 0.26032  ,  mIoU_highres: 0.25623  ,  Loss/weighted_GIoU_highres: 0.76925  ,  mIoU_pred_center_trafo: 0.11938  ,  mIoU_pred_center_lowres: 0.10769  ,  mIoU_pred_center_highres: 0.11011  ,  Loss/total: 34.62496  ,  Loss/GIoU: 2.29553  ,  Loss/weighted_GIoU: 2.29553  ,  Loss/clf_loss_test: 0.32329  ,  Loss/weighted_clf_loss_test: 32.32943
[train: 1, 940 / 40000] FPS: 0.7 (0.7)  ,  mIoU_trafo: 0.26540  ,  mIoU_lowres: 0.25916  ,  mIoU_highres: 0.25507  ,  Loss/weighted_GIoU_highres: 0.77014  ,  mIoU_pred_center_trafo: 0.11894  ,  mIoU_pred_center_lowres: 0.10739  ,  mIoU_pred_center_highres: 0.10939  ,  Loss/total: 34.33679  ,  Loss/GIoU: 2.29827  ,  Loss/weighted_GIoU: 2.29827  ,  Loss/clf_loss_test: 0.32039  ,  Loss/weighted_clf_loss_test: 32.03853
Training crashed at epoch 1
Traceback for the error!
Traceback (most recent call last):
  File "ltr/../ltr/trainers/base_trainer.py", line 70, in train
    self.train_epoch()
  File "ltr/../ltr/trainers/ltr_trainer.py", line 97, in train_epoch
    self.cycle_dataset(loader)
  File "ltr/../ltr/trainers/ltr_trainer.py", line 66, in cycle_dataset
    for i, data in enumerate(loader, 1):
  File "ltr/../ltr/data/loader.py", line 208, in __iter__
    yield next(self.iterator)
  File "/home/hlwn032u1/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/hlwn032u1/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/home/hlwn032u1/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/hlwn032u1/.local/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
Exception: Caught Exception in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/hlwn032u1/.local/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/hlwn032u1/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/hlwn032u1/.local/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "ltr/../ltr/data/sampler.py", line 794, in __getitem__
    return self.processing(data)
  File "ltr/../ltr/data/processing.py", line 1955, in __call__
    bboxes=data['test_anno'], new_roll=False)
  File "ltr/../ltr/data/transforms.py", line 62, in __call__
    out = t(**out, joint=joint_mode, new_roll=new_roll)
  File "ltr/../ltr/data/transforms.py", line 117, in __call__
    params = (self._get_image_size(input_vars),) + self._rand_params
  File "ltr/../ltr/data/transforms.py", line 140, in _get_image_size
    raise Exception('Unknown image type')
Exception: Unknown image type

Thanks, Yasser

2006pmach commented 3 months ago

Hey, from what you shared it is impossible for me to know what is causing the issue. I think you should try to understand which image is causing the problem and making sure that it uncorrupted.