Training Results on the Homography estimation

[faruknane]

Hi,

We have worked on the paper for hours this week. We followed the first part of the paper which is training the initial registration network. We experimented with lots configurations and augmentation settings. We have achieved the best results using mobilenet_v3_large model. The paper used Resnet18 model. Also, I need to mention that it was somehow hard to do augmentation on images where you also have to augment the homography matrix in the same way.

Training configuration:

HorizontalFlip(0.5, 115),
RandomCrop(0.8, 1280, 1000),
ResizeImage((300,300)),
ColorShift(0.5, 25, 25, 25),
Noise(0.3, 5, 5),
Noise(0.3, -5, 5),

No augmentation is applied on the validation images.

We used the train/val set of World Cup dataset as the training set and the test set as our validation set in our setup. The model predicts where the 4 points on the original frame are projected on the template image.

We followed the same procedure as above.

In the training part, the model was able to generalize well to the validation set. We have trained the model over 61440 epochs (total of 12 millions image iteration) on using RTX 4080 mobile GPU. I'm sharing a few pictures from the validation set results below.

Note: I manually prepared the template image using photoshop, thus there might be imperfections on the template image line positions.

In our case, I think if we try to predict the corner points of play field, we might achieve higher accuracy, since the corner points we would be predicting are on the image directly (>%95 of cases in our dataset) whereas in the current case, the model predicts the projected locations of 4 points which also does not represent any uniqueness (not located at any specific area), thus the model infers the necessary information from whole image. In our case, I believe it would be a local understanding therefore might produce more precise results.

[marcimarc1]

Hi @faruknane, Could you share the code, please?

[faruknane]

I pushed the code @marcimarc1

Training log (MSE Loss):

Train Epoch: 0 [214016/214016] Loss: 0.0029502278
Val Epoch: 0 [186/186] Loss: 0.0009180165
Train Epoch: 1 [214016/214016] Loss: 0.0006221689
Val Epoch: 1 [186/186] Loss: 0.0006758254
Train Epoch: 2 [214016/214016] Loss: 0.0004973794
Val Epoch: 2 [186/186] Loss: 0.0005936610
Train Epoch: 3 [214016/214016] Loss: 0.0004088524
Val Epoch: 3 [186/186] Loss: 0.0005189831
Train Epoch: 4 [214016/214016] Loss: 0.0003491430
Val Epoch: 4 [186/186] Loss: 0.0004844049
Train Epoch: 5 [214016/214016] Loss: 0.0002996890
Val Epoch: 5 [186/186] Loss: 0.0005015272
Train Epoch: 6 [214016/214016] Loss: 0.0002563630
Val Epoch: 6 [186/186] Loss: 0.0004512609
Train Epoch: 7 [214016/214016] Loss: 0.0002179914
Val Epoch: 7 [186/186] Loss: 0.0004995224
Train Epoch: 8 [214016/214016] Loss: 0.0001851324
Val Epoch: 8 [186/186] Loss: 0.0004162563
Train Epoch: 9 [214016/214016] Loss: 0.0001555559
Val Epoch: 9 [186/186] Loss: 0.0004804539
Train Epoch: 10 [214016/214016] Loss: 0.0001302532
Val Epoch: 10 [186/186] Loss: 0.0004185286
Train Epoch: 11 [214016/214016] Loss: 0.0001080968
Val Epoch: 11 [186/186] Loss: 0.0004476402
Train Epoch: 12 [214016/214016] Loss: 0.0000917610
Val Epoch: 12 [186/186] Loss: 0.0004189240
Train Epoch: 13 [214016/214016] Loss: 0.0000810071
Val Epoch: 13 [186/186] Loss: 0.0004333108
Train Epoch: 14 [214016/214016] Loss: 0.0000743926
Val Epoch: 14 [186/186] Loss: 0.0004413294
Train Epoch: 15 [214016/214016] Loss: 0.0000714826
Val Epoch: 15 [186/186] Loss: 0.0003815373
Train Epoch: 16 [214016/214016] Loss: 0.0000693955
Val Epoch: 16 [186/186] Loss: 0.0004305751
Train Epoch: 17 [214016/214016] Loss: 0.0000678344
Val Epoch: 17 [186/186] Loss: 0.0004202581
Train Epoch: 18 [214016/214016] Loss: 0.0000665483
Val Epoch: 18 [186/186] Loss: 0.0004061744
Train Epoch: 19 [214016/214016] Loss: 0.0000664344
Val Epoch: 19 [186/186] Loss: 0.0003963611
Train Epoch: 20 [214016/214016] Loss: 0.0000656217
Val Epoch: 20 [186/186] Loss: 0.0004124599
Train Epoch: 21 [214016/214016] Loss: 0.0000644995
Val Epoch: 21 [186/186] Loss: 0.0004610160
Train Epoch: 22 [214016/214016] Loss: 0.0000634274
Val Epoch: 22 [186/186] Loss: 0.0004121763
Train Epoch: 23 [214016/214016] Loss: 0.0000633816
Val Epoch: 23 [186/186] Loss: 0.0004091429
Train Epoch: 24 [214016/214016] Loss: 0.0000630360
Val Epoch: 24 [186/186] Loss: 0.0003804310
Train Epoch: 25 [214016/214016] Loss: 0.0000625678
Val Epoch: 25 [186/186] Loss: 0.0004142266
Train Epoch: 26 [214016/214016] Loss: 0.0000621694
Val Epoch: 26 [186/186] Loss: 0.0004018158
Train Epoch: 27 [214016/214016] Loss: 0.0000619113
Val Epoch: 27 [186/186] Loss: 0.0004231525
Train Epoch: 28 [214016/214016] Loss: 0.0000615244
Val Epoch: 28 [186/186] Loss: 0.0003970849
Train Epoch: 29 [214016/214016] Loss: 0.0000615867
Val Epoch: 29 [186/186] Loss: 0.0003822143
Train Epoch: 30 [214016/214016] Loss: 0.0000600560
Val Epoch: 30 [186/186] Loss: 0.0003928860
Train Epoch: 31 [214016/214016] Loss: 0.0000601915
Val Epoch: 31 [186/186] Loss: 0.0003902219
Train Epoch: 32 [214016/214016] Loss: 0.0000600173
Val Epoch: 32 [186/186] Loss: 0.0004080824
Train Epoch: 33 [214016/214016] Loss: 0.0000601128
Val Epoch: 33 [186/186] Loss: 0.0003761074
Train Epoch: 34 [214016/214016] Loss: 0.0000597028
Val Epoch: 34 [186/186] Loss: 0.0003930548
Train Epoch: 35 [214016/214016] Loss: 0.0000594529
Val Epoch: 35 [186/186] Loss: 0.0004084161
Train Epoch: 36 [214016/214016] Loss: 0.0000586768
Val Epoch: 36 [186/186] Loss: 0.0003919127
Train Epoch: 37 [214016/214016] Loss: 0.0000587407
Val Epoch: 37 [186/186] Loss: 0.0004099327
Train Epoch: 38 [214016/214016] Loss: 0.0000582755
Val Epoch: 38 [186/186] Loss: 0.0003836764
Train Epoch: 39 [214016/214016] Loss: 0.0000581359
Val Epoch: 39 [186/186] Loss: 0.0003916030
Train Epoch: 40 [214016/214016] Loss: 0.0000583316
Val Epoch: 40 [186/186] Loss: 0.0003874358
Train Epoch: 41 [214016/214016] Loss: 0.0000580278
Val Epoch: 41 [186/186] Loss: 0.0003857910
Train Epoch: 42 [214016/214016] Loss: 0.0000575490
Val Epoch: 42 [186/186] Loss: 0.0003852894
Train Epoch: 43 [214016/214016] Loss: 0.0000582569
Val Epoch: 43 [186/186] Loss: 0.0003796277
Train Epoch: 44 [214016/214016] Loss: 0.0000572272
Val Epoch: 44 [186/186] Loss: 0.0003821638
Train Epoch: 45 [214016/214016] Loss: 0.0000569193
Val Epoch: 45 [186/186] Loss: 0.0003825395
Train Epoch: 46 [214016/214016] Loss: 0.0000571690
Val Epoch: 46 [186/186] Loss: 0.0003919127
Train Epoch: 47 [214016/214016] Loss: 0.0000569697
Val Epoch: 47 [186/186] Loss: 0.0004003892
Train Epoch: 48 [214016/214016] Loss: 0.0000572798
Val Epoch: 48 [186/186] Loss: 0.0003801112
Train Epoch: 49 [214016/214016] Loss: 0.0000565451
Val Epoch: 49 [186/186] Loss: 0.0003952249
Train Epoch: 50 [214016/214016] Loss: 0.0000563597
Val Epoch: 50 [186/186] Loss: 0.0003845988
Train Epoch: 51 [214016/214016] Loss: 0.0000563986
Val Epoch: 51 [186/186] Loss: 0.0003844325
Train Epoch: 52 [214016/214016] Loss: 0.0000561841
Val Epoch: 52 [186/186] Loss: 0.0003918427
Train Epoch: 53 [214016/214016] Loss: 0.0000564877
Val Epoch: 53 [186/186] Loss: 0.0004044866
Train Epoch: 54 [214016/214016] Loss: 0.0000565667
Val Epoch: 54 [186/186] Loss: 0.0004388800
Train Epoch: 55 [214016/214016] Loss: 0.0000559723
Val Epoch: 55 [186/186] Loss: 0.0004184083
Train Epoch: 56 [214016/214016] Loss: 0.0000559571
Val Epoch: 56 [186/186] Loss: 0.0004318865
Train Epoch: 57 [214016/214016] Loss: 0.0000557004
Val Epoch: 57 [186/186] Loss: 0.0003975165
Train Epoch: 58 [214016/214016] Loss: 0.0000553367
Val Epoch: 58 [186/186] Loss: 0.0003943583
Train Epoch: 59 [214016/214016] Loss: 0.0000558425
Val Epoch: 59 [186/186] Loss: 0.0004210808
Train Epoch: 60 [214016/214016] Loss: 0.0000554114
Val Epoch: 60 [186/186] Loss: 0.0004002610
Train Epoch: 61 [94208/214016] Loss: 0.0000558580E

[marcimarc1]

@faruknane Thank you. Looks good. Could you add as a metric average IOU. Good luck with the exams

[faruknane]

@faruknane Thank you. Looks good. Could you add as a metric average IOU. Good luck with the exams

Thank you @marcimarc1! I will work on that as soon as possible.