Open mazatov opened 2 years ago
Hi Mike, I see you trained for a very few steps, the accuracy will start increasing much later during the training process. You could try setting the soccernetv3.training_subset config to a smaller value (0.01 for example) for a much shorter training time and to see what happens.
Ah yeah, got it. Thanks. I was assuming it was starting form pretrained Market weights and was surprised to see 0. Training from scratch makes sense it would start with 0.
@VlSomers follow up question :)
On a small subset of 1% I got a high accuracy. However, when I train on 50% of the dataset I still get pretty low accuracy while the Map score is high. Do you know how this accuracy is calculated? Seems so strange we'd have a decent map score with such low accuracy.
Below are the last few lines of the calculations.
epoch: [50/50][1276/1286] time 0.972 (0.986) data 0.000 (0.018) eta 0:00:09 loss_t 0.0531 (0.0428) loss_x 11.2660 (11.2688) acc 0.0000 (0.1678) lr 0.000003
epoch: [50/50][1277/1286] time 0.977 (0.986) data 0.000 (0.018) eta 0:00:08 loss_t 0.0805 (0.0428) loss_x 11.2628 (11.2688) acc 3.1250 (0.1701) lr 0.000003
epoch: [50/50][1278/1286] time 0.970 (0.986) data 0.000 (0.018) eta 0:00:07 loss_t 0.0501 (0.0428) loss_x 11.2664 (11.2688) acc 0.7812 (0.1706) lr 0.000003
epoch: [50/50][1279/1286] time 0.964 (0.986) data 0.006 (0.018) eta 0:00:06 loss_t 0.0356 (0.0428) loss_x 11.2624 (11.2688) acc 2.3438 (0.1723) lr 0.000003
epoch: [50/50][1280/1286] time 0.965 (0.986) data 0.000 (0.018) eta 0:00:05 loss_t 0.0842 (0.0429) loss_x 11.2630 (11.2688) acc 2.3438 (0.1740) lr 0.000003
epoch: [50/50][1281/1286] time 0.964 (0.986) data 0.000 (0.018) eta 0:00:04 loss_t 0.0539 (0.0429) loss_x 11.2661 (11.2688) acc 0.0000 (0.1738) lr 0.000003
epoch: [50/50][1282/1286] time 0.962 (0.986) data 0.000 (0.018) eta 0:00:03 loss_t 0.0410 (0.0429) loss_x 11.2639 (11.2688) acc 0.0000 (0.1737) lr 0.000003
epoch: [50/50][1283/1286] time 0.963 (0.986) data 0.000 (0.018) eta 0:00:02 loss_t 0.0328 (0.0429) loss_x 11.2647 (11.2688) acc 0.7812 (0.1742) lr 0.000003
epoch: [50/50][1284/1286] time 0.963 (0.986) data 0.000 (0.018) eta 0:00:01 loss_t 0.0755 (0.0429) loss_x 11.2599 (11.2688) acc 3.1250 (0.1765) lr 0.000003
epoch: [50/50][1285/1286] time 0.965 (0.986) data 0.000 (0.018) eta 0:00:00 loss_t 0.0721 (0.0429) loss_x 11.2589 (11.2688) acc 0.7812 (0.1769) lr 0.000003
epoch: [50/50][1286/1286] time 0.963 (0.986) data 0.000 (0.018) eta 0:00:00 loss_t 0.0410 (0.0429) loss_x 11.2601 (11.2688) acc 0.0000 (0.1768) lr 0.000003
=> Final test
##### Evaluating soccernetv3 (source) #####
Extracting features from query set ...
Done, obtained 11638-by-512 matrix
Extracting features from gallery set ...
Done, obtained 34355-by-512 matrix
Speed: 0.0166 sec/batch
Computing distance matrix with metric=euclidean ...
Exporting ranking results to 'log/ranking_results_soccernetv3_2022-05-19_14_22_54_648.json' for external evaluation...
Computing CMC and mAP ...
** Results **
mAP: 63.6%
CMC curve
Rank-1 : 52.4%
Hi Mazatov, it seems indeed strange, as a comparison, when using a soccernetv3.training_subset of 0.02, I start getting 100% accuracy after +/- 20 epochs. The only explanation I see is the "infeasibility" of the classification task when the training dataset becomes too big: if you train on 50% of the dataset, you'll end up with so much training identities (and multiple identities for the same player, as explained in the README and the video tutorial), that classifying a sample into its unique correct identity becomes infeasible. However, the network still learns something (the triplet loss plays an important role for that) and you still get nice final ranking performance.
Hi @VlSomers , yeah I get similar results on 0.02. That's an interesting thought about lots of idetities. Overall the trained model still performs well on the test dataset so the triplet loss is working. I was assuming the accuracy just measures if it's splitting classes well within the batch, or within the triplet, so it never actually compares all the identities. Do you know the accuracy is calculated here?
Hey @VlSomers , might be too late but decided to try my hand at this challenge as well while there is some time. I'm justing testing out your benchline code and in the outputs I get accuracy always 0, which doesn't seem right haha. Do you have any idea what I might be doing wrong with the benchmark code?