search

zylo117 / Yet-Another-EfficientDet-Pytorch

The pytorch re-implement of the official efficientdet with SOTA performance in real time and pretrained weights.
GNU Lesser General Public License v3.0
5.21k stars 1.27k forks source link

Calculation on final mAP(Validation dataset or Test dataset) #690

Open Ronald-Kray opened 3 years ago

Ronald-Kray commented 3 years ago

@zylo117

I'm working on calculating mAP of the Object detection algorithms(Yolv5, Yolov4, EfficientDet).

In my opinion, there are 2 ways to calculate mAP(Assume that dataset is all labeled image).

1. Split dataset ratio as Train: Val=80:20, and just finish mAP calculation on Val dataset.

2. Split dataset ratio as Train: Val: Test=60:20:20--> After calculating mAP on Val dataset, only one more do mAP calculation on test dataset to assess the performance of a fully trained model. Only 1 epoch mAP calculation on Test dataset would be enough because it is already fully trained on Val dataset. In some references said, during training, Val dataset is already referred to on the training dataset, and therefore, calculating mAP is a more accurate way.

I think the first one was is correct, but in my research group discussion that someone said the second one is correct. Still, I'm not sure which one is correct. Can you give me some advice or references on this?

zylo117 commented 3 years ago

In the second opinion, val set is basically the same as test set, because both of them are not trained. So picking a test set like this is meaningless, no unless if you are in a competition that someone else have to use that test set to evaluate your model.