search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.75k stars 7.96k forks source link

Calculation on final mAP(Validation dataset or Test dataset) #8093

Open Ronald-Kray opened 3 years ago

Ronald-Kray commented 3 years ago

@AlexeyAB @WongKinYiu I'm working on calculating mAP of the Object detection algorithms(Yolv5, Yolov4, EfficientDet).

In my opinion, there are 2 ways to calculate mAP(Assume that dataset is all labeled image).

1. Split dataset ratio as Train: Val=80:20, and just finish mAP calculation on Val dataset.

  1. Split dataset ratio as Train: Val: Test=60:20:20--> After calculating mAP on Val dataset, only one more do mAP calculation on test dataset to assess the performance of a fully trained model. Only 1 epoch mAP calculation on Test dataset would be enough because it is already fully trained on Val dataset. In some references said, during training, Val dataset is already referred to on the training dataset, and therefore, calculating mAP is a more accurate way.

My question is that when calculating official mAP, Researchers use only a validation dataset? Or they calculate mAP of the test dataset from the model contributed by the validation dataset?

WongKinYiu commented 3 years ago

I usually use:

  1. training data for training.
  2. validation data for choosing hyper-parameters.
  3. testing data for calculate mAP.
Ronald-Kray commented 3 years ago

@WongKinYiu

I usually use:

  1. training data for training.
  2. validation data for choosing hyper-parameters.
  3. testing data for calculate mAP.

You mean that you spilt data Training data: Validation data: Testing data: 60:20:20 in the object detection problem? but, we can calculate mAP with only validation data (e.g.Training data: Validation date= 80:20). testing data for calculate mAP is necessary?

WongKinYiu commented 3 years ago

Because validation data is used for choosing the best model, that means the training may over-fit the validation data (this case is usually occur on the dataset which has only train/validation set or which provide ground truth for testing set). Only metrics on testing set without participate in developing step is objective results.

Ronald-Kray commented 3 years ago

@WongKinYiu What about a custom dataset? Generally for papers on custom datasets, they only present mAP on Validation set in object detection problems. Is it wrong?

WongKinYiu commented 3 years ago

For custom dataset, it is hard to show your train/val/test set are separated well. In this case, you could show the mean and variance accuracy of cross validation results. Leave one out, jack knife, ... are often used methods.