search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.2k stars 21.35k forks source link

Implementation details validation_metric_type 'voc', 'coco' and 'coco_voc' not clear #108986

Open RMValcon opened 1 year ago

RMValcon commented 1 year ago

Hi,

I have a three specific questions w.r.t. the available validation_metric_types: 'voc', 'coco' and 'coco_voc'

  1. For 'voc', some microsoft documentation refers to the VOC site: http://host.robots.ox.ac.uk/pascal/VOC/ However, Pascal VOC uses a different implementation before and after 2010. Before 2010, the AP was approximated using 11 points in the precision-recall diagram, after 2010, the AP was calculated by integrating the area under the curve. Both versions only evaluate the AP at an IoU threshold of 0.5. Which version does AzureML implement? Furthermore, I assume that the validation_iou_threshold hyperparameter changes this single IoU threshold from 0.5 to a user-defined value?
  2. For 'coco' some microsoft documentation refers to the COCO site: https://cocodataset.org/#detection-eval Which states the AP is calculated by taking the average AP for 10 different IoU threshold values between 0.5 and 0.95. Does the validation_iou_threshold hyperparameter change the lower IoU threshold value in combination of validation_metric_type='coco'? I assume not?
  3. Finally, how is 'coco_voc' implemented? I could not find any documentation.

Thank you for you response!

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ManoharLakkoju-MSFT commented 1 year ago

@RMValcon Thanks for your feedback! We will investigate and update as appropriate.

RamanathanChinnappan-MSFT commented 1 year ago

@RMValcon

I've delegated this to @swatig007 , a content author, to review and share their valuable insights.

RMValcon commented 1 year ago

Hi, is there any update? Thanks.