search

open-mmlab / mmdetection

OpenMMLab Detection Toolbox and Benchmark
https://mmdetection.readthedocs.io
Apache License 2.0
29.02k stars 9.36k forks source link

Will "Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details" be implemented for promoting research on the long-tail datasets? #6895

Open mengqiDyangge opened 2 years ago

mengqiDyangge commented 2 years ago

Describe the feature Implement new long-tail dataset evaluation metrics "APpool" and "APfix" in paper "Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details".

Motivation The new evaluation metrics "APpool" and "APfix" are proposed in this paper for evaluating the performance of different methods on the long-tail datasets like LVIS. We think implement these metrics will effectively promote the study of the long-tail problem.

Related resources You can look at this repo of large-vocab-devil based on Detectron2, which is not supported in mmdetection. https://github.com/achalddave/large-vocab-devil

Additional context It really worths to be implemented. After that different models can be evaluated more comprehensively on long-tail datasets such as LVIS.

Czm369 commented 2 years ago

Thank you for your proposal.This method is not planned in our development plan, but we also think the evaluation metrics are very important for the long-tail problem. We welcome you to implement this method, and we are very willing to cooperate with you to further improve the code.