search

david2611 / pdq_evaluation

Evaluation code for using probabilistic detection quality (PDQ) measure for probabilistic object detection tasks. Currently supports COCO and robotic vision challenge (RVC) data.
Other
39 stars 7 forks source link

Can the PDQ metric somehow be used in active learning? #5

Closed pieterblok closed 3 years ago

pieterblok commented 3 years ago

@david2611, sorry to bother you. First of all, thank you for the great work on the PDQ metric! As your Youtube presentation showed, this metric is very applicable to probabilistic deep learning algorithms.

One specific question regarding the code: can the PDQ metric somehow be used in active learning in which there are no ground truth labels? In active learning, you want to pool the most uncertain images from an unlabeled dataset, to incrementally retrain the algorithm on these uncertain images. I was thinking of using the PDQ metric to get a better uncertainty measure on the unlabeled images.

However, in most functions of the code there is a specific input needed for the ground truth instances, labels, etc, but that is lacking in active learning. Is there any way to (easily) circumvent this, and use PDQ to measure the uncertainty of different predictions on the same image when using Monte Carlo dropout (during image inference)?

Thanks in advance!

david2611 commented 3 years ago

Hi @pieterbl86 and thanks for your interest in PDQ! Unfortunately I think the answer to your question is no.

PDQ is designed to evaluate how well a probabilistic object detector has detected as specific object much in the same way that mean average precision is used in classical object detection. PDQ can't tell how well you have spatially or semantically described an object if it doesn't know what object you were trying to describe.

The best I can offer would be that potentially you could use the same basic outline of how PDQ merges spatial and semantic quality within your setting, depending on how you are planning to use it. If I understand correctly, you are wanting to use something like PDQ to determine if you have images/detections that your system is particularly uncertain about when performing probabilistic object detection. In that setting a "poor" detection is one with high spatial and/or semantic uncertainty.

In such a case you could, determine for each detection a level of semantic uncertainty (e.g. using entropy) and a level of spatial uncertainty (e.g. using covariance values attained for each corner in someway perhaps). Once you have some score for spatial and semantic uncertainty you can just do what we did and use a geometric mean to find a happy medium between the two and use that score as how "uncertain" the overall detection was.

Again don't know if I am correctly understanding your problem or if the "solution" of mine would work if I did but hopefully it is of some use to you.

pieterblok commented 3 years ago

Hi @pieterbl86 and thanks for your interest in PDQ! Unfortunately I think the answer to your question is no.

PDQ is designed to evaluate how well a probabilistic object detector has detected as specific object much in the same way that mean average precision is used in classical object detection. PDQ can't tell how well you have spatially or semantically described an object if it doesn't know what object you were trying to describe.

The best I can offer would be that potentially you could use the same basic outline of how PDQ merges spatial and semantic quality within your setting, depending on how you are planning to use it. If I understand correctly, you are wanting to use something like PDQ to determine if you have images/detections that your system is particularly uncertain about when performing probabilistic object detection. In that setting a "poor" detection is one with high spatial and/or semantic uncertainty.

In such a case you could, determine for each detection a level of semantic uncertainty (e.g. using entropy) and a level of spatial uncertainty (e.g. using covariance values attained for each corner in someway perhaps). Once you have some score for spatial and semantic uncertainty you can just do what we did and use a geometric mean to find a happy medium between the two and use that score as how "uncertain" the overall detection was.

Again don't know if I am correctly understanding your problem or if the "solution" of mine would work if I did but hopefully it is of some use to you.

Hello @david2611 , thank you for your quick and clear reply. Yes, you are exactly describing the use-case: calculate the semantic and spatial uncertainty of the instance segmentation algorithm (masks), so that I can pool the images the algorithm is most uncertain about (and use these images to retrain the algorithm in an active-learning loop).

Based on your answer, I think I will continue to develop the uncertainty measure, using the publication of Douglas Morrison (I guess he's your colleague at QUT?) : https://nikosuenderhauf.github.io/roboticvisionchallenges/assets/papers/CVPR19/rvc_4.pdf

I will contact you if PDQ can be used later in my research. For now I will close the topic.