search

zhmiao / OpenLongTailRecognition-OLTR

Pytorch implementation for "Large-Scale Long-Tailed Recognition in an Open World" (CVPR 2019 ORAL)
BSD 3-Clause "New" or "Revised" License
839 stars 128 forks source link

Openset f-measure calculation problem #37

Closed Lilyo closed 4 years ago

Lilyo commented 5 years ago

Could you please show us how do you determine the "Positive" & "Negative" for F-measure? think like some other guys also concern about that :)

Lilyo commented 5 years ago

hi, @zhmiao , I have checked the paper you mention,

Original content: "For example, when testing the system with images from validation set, fooling set and open set (see Fig. 3), true positives are defined as the correct classifications on the validation set, false positives are incorrect classifications on the validation set and false negatives are images from the fooling set and open set categories that the system incorrectly classified as known examples."

right now I known you follow the same setting from above paper! BUT, the original meaning of "false negatives" is mean data from validation set, and incorrectly classified to fooling set and open set categories. Why they define as "images from the fooling set and open set categories that the system incorrectly classified as known examples"?

zhmiao commented 5 years ago

Hello @Lilyo , thank you very much for asking. Yes, I agree this metric is a little bit confusing. The reason why the authors treat images from the open set that are incorrectly classified as known samples as false negative is because this whole setting is fundamentally an open set detection problem. We need to identify open set samples at the same time be correct for the closed set classification. When considering both aspects, the authors designed this specific metric. This definitely is not a perfect metric, and we think it is still an open problem that needs improvement from all different directions. Does that make sense?

Lilyo commented 4 years ago

Yes, @zhmiao , this makes more sense now!