search

happynear / FaceDatasets

Some scripts to process face datasets.
BSD 2-Clause "Simplified" License
329 stars 121 forks source link

I guess you misunderstood how open-set evaluation work form your Readme file #10

Open meanmee opened 5 years ago

happynear commented 5 years ago

Can you explain more clearly?

meanmee commented 5 years ago

There is nothing to do with overlapping identities for open-set evaluation, for more details I'd like to recommand you a blog(I suppose you are able to read Chinese): https://zhuanlan.zhihu.com/p/57350424

happynear commented 5 years ago

This post is about "open-set retrieval" or "close-set retrieval". The problem I declare in this repo is about "open-set recognition" or "close-set recognition". Maybe you can search for "open-set recognition" to learn more.

meanmee commented 5 years ago

I just searched and sadly I didn't find any blog or paper which supports your clarification, All I get is if there goes the phrase "open-set", It links to the definition as same as the blog I send you. Could you show me some papers about "open-set recognition"?

happynear commented 5 years ago

Search it on google, and the first link is about open-set recognition. You can read the papers listed on the post. https://www.wjscheirer.com/projects/openset-recognition/ https://www.wjscheirer.com/papers/wjs_tpami2013_openset.pdf

happynear commented 5 years ago

The first figure of SphereFace paper clearly describes the difference between "close-set recognition" and "open-set recognition". https://arxiv.org/pdf/1704.08063.pdf

meanmee commented 5 years ago

Thanks， I'll try to read these papers

meanmee commented 5 years ago

I checked the paper 'Towards Open Set Recognition' and realized I‘ve read it before. According to my understanding, The phrase 'Open-Set' means there might be some classes we don't provide during training, And for face recognition, since we can't put the same identity both in training set and testing set, which is a common sense, so the 'open-set' in face recognition means there will be some probes that we don't have the corresponding galleries. Still, I don't see any word related to your definition of 'open-set recognition'.

happynear commented 5 years ago

I think you already know that there exists a research topic called "open-set recognition". It is different with "open-set retrieval". To be simple, recognition is about training and testing set, retrieval is about probe and gallery set.

Face recognition is a big area including many tasks such as verification, retrieval, tracking etc. The test set varies according to the task.

For the task of face retrieval, we have four sets: training, probe, gallery, imposter. The probe, gallery and imposter are all testing set. For verification, the testing set includes probe set and distractor set. Some dataset's verification protocols use pairs of samples as testing set.

Open-set face recognition is a much bigger topic than open-set face retrieval. It is about training and testing set. Open-set means that there should not be overlapped identities between training and testing set, no matter what the specific task is. Actually nowadays we train a model for many tasks, not only retrieval.

In the end I want to tell you that "open-set" is only a name. Like many names in deep learning area such as transfer learning and meta learning, the meanings may change. This area is developing very fast and the meanings can be re-defined by anyone through published papers. When the papers are widely accepted by the researchers, we have a new consensus about the meaning. State-of-the-art papers such as SphereFace and AM-Softmax use the phrase "open-set", they are widely accepted by face recognition researchers, so that is the meaning.

meanmee commented 5 years ago

What you said is informative to me, Thanks.