search

IrvingMeng / MagFace

MagFace: A Universal Representation for Face Recognition and Quality Assessment, CVPR2021, Oral
Apache License 2.0
626 stars 85 forks source link

Low ACC for KDEF dataset #37

Closed rzamarefat closed 2 years ago

rzamarefat commented 2 years ago

Hi, thank you for this awesome repo. I have tried to generate embs on KDEF dataset. This dataset contains samples such as: AF01AFFL AF01AFHL AF01AFS I have used SVM for classifying the generated embs but got a very low acc of 50%. Please note that I have removed images of faces which are depicting the profile. Based on my explanation, what can be the problem? Should I remove the images which are partially straight also (images like the second image)?

IrvingMeng commented 2 years ago

Hi,

Could you provide details of feature generation? For example, how do you align the faces or what scripts are used?

rzamarefat commented 2 years ago

Thank you for your response. Firstly based on the following images, which are the cropped versions of the images shown above, our dataset does not need any alignment (correct me if I'm making a mistake) AF01ANS AF01DIS AF01HAHL As you can see these images are cropped in a way that the images are focused on the face of the person. However, I don't do any alignment because I reckon that they are aligned. In terms of feature generation I am using the module provided here ( "MagFace/inference/gen_feat.py" ). Actually I have used the very same way for other datasets and the generated features were, thanks to your brilliant work, of great quality. For example, when I used SVM for classifying IDs of people within a dataset, the achieved score were above 99.0. But for this dataset (so-called KDEF) the biggest score for SVM that I get is approximately around 45 to 50%. This caused me a insolvable problem. Thank you in advance for your help

IrvingMeng commented 2 years ago

I notice that your cropped images are different from the corrected ones, especially for profile faces. For example, noses tend to be placed in the centers of example images[1] while your third image is at the contrary of that. As the alignment step highly affects recognition performances, I suggest you re-align the faces to see if that's the cause.

[1] https://github.com/IrvingMeng/MagFace/tree/main/inference/toy_imgs

rzamarefat commented 2 years ago

I have done as you said and now my images such as the following: AF01ANFL AF01NEHL Please note that I am using MTCNN for alignment. The ACC of SVM is increased to 70% but still it is very low.

IrvingMeng commented 2 years ago

I see. Do you normalize all the features before using SVM. Otherwise, the low performance may caused by the domain invariance. You may need to re-train the model if that's the case.

rzamarefat commented 2 years ago

Actually I haven't normalize them. And this may cause the problem of low acc (I will try that and see if it solves it). Besides, I have two questions. 1- What do you mean by domain invariance? 2-Can we actually train MagFace on any other dataset containing images of sth else other than faces of human beings? As the developer of this brilliant paper can you see it possible that after training we could get any good quality representations of some other objects?

IrvingMeng commented 2 years ago
  1. domain invariance means the distributions of training and testing dataset are different.
  2. I cannot guarantee good results for datasets besides faces. However, I believe it can still work in theory as long as the data distributions are suitable.