Face Verification using caffe step by step

[cheer37]

@happynear I tried to test the this project (Face Verification demo), But i couldn't. VerificationDemo.m is not execution file? By the way, i want c/c++ version, Would you provide me with it? honestly, I am a newbie with caffe, so i hope you are guiding me step by step and will be successful to test the face verification at the end of this session. Thanks in advance.

[happynear]

I am now busy with a conference paper deadline. I will return to face recognition in the end of this year. I will try to implement triplet loss training.

At that time, maybe I will provide a step by step tutorial about how to create your own face verification application.

However, I will do this simultaneously with learning mxnet (https://github.com/dmlc/mxnet). Maybe the tutorial will be based on mxnet and python.

[happynear]

I suggest you to learn how to use Caffe firstly. You can read the documentation of the official Caffe.

Moreover, reading papers is also important, such as DeepID, DeepID2, DeepFace, FaceNet etc.

[cheer37]

@happynear I have already investigated the caffe official site several times. But i couldn't find what i want know like passing arguments, and also they described the all contents incomprehensibly. Also, i have read the papers you suggested. By the way, why did you make a another project for face verification instead of using caffe-windows? Why it can't be made like mnist project by training with solver. And verification means two class classification, so how did you make a image input to train and classify. Thanks.

[happynear]

I haven't successfully trained a siamese network, which is more suitable for verification. Maybe triplet loss is better. I will try it.

This project is mainly written by Matlab, and I don't think it is proper to put it into caffe. In fact, someone has also suggested me to remove the experimental layers from my caffe repository.

MNIST is just a toy dataset in these days. You must understand that face recognition is too close to money. Even though tens of institutions have claimed that they got accuracy above 99% on LFW, none of them have made their model public available.

[cheer37]

I know about it. I am also researching the face recognition algorithm. nowadays, there are dozens of institutions reached at high accuracy in LFW using deep learning, apparently, so i am interested in deep learning and begin to develop. of course how to design the net (model) is my bussiness, but i didn't understand how to use caffe yet. so i am asking a aid from you. firstly, i can't understand the meaning of deploy.prototxt and mean.binaryproto files. secondly, for face verification, we can't feed images in database to the net, because of two class classification, we can select pair of images as a labeling it's from the same or not, but how can we make an input image from pair of two images? Thanks.

[happynear]

If you do not know what is deploy.prototxt and mean.binaryproto, I suggest you read the offical document and try to run some examples.

If you have read the DeepID paper, you will know that the net is trained purely by softmax. Then the trained model is transfered to do the verification job.

In the DeepID2 paper, they added a constrastive loss layer, to make the objective function more close to the verification problem. In that network, the loss is calculated from two images. You can refer to https://github.com/happynear/FaceVerification/blob/master/caffe_proto/mnist_siamese_train_test_concate.prototxt for the network definition. It has two image layers for training and two image layers for testing.

[cheer37]

ok, i see. I will study some more about the caffe. by the way, what are you indicating at the official document? They are one posted in caffe site.? would you post here the links of them. Thanks.

[HKUSTmji]

Dear Feng, very good work! I also have few queries about the face identification:

1. you mentioned that you haven't successfully run the siamese network, why? The siamese demo in caffe is very easy to run.
2. you also mentioned the triplet loss layer, did you implemented it? it's that easy? will you use caffe to implement it?
3. For the LFW dataset, many subject only has one image. How it be possible to be used for face classification training?
4. do you know the official train-test partition for the LFW dataset? I never got that high accuracy mentioned by the existing paper.

Thanks in advance

[happynear]

@HKUSTmji

1. The loss of siamese network cannot converge with my implementation. You can find my net model definition in https://github.com/happynear/FaceVerification/blob/master/caffe_proto/mnist_siamese_train_test_concate.prototxt , and I don't think I did something wrong.
2. I have merged a triplet loss layer in my caffe repo. After cvpr, I will continue to do research on face recognition. I will try to re-implement FaceNet.
3. I use CASIA-Webface for training, and test in LFW.
4. Of course, the training set of LFW is too poor to train a CNN model. You'd better try CASIA-Webface for training.

[HKUSTmji]

@happynear Thanks a lot for the wonderful reply.

1. is the reason, why your siamese network don't converge, that contrastive loss doesn't work or the dataset is too small?
2. if contrastive loss doesn't work, which .prototext was used to train the face verification network?
3. is there any good .caffemodel available for the face identification?
4. would you mind mentioning some tricks for the training process? because when I train such big network, it very easy goes overfitting.
5. did you firstly train the DeepID (or the DeepID2 or DeepID2+) using the CASIA dataset (how can I get this dataset if it only receive download application from orgnization?); Secondly, use the extracted CNN feature to do the identification? could you please give more information?
6. the threshold parameter 'm' in contrastive loss is supposed to dynamically change, however, the 'm' parameter in caffe's contrastive loss layer may be constant. will it affect much? or any other comments?

Thank you again ~

[happynear]

1. The dataset is large enough, it has about 0.5M photos.
2. I used a pure softmax network, its definition is in https://github.com/happynear/FaceVerification/blob/master/caffe_proto/CASIA_train_test.prototxt .
3. Sorry I cannot provide the caffemodel. I have financial support for this project. Sharing code is my biggest privilege. The most useful tool to avoid overfitting is to collect more data.
4. same with question 2.
5. the m is difficult to tune. In the DeepID2 paper, they use grid search to find the current m every a few epoches. However, when I set m to 0, the network still can not converge.

[cheer37]

@happynear Hi, Feng Wang. First, i am going to debug the project as i tried. I tried to build with release 3rdparty you offered in debug mode. But it raises error and dont compile. Would you take care of your repository with debug mode more in this time. If you are willing to do, please post it and tell me. Thanks in advance.

[happynear]

@cheer37 You can debug caffe in release mode if you do not want to debug 3rdparty libraries. Please follow the instructions at https://msdn.microsoft.com/en-us/library/fsk896zz.aspx .

Or you can build your 3rdparty libraries by https://github.com/willyd/caffe-windows-dependencies .

[cheer37]

Thanks.

[cheer37]

@happynear DeepID did face identification among people, didn't they. You also did it? In real life, face verification rather than identification is more important. So, i want to know how to feed input to DNN for verification. At identification, we can feed image with his/her label, but at verification, problem turns into 2 class classification. so how to deal with it? Would you clarify?, Thanks

[happynear]

In DeepID, features are extracted from a identification model. Then a joint bayesian model is trained to give the score of the similarity of pairs of features. In another word, the verification model is transferred from a identification model. They share the same feature extraction step, while verification have more following steps.

[cheer37]

@happynear Did you finish your cvpr? If you did or will be finished, then let me see it. Tnanks.

[happynear]

@cheer37 , Sorry I am not willing to upload it now. I will upload it to arXiv once I have refactored my codes.

[cheer37]

ok, then would you let me know at that time you uploaded to arXiv?

[happynear]

I have lots of engineering work to do. I can only do some research in my spare time. Acturally I do not know when.

[cheer37]

@happynear I tried to run your FaceVerification project, but i couldn't. When running the VerificationDemo.m, An error occured : Undefined variable "caffe" or class "caffe.reset_all". Error in VerificationDemo(line 2)

And also, following codes are using some files you not offered. MatAlignment('init_model','D:\ThirdPartyLibrary\dlib-18.14\shape_predictor_68_face_landmarks.dat'); meanC = caffe.read_mean('D:\deeplearning\caffe-windows\examples\FaceVerification\mean.proto'); net = caffe.Net('D:\deeplearning\caffe-windows\examples\FaceVerification\CASIA_demo.prototxt','D:\deeplearning\caffe-windows\examples\FaceVerification\siamese_iter_500000.caffemodel', 'test');

What should i do to execute your project. Thanks.

[cheer37]

Wouldn't you provide the stuffs which are needed to execute?

[happynear]

No, I have signed an agreement that the model should not be publicly shared. You can get the model by other source files.

If you want a model, here is one that is even better than mine https://github.com/AlfredXiangWu/lfw_face_verification_experiment/ .

[cheer37]

CASIA_demo.prototxt is just deploy.prototxt? Ok, then i don't ask two files CASIA_demo.prototxt and siamese_iter_500000.caffemodel. However, why don't you explain how to fix error? And what about rest two files shape_predictor_68_face_landmarks.dat and mean.proto. As seeing supply training data, usually there are two files model(deploy.prototxt) and weights(.caffemodel) without mean.proto. How can I produce mean.proto? Please help me execute the FaceVerification project you offered. Thanks.

[happynear]

1. The prototxt files used for training are all in https://github.com/happynear/FaceVerification/tree/master/caffe_proto .
2. mean.proto is widely used for large datasets such as ImageNet. It can be generated by compute_image_mean.exe.
3. shape_predictor_68_face_landmarks.dat is from dlib. You can find it in http://dlib.net/ .

[cheer37]

thanks, But what about a error. When running the VerificationDemo.m, An error occured : Undefined variable "caffe" or class "caffe.reset_all". Error in VerificationDemo(line 2)

[happynear]

You may not add the ./matlab to your MATLAB path list. try this:

addpath('caffe-windows-root/matlab`);

[cheer37]

Firstly, Relating to the execution matlab wrapper. I tried to addpath('F:/FreeSource/DeepLearning/caffe-windows-master/matlab/') But does not register it's subfolder called +caffe, so I tried to addpath('F:/FreeSource/DeepLearning/caffe-windows-master/matlab/+caffe') But warning happens - Warning: Package directories not allowed in MATLAB path:

also error happens - Invalid MEX-file '...\caffe-windows-master\matlab+caffe\private\caffe_.mexw64': The specified module could not be found. How can i solve these problems.

Secondly, Relating to the FeatureExtraction. I found the answer about how to feed a pair of two images as an input in net for verification. We can't do it, but we must extract features according each image at first, then calculate the distance between two image features, thus DNN is used to feature extraction for face verification. So, i am doing feature extraction using your windows port, but i can't figure the arguments to feed (command line)out, I investigated caffe official documents and your code, but i couldn't, Would you tell me? Thanks.

[happynear]

1. The Invalid MEX-file error is mostly caused by the third party dlls. You may still not add 3rdparty/bin to your environment variable PATH.
2. The testing model definition is in https://github.com/happynear/caffe-windows/blob/master/examples/FaceVerification/CASIA_train_test.prototxt . It takes two images as input and output their feature distance.

[cheer37]

i did addpath('caffe-windows-root/matlab`); But +caffe subfolder doesnt be added. so, it couldn't find caffe_.mexw64 file. prefixed at caffe +symbol disturb to be added, isn't it?

[happynear]

Folders with + symbol need not to be added to matlab's path.

[cheer37]

Then How can matlab find the caffe_.mexw64 file? Error follows.

VerificationDemo

Invalid MEX-file '...\caffe-windows-master\matlab+caffe\private\caffe_.mexw64': The specified module could not be found.

Error in caffe.resetall (line 5) caffe('reset');

Error in VerificationDemo (line 2) caffe.reset_all();

[happynear]

The Invalid MEX-file error is mostly caused by the third party dlls. You may still not add 3rdparty/bin to your system environment variable PATH.

[cheer37]

You are saying in visual studio, ain't you?, I added it already.

[happynear]

System environment variable.

You can check it by opening the ./bin/caffe.exe. Please check If their is some error messages popup.

[cheer37]

Oh, error happens requesting 3rdparty dlls. How do i register environment variable? I registered the 3rdparty dll's path in a user variable path. caffe.exe command doesnt popup error now, but error is the same in matlab faceverification project. What's the problem?

[cheer37]

@happynear I fixed that error, it's due to cuda sdk. I was using cuda sdk 7.0.18 and upgraded it to 7.0.28, so that error disappeared. But now, i get following error.

Undefined function or variable 'MatAlignment'. Error in VerificationDemo (line 19) MatAlignment('init_model','D:\ThirdPartyLibrary\dlib-18.14\shape_predictor_68_face_landmarks.dat'); Would you help me? Thanks.

[happynear]

This is the face alignment model from dlib. You can download it from dlib.net .

[cheer37]

I did it already. Error saying MatAlignment has not been defined. even if there exists cpp implementation. How can i deal with this problem?

[happynear]

This cpp, should firstly be compiled with OpenCV and dlib. Matlab can only run the compiled mexw64 file. The configuration of it is similar with caffe matlab wrapper.

[cheer37]

How can i build it?

[happynear]

Create a new project, adding dlib, opencv, matlab's include and library paths into it. Compile it as an DLL and rename it to MatAlignment.mexw64.

[cheer37]

It might causes some errors and take a time. You must have it, would you provide me with it? Thanks.

[happynear]

I have uploaded it : http://pan.baidu.com/s/1mgCPAbA .

[cheer37]

Thank you very much. Feng Wang.

[cheer37]

Feng wang. I downloaded it in FaceVerification folder and run. It causes invalid mex error like below.

Invalid MEX-file 'F:\FreeSource\DeepLearning\FaceVerification-master\MatAlignment.mexw64': The specified module could not be found.