Explain about parameters of Candide face model

[vudangthinh]

I saw that on the function you write to read parameters from candide.npz

def load3DFaceModel(filename):
    faceModelFile = np.load(filename)
    mean3DShape = faceModelFile["mean3DShape"]
    mesh = faceModelFile["mesh"]
    idxs3D = faceModelFile["idxs3D"]
    idxs2D = faceModelFile["idxs2D"]
    blendshapes = faceModelFile["blendshapes"]
    mesh = fixMeshWinding(mesh, mean3DShape)
    return mean3DShape, blendshapes, mesh, idxs3D, idxs2D

It is seem that content of file candide.npz is different from candide file from http://www.icg.isy.liu.se/candide/candide3.wfm And can you explain the meaning of mean3DShape, mesh, idxs3D, idxs2D, blendshapes array?

[MarekKowalski]

Hi,

The content of candide.npz is basically a processed version of the original candide model you posted in your link.

• mean3DShape: this corresponds to the neutral state of the face, which is the vertex list in Candide.
• mesh: the original mesh provided by Candide as face list.
• idxs3D and idxs2D: those are manually marked indices of corresponding points between the Candide model (idxs3D) and the face alignment point set (idxs2D). An example of a corresponding point would be the left outer eye corner. Both Candide and the face alignment point set have that model marked, thus its index is probably specified in idxs3D and idxs2D.
• blendshapes: blendshapes are shapes you can add to the Candide neutral state face to modify it. An example of a blendshape would be a smile or an eye brow raise. The blendshapes are derived from the Animation Units defined in Candide.

If you have any other questions, or if that was not clear, please let me know.

Marek

[vudangthinh]

Thank you so much for answering me.

I used this program to swap my face with a picture, but I noticed that when my eyes close, the eyes of swapped face are not closed.

Thought of the reason is that face model lacked of Shape Units, so that I added Shape Units parameters to optimize the accuracy of face model when fitting it in landmark points. But even when did that the eyes are still not closed.

Do you think the reason of this is related to when fitting Candide face model or swapping faces? And how to fix that? Thanks,

[MarekKowalski]

Hi,

Congratulations on converting the app to C++! There might be several reasons for this behaviour:

• The 3D model vertices that are moved by the shape unit you added are not connected to any 2D landmark vertices. You would have to looks at idxs3D and check.
• The landmark detection does not actually represent the eyelid closure. To check that you would have to draw the detected landmarks and see if the landmarks on the eyes actually "close" when you close your eyes.
• If it is none of the two above it might be a problem with the texture warping around the eye region. For example if the eyelid region is very small on the photo you are swapping, there might be some issues, not sure about that though.

What I would suggest is that you try removing the triangles of the mesh that are located in the eye region so there are holes in those areas of the model. If you do that the swapped face will show the user's real eyes as it does with the mouth. This might prove more realistic than the current effect.

Let me know what you did and how it went.

Marek

[vudangthinh]

Thank you,

I will try your suggestions and tell you what I found. However, I checked vertices in shape units. There are some vertices connected to 2D landmark vertices. Besides, when fitting 3D face model in 2D landmark vertices with Shape Units, it become fitter.

[abhilashi]

@vudangthinh Great work converting the code base into C++. Are you planning to open source the implementation in c++? We intend to do something similar. We would like to build atop of your work instead of re-inventing the wheel! How does that sound to you?

@MarekKowalski Apologies for hijacking the thread :( No private messaging on Github yet.

[MarekKowalski]

@vudangthinh Did you add any new shape units that would close the eyelids? If not, that might be the issue., I'm not sure if the original model had any. If you did, make sure that the vertices in those are attached to the 2D landmarks.

[vudangthinh]

@abhilashi Sorry, I cannot open source it. However, I think with the source by Marek, you can easy to convert it to C++. If you have any private messages, you can email to me thinhbn2408@gmail.com

@MarekKowalski When adding shape units, the eyelids close a little bit, but it's not absolutely close.

[ShirleyYim]

@tvd2840 hello，when convert the app to C++, do you use opengl to subsitute the pyOpenGL and pygame related part?

[vudangthinh]

Yeah, it boosts the speed of swapping.

[xiexiexxs]

@MarekKowalski hi ,from your answer,i got that you make some blendshapes by changing shape unit parameters or action units parameters in candide , and then using as the fitting funtion instead of the one in original candide model, am i right? Additionally, i have a question how to fit the original function with shape unit parameters and action units parameters ? what's the dimension(row,colume) of the shape unit parameters? thanks.

[DmitryBorin]

Hello!

Could you please share the C++ implementation? Thanks!

[MarekKowalski]

@xiexiexxs Hi, I convert the shape and action units into blendshapes, which are S_i in the equation that you linked. As for the fitting function, yes that is the one you linked.

The dimensionality of the blendshapes is the same as S_0, I don't really remember what it is. You can check it in the code however, the name of the variable is meanShape3D or something like that.

@DmitryBorin Hi, one of my other repositories https://github.com/MarekKowalski/HoloFace implements a similar functionality in C++. Look at the Fit3D file under the link below: https://github.com/MarekKowalski/HoloFace/blob/master/Libraries/src/fit3D.cpp

Marek

[DmitryBorin]

@MarekKowalski Thank you for your answer! I watched this implementation, but I would like to test it on Linux. So, maybe you have an implementation of the basic functionality (fitting with candide, find rotation/translation, etc) without special Windows libs? Thanks!

[MarekKowalski]

Hi,

I do not have any other implementations. I am however pretty sure that none of the model fitting code actually requires windows-only libraries.

Marek

[NoodlesVision]

@MarekKowalski hi, you said that "I convert the shape and action units into blendshapes", counld you plz explain how to do it ? which shape units and animation units did you pick? and why?

also you said "blendshapes: blendshapes are shapes you can add to the Candide neutral state face to modify it. An example of a blendshape would be a smile or an eye brow raise. The blendshapes are derived from the Animation Units defined in Candide."

i am confused about how are the blendshapes from? thanks?

[MarekKowalski]

Hi,

Below I am pasting a link to a very "dirty" script that does the conversion to blendshapes, so you can check it out yourself. https://www.dropbox.com/s/jbxqvm4o0qwv4fb/candide.py?dl=0

Best regards

Marek

[match08]

candide.py open candide.txt? what is candide.txt?

[MarekKowalski]

Candide.txt is a file describing the CANDIDE face model. Yes, the Python script opens that file. It also processes it into an .npz file.

[longhopefor]

hello, sorry for asking how can i add some blendshapes by myself, such as fit the glasses, is there some example?

[MarekKowalski]

In order to add glasses you would have to add more than just blendshapes. You would have to add them to the mesh. You could do this by saving the mesh to a format than can be read by some 3D modelling software (like Blender) and editing it there. You would then need to be able to load the mesh back into the app.

[longhopefor]

thank you very much for you reply！