Exporting face as UV Texture Map

[mufaawan]

I want to store image of detected face as Texture Map. How can I do that? I have identified all normalized landmarks using FaceMesh. How much can I acheive similar to below image?

https://i.pinimg.com/originals/b2/44/ca/b244ca9f645e1c8cf5f365e8f1c73123.jpg

[sgowroji]

Hi @mufaawan Could you please share details in which platform are you targeting the above.

[mufaawan]

@sgowroji I would like to have Python based solution. But C++ would also work for me.

[mufaawan]

@sgowroji waiting for response :)

[mgyong]

@kostyaby Any comments on this?

[kostyaby]

Hey @mufaawan,

What you are trying to do is a bit custom, so please be ready to code some additional logic. Luckily, you should be able to implement it in Python as the approach I'm about to suggest only requires the MediaPipe Face Mesh solution

First, you need'll some face topology that establishes semantic relationship between 3D positions of the face mesh vertices and some 2D texture coordinates. I'd start with the canonical face model used in MediaPipe Face Geometry module (UV visualization; Face Geometry formats: metadata schema, mesh 3D schema, metadata text). What you need at this point is XYZ <-> UV correspondence + triangular topology, so grabbing them from the Face Geometry metadata file is a good starting point for now (you can tweak the mapping / topology later if needed)

The next thing you need is to warp the image by applying UV -> XYZ mapping. You'll do it triangle-by-triangle of the chosen topology like in this OpenCV tutorial. First, use cv::getAffineTransform to compute an affine transform matrix for two sets of three 2D point (XYZ coords <-> UV coords for a single triangle). Then, use that matrix with cv::warpAffine to warp a triangular patch of the input camera stream into a triangle patch on your result image

Those steps should do the trick. Please, let me know if you have any questions!

[sgowroji]

Hi @mufaawan, Did you get a chance to go through the above comment. Thanks!

[sgowroji]

Hi @mufaawan, Assuming you got the clear understanding with the above comments, we are closing this issue. Please feel free to open if you still not clear.

[mufaawan]

@kostyaby I am here to thank you a lot for the great guidance. I have achieved what I wanted to. I worked last week on what you have guided me with. I really appreciate the details answer.

Thank you @sgowroji for routing my request to relevant person. :)

[JJBT]

@mufaawan could you please share a code-snippet that converts face to Texture Map. I would be really appreciate it.

[Hisham-Tariq]

@mufaawan Hi i need help in my FYP i am trying to create this as a Final Year project of my CS degree i need your help.

[CodingBitsDev]

I'm currently standing in front of the same problem. I'm struggling to understand the topology part of the explanation. A code example in how to use the face geometry metadata for this would be appreciated.

[kostyaby]

Hey @Renji3,

In graphics, there's a concept of Polygon mesh: in order to represent some 3D objects, you define 2 buffers: vertex buffer (defines vertices of the mesh) and index buffer (defines faces of the mesh; those connect the vertices into some topology). A commonly used kind of polygon meshes is a triangular mesh; for those meshes, faces are always 3D triangles and not any other complex shapes

What you need at this point is XYZ <-> UV correspondence + triangular topology, so grabbing them from the Face Geometry metadata file is a good starting point for now (you can tweak the mapping / topology later if needed)

What I meant by this is that there's a geometry_pipeline_metadata_from_landmarks.pbtxt file which enlists 468 canonical face mesh landmarks; each landmarks is described by 5 float values: 3D position (XYZ) and texture coordinate (UV). These vertex_buffer values have the 5 * 468 float values. First, you'd need to parse those to get the UV values (i.e. every 4th and 5th float values out of 5 consequent ones). Then, you get the XYZ coordinates from the MediaPipe FaceMesh solution - and now for each 3D position XYZ) you have a texture coordinate (UV)

Then, you need to parse these index_buffer values - the first 3 values describe the first triangular mesh face, the second 3 values describe the second triangular mesh face and so on

[apple2373]

Hi, I was able to implement this. https://github.com/apple2373/mediapipe-facemesh

[wizVR-zhangjun]

@mufaawan Hi teacher, I have trouble using CV ::getAffineTransform and CV ::warpAffine, can you tell me your core idea

[aishwarya123mathpati]

Hi Developers! I want to write or store the mediapipe 468 face landmarks to a FBX file format. I have tried with multiple ways but not able to get solution. I am able to generate obj file, in obj file I am having only face landmarks are there. I tried with fbx , pyfbx , python fbx sdk but none of the library has helped me to solve this issue.

Thanks in advance!

[mufaawan]

Hello, if you still need help you can reach me via email (mufaawan@gmail.com) . We can schedule a zoom call to help get your things done.

[mufaawan]

Hi Developers! I want to write or store the mediapipe 468 face landmarks to a FBX file format. I have tried with multiple ways but not able to get solution. I am able to generate obj file, in obj file I am having only face landmarks are there. I tried with fbx , pyfbx , python fbx sdk but none of the library has helped me to solve this issue.

Thanks in advance!

Hello, if you still need help you can reach me via email (mufaawan@gmail.com) . We can schedule a zoom call to help get your things done