Textured mesh mapping of lips is lost

[alter-sachin]

Hello Great work! I have used this repo and other flame repos to create a textured mesh animation with sound. I would request your help in the texture mapping coming out as more realistic.

As you can see the first image is a textured mesh and when it is animated the result is the next picture.

As you must notice the lip region is troublesome, a particular region that should be textured like the skin get a texture of the face. I have used the TF_FLAME repo to map textures onto all meshes generated by VOCA.

Best Regards

[TimoBolkart]

Hello,

can you please provide some more details on what you did for this? Did you map the image of an altered geometry? Why is the lip mapped on the mesh in the above image but not in the image below?

[alter-sachin]

Sure, I used the data in TF_FLAME with the image and the npy stored with landmarks to generate the result.

python fit_2D_landmarks.py --model_fname './models/female_model.pkl' --flame_lmk_path './data/flame_static_embedding.pkl' --texture_mapping './data/texture_data_512.npy' --target_img_path './data/imgHQ00088.jpeg' --target_lmk_path './data/imgHQ00088_lmks.npy' --out_path './results'

This result as you must know contains an OBJ file, an MTL file and texture file.

Now I take this result obj file and generate VOCA meshes that creates a set of meshes that is animated according the the wav input.

Now in order to create a textured voice animation I take these set of meshes and run build_texture_from_image.py on all of these meshes https://github.com/TimoBolkart/TF_FLAME/blob/master/build_texture_from_image.py

Then i imported the series of meshes into blender and created a talking animation with textures. This seemed like the only way to have a talking textured person from all the repositories related flame and voca. I would be glad to receive your guidance and sort this issue.

[TimoBolkart]

The function build_texture_from_image.py takes an image and projects it into a mesh to create a texture map. This requires the 3D mesh and the image to be in correspondence, established by fitting the model to the image. Instead of building a texture for each frame of the VOCA generated sequence (which is wrong) just take the fixed texture and combine it with the VOCA generated mesh for visualization. There is no need to update the texture for every frame. Please let me know if this is unclear.

[alter-sachin]

Yes, thank you for replying. I did realise the code was projecting that single texture onto the different meshes.

I am confused when you say "just take the fixed texture and combine it with the VOCA generated mesh for visualization" . I could not point out how to do this.

I tried using the same texture and imported it to the new OBJ's generated by VOCA but blender does not show them. On applying them manually as a image texture the results are not right. I get distorted textures

I have been trying to figure this out. Could you guide me what I am missing ?

[TimoBolkart]

Here some code that should do the job. Just specify the path of the source textured FLAME mesh (texture_mesh_fname), the FLAME texture (texture_img_fname) and the path of the VOCA result meshes. The code outputs in another folder all meshes with applied texture on.

import os                                                                                                                                                                                          
import glob
from psbody.mesh import Mesh                                                                                                                                                                           

# Replace the file name of the textured mesh the texture image, and the result folder of the VOCA meshes
texture_mesh_fname = 'TF_FLAME/results/imgHQ01148.obj'                                                                                                                      
texture_img_fname = 'TF_FLAME/results/imgHQ01148.png'     
voca_result_path = 'animation_output/meshes'                                                                                                                                                           

out_path = voca_result_path + '_textured'                                                                                                                                                              
if not os.path.exists(out_path): os.makedirs(out_path)                                                                                                                                                 

texture_mesh = Mesh(filename=texture_mesh_fname)                                                                                                                                                       
voca_mesh_fnames = sorted(glob.glob(os.path.join(voca_result_path, '*.obj')))                                                                                                                          

for fname in voca_mesh_fnames: 
    voca_mesh = Mesh(filename=fname) 
    voca_mesh.vt = texture_mesh.vt 
    voca_mesh.ft = texture_mesh.ft 
    voca_mesh.set_texture_image(texture_img_fname) 
    out_fname = os.path.join(out_path, os.path.splitext(os.path.basename(fname))[0] + '.obj') 
    voca_mesh.write_obj(out_fname) 

[Augnine]

@TimoBolkart Thanks for your code，and I have another problem Now I get textured mesh such as the following filess： 00000.mtl，00000.obj，00000.png ，......，00305.mtl，00305.obj，00305.png But When I want to generate video with sound，I find the result is without texture .How to slove it?

[TimoBolkart]

I added a demo to the VOCA repository for providing a texture optionally to the inference code. Running this outputs textured meshes and renders the textured sequence.