facebookresearch / meshtalk

Code for MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
Other
368 stars 56 forks source link

Question about failing to animate face #38

Closed CIntellifusion closed 1 year ago

CIntellifusion commented 1 year ago

Hello, I have tried to use my .obj file to train meshtalk. And I have done training. But when I render the face, the output mp4 file is totally white.

I checked the the vertices loaded in the render.py , they are alright.
I also trained meshtalk just using zero tensors on the original project. And it's all right.

I was wondering that you may have some idea about failing to rendering??

Thanks.

CIntellifusion commented 1 year ago

I think this issue may have something to do with function render in renderer.py.

    if not video_output[-4:] == '.mp4':
        video_output = video_output + '.mp4'
    images = th.cat([self.render(v).cpu() for v in th.split(verts, batch_size)], dim=0)
    images = 255 * images[:, :, :, :3].contiguous().numpy()
    print("images ",images.shape)
    print("view", np.sum(images != 255))

the number of pixels that are not white is very low...

CIntellifusion commented 1 year ago

well, I tried to render video using predicted vertices using other render functions. It seems that the predcition is wrong. So this is may has nothing to do with animate_face.py

CIntellifusion commented 1 year ago

I found the problem lying in load_template function in helpers.py . I guess there might be some subtle difference between .obj file and .fly file. So I implemented this function by using .ply file. And it seems work.