search

Rudrabha / Wav2Lip

This repository contains the codes of "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild", published at ACM Multimedia 2020. For HD commercial model, please try out Sync Labs
https://synclabs.so
10.18k stars 2.19k forks source link

Video face is shaking #226

Closed DWCTOD closed 3 years ago

DWCTOD commented 3 years ago

Hi，I'm sorry to brother you. When I was using this project to train model in my dataset, the face is shaking in the result video. I don't know what happen. Maybe the problem is due to "Training the expert discriminator" ? My dataset is only one person, total time about 1h. train loss : 0.01 and syncnet loss is more small. Train wav2lip.py result is look very good, but when i inference the result is not very good.

Could you help me ,how can I try to fix this problem. Thanks very much

with torch.no_grad():
    average_sync_loss = eval_model(test_data_loader, global_step, device, model, checkpoint_dir)

 if average_sync_loss < .75:
    hparams.set_hparam('syncnet_wt', 0.01) # without image GAN a lesser weight is sufficient

Maybe that casue face shaking. Because I hadn't trained this syncnet model, only train l1_loss!

prajwalkr commented 3 years ago

There could be a lot of issues when training on your own data. Have you sync-corrected your data first?

prajwaljpj commented 3 years ago

@prajwalkr Is there a Model you use for Sync Correction? if so could you please point me to one. I have the same issue. The face is flickering a lot. Thanks in advance!