mouth clarity

[varunjain99]

I'm getting a slightly blurry / artifacty mouth like in the image below

Things I tried that didn't help:

• different amounts of lip finetuning
• denoising training and inference audio

Any ideas on how to improve mouth clarity and make lips less blurry?

[mogupta]

@varunjain99 https://github.com/ZiqiaoPeng/SyncTalk/issues/211 Please share the set of commands you used for training ? I am also starting our training. may be we can help each other.

[varunjain99]

@mogupta I've tried various setups.

Mainly, I'm using the standard commands in the readme for ave training

• first preprocessing
• training first stage to 60k iterations
• finetuning to 100k iterations

[mogupta]

@varunjain99 This is the closest I could et today. Will try torso training tomorrow https://github.com/user-attachments/assets/1e8a3ea6-050a-4cf9-8f84-928bc780131c

[schxnhxlz]

Try different encoders like deepspeech or hubert. Its also important to have video matieral where the mouth is closed.