search

Rudrabha / Wav2Lip

This repository contains the codes of "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild", published at ACM Multimedia 2020. For HD commercial model, please try out Sync Labs
https://synclabs.so
10.28k stars 2.21k forks source link

SyncNet converging too quickly for LRS2 #293

Closed Mayur28 closed 3 years ago

Mayur28 commented 3 years ago

Hi,

I am trying to train the expert lip sync discriminator using LRS2 and using the code from the repo as is (without any changes), but I am unsure why is the model converging much quicker than it should be. Default parameters are used. As mentioned in one of the previous issues, I was expecting the loss to be around 0.69 for quite some time, and to only start to decrease after 150K steps. I have a loss curve that I am getting. image

Any assistance would be highly appreciated. Thanks

zhanghm1995 commented 3 years ago

@Mayur28 , Hi, did you use the LRS2 dataset or other dataset? The loss plot was drawed by tensorboard?

Mayur28 commented 3 years ago

Hi @zhanghming,

I was using LRS2 and I was using weights and biases to plot the loss function. I realized my mistake when I posted my previous mistake - I erroneously ran my code on my small subset of the dataset instead of the full dataset, therefore the model converged quickly. Apologies for that.

zhanghm1995 commented 2 years ago

@Mayur28 , Hi, thanks for you reply. I wonder whether you continued do some work based on this repo? Have you re-implemented the performance similar with the author's?

Mayur28 commented 2 years ago

Hi @zhanghm1995 ,

I have continued to work and extend the code in the repo - My re-implementation is able to double the performance of the code in the repo. From my several executions of the code in the repo, I am only able to achieve 40% GPU utilization at most, and with my implementation, I can achieve 90% on a GTX 3090 GPU. Unfortunately I am unable to share my implementation as it is currently being used for my research. Once I am done, I may consider making it public.

xopclabs commented 2 years ago

Hi @Mayur28! How is your research work going? I'm quite interested in yours implementation of wav2lip. Do you plan on open-sourcing it any time soon? Thanks anyway!

Mayur28 commented 2 years ago

Hi @xopclabs ,

My research has been delayed by a few months unfortunately, I am expecting to complete my research by the end of the year and at that point I will decide whether I will open-source the implementation or not. I will keep you posted.