search

Rudrabha / Lip2Wav

This is the repository containing codes for our CVPR, 2020 paper titled "Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis"
MIT License
692 stars 152 forks source link

Great project! #11

Closed sberryman closed 4 years ago

sberryman commented 4 years ago

Excellent work and I love that you used a synthesized voice for the project video. I spent a lot of time on Corentin's Real Time Voice Cloning project and noticed you used some of his code as your base.

Just wondering what you plan on doing from this point with the project. Are you going to continue working on it or have you already moved on to new projects?

Rudrabha commented 4 years ago

Hello, Thanks a lot for your interest in our work. Though we do have plans to improve this work later, we are not working on the project at the moment. This work was done around October last year and since then we have moved on to a couple of other projects which are under review at the moment.

sberryman commented 4 years ago

@Rudrabha, thanks for the quick reply. Anything you can share at the moment? I'm really looking forward to see what you do next!

Are you still working on a cross between the vision and audio domain? I would love to see a GAN similar to talking heads which has accurate lip sync to synthetic audio.

For context, I started training a talking heads model based on this code. It has been training for about 44 hours and is on ~Epoch 5.4.

The model has just started learning about baseball caps, glasses and facial hair. Skin is still very smooth and it struggles to capture facial aging. Nose shape is progressing as well as open and closed eye lids. Eye gaze seems be for straight forward in all the generated images.

Example 1:

image

Example 2:

Screenshot-20200710112120-379x536

Example 3:

Screenshot-20200710111120-372x500

Example 4:

Screenshot-20200709125514-380x546

Rudrabha commented 4 years ago

You can check out our work that was published in ACM Multimedia here. For any related query/comment on the lipsync project, please open an issue in the LipGAN repository.

sberryman commented 4 years ago

Well look at that, you already built it! Project video is impressive, thanks for pointing me to the LipGAN project.