This code, in theory, would make excellent VTuber software

Rudrabha / Wav2Lip

This repository contains the codes of "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild", published at ACM Multimedia 2020. For HD commercial model, please try out Sync Labs

https://synclabs.so

10.85k stars 2.3k forks source link

This code, in theory, would make excellent VTuber software #379

Open NeuroNoNeuro opened 2 years ago

NeuroNoNeuro commented 2 years ago

For folks like me, who do not want to use a microphone, but would prefer TTS audio files - it would be very nice to simply make a rudimentary animation, use an open-source TTS audio file (.wav, .mp3, etc.), and sync the two together. Is there any way one could do this outside of the website, as an application, not by coding?

lostvikx commented 2 years ago

Check Google Colab link which is provided, also watch the YouTube video.

JQuezada0 commented 2 years ago

@NeuroNoNeuro Was thinking about this as well. Especially in combination with https://github.com/CorentinJ/Real-Time-Voice-Cloning; one could clone their own voice so it's at least personalized.

I've gotten that voice cloning working, but never dockerized, and I just got this repo working on AWS EC2. Stringing together an application from this is entirely possible, but I'd like to investigate what the actual costs are, especially if a GPU is required.