search

DanielSWolf / rhubarb-lip-sync

Rhubarb Lip Sync is a command-line tool that automatically creates 2D mouth animation from voice recordings. You can use it for characters in computer games, in animated cartoons, or in any other project that requires animating mouths based on existing recordings.
Other
1.71k stars 208 forks source link

How to integrate this library in react or nodejs? #133

Closed Aditya16828 closed 5 months ago

Aditya16828 commented 8 months ago

I am creating a live interactive 3D chatbot that would give creative answers to the users just like a human. Most of my work is done and I am just stuck in doing the lipsync part with the response audio that is received from my dialogflow model. Can someone plz help me out with a solution?

DanielSWolf commented 8 months ago

Rhubarb isn't a library, it's a command-line tool. All you need to do is execute it from your node process. The CLI is documented in the README file.

Aditya16828 commented 6 months ago

Yes I get it, but when I am using it to create a web-app, fetching audio response from the api calls involved and then running the cli command through the node application, it is taking a lot of time. Is there any way to directly use Rhubarb LipSync and also reduce the time involved.
The flow of my application is currently like:
Fetch the audio -> convert it to .wav format -> save it -> run the rhubarb cli command to generate lip shapes -> use the json file to perform the lipsync.
Now, this is used only once, and every single time my api is called, a new audio will be fetched and hence my previous output is completely of no use.
So, I want to optimise this. Can you plz help me out.

deckarep commented 6 months ago

@Aditya16828 - I am just a random guy on the internet that has used this library quite successfully in the past.

First you have to understand that this is designed as a command line application that processes audio files to generate lip-sync que points. This processing and analyzing of the audio files is somewhat CPU intensive.

I don’t know the details since I’m not the author but that time-consuming processing is where the magic happens. To ask the original author to optimize it is not really their problem and that’s not how open source works. If you need it optimized it’s open source!

With that aside: if you still want to use this in a node web app you must realize that node is an asynchronous framework and will choke on long running processes like calling out to a cpu heavy process. It can’t do that and service other http requests because it will be blocked waiting on rhubarb to complete. The whole nodejs event loop will be locked up.

So to get around this limitation you must ensure that rhubarb is invoked from an OS thread. This thread will be able to wait for Rhubarb to finish. This will allow your node app to still be snappy and performant and allow other http requests to work concurrently.

It’s not a matter of optimizing rhubarb. It’s a matter of you understanding different execution models and managing a CPU workload (the Rhubarb app) correctly and then understanding how to correctly synchronize your workflow to coordinate when it runs to completion and then getting the results out.

All of this is meant to help you get started.

DanielSWolf commented 5 months ago

Thanks @deckarep for the excellent explanation!

I'm closing the issue, since I don't have plans to turn Rhubarb into a library consumable from a Node process.