DanielSWolf
commented
6 years ago Hi Dave,
Live lip sync for a Cosplay costume certainly sounds interesting! :-)
The problem
Let me start by saying that Rhubarb Lip Sync cannot do what you need. Rhubarb is not a real-time appliation but always works on "utterances" (segments of typically a few seconds length). To work in real time, one would have to rewrite the entire application.
In addition, even if you were to write a new application from scratch, I'm fairly certain that its results wouldn't really look convincing. Not matter how good your application is, the mouth animation would always appear to be late and out of sync.
Let me elaborate on this. Speech consists of phones (that is, basic sounds). For instance, the word "glue" consists of the three phones G, L, OO. People often assume that each phone has a single corresponding mouth shape. That isn't the case. What really happens most of the time is that the lips "anticipate" the next vowel, that is, they move early. Watch yourself in a mirror saying "glue". (Important: say it quickly, as in regular conversation; don't over-enunciate!) You'll notice that your lips immediately form the OO shape while you're still saying the G phone.
Your brain knows what the next vowel will be, so your lips will form it ahead of time. A program doing real-time lip sync cannot possibly know what vowel will follow. The word "glee", for instance, also starts with G and L. There is no audible difference in these consonants, but the mouth would have to form a completely different shape.
Here's a different form of the problem. Take the word "apparently". Obviously, the mouth has to be open for the first vowel, then closed for the P phone, then open again for the second vowel. The thing is that P is a so-called plosive consonant. That is, the sound isn't made with your mouth closed (like an M, for instance), but the sound is generated when your lips open. So to make a P sound, your mouth first has to close, then open, and then you hear the sound. That means that any program that performs real-time lip sync based on what it hears will be much too late. By the time it hears the P sound, the mouth is supposed to have closed and opened again. Closing it now will look severly out-of-sync.
Here are some more temporal factors working against you:
- Unless you're working in hardware, you'll always have some latency in signal processing.
- I don't have any practical experience with servo motors, but I guess it must take them in the order of 100ms to go from fully open to fully closed mouth.
- There is a general rule in traditional animation: When in doubt, animate ahead of time, but never late. So if your animatronic mouth were always 100ms early, nobody would notice. Make it 100ms late, and it looks out of sync.
Solution 1: Optimization
Now that I've explained the problem, let me try to show two options that might work. First, if you are determined to use a microphone-based approach, what you need is real-time phone recognition. Googling this term will give you a number of scientific papers on the matter. It certainly seems possible to get it to work with minimal delay; see this video for a demonstration.
Once you receive a phone, you cannot afford to wait for additional phones in order to make more informed decisions. So you'll probably get best results with a simple lookup table that assigns one mouth shape to each phone. For inspiration, you can check out the source code of Rhubarb 0.2.0. This very early version still used such a simple lookup table. The mouth shapes A through H referenced in the code are documented in the current README file.
Solution 2: Monitoring the mouth
As we've seen, the mouth is always faster than audible speech. So I believe that a better approach would be monitoring the mouth. If there is enough room in the mask, you might be able to implement a camera-based system. First, apply some marker points to the mouth to simplify tracking. You only need the vertical and horizontal opening, so four markers may suffice. Then, use a macro camera to film the mouth at close distance, and a library like OpenCV for feature tracking. (Here is a video showing that the Raspberry Pi camera can easily be turned into a macro camera.)
If the mask is large enough that you can position the camera at a few centimeters from the mouth, this approach might work quite nicely. As an alternative, you could use some kind of mechanic sensors. I have no experience with this, but I'm confident that there are sensors which allow you to reliably track the movements of four points on the cosplayer's face.
marcwolf1960
Hi. I found your Github with the Rhubarb Lip Sync app on it and I was wondering if you could give me some advice.
I do animatronics for Cos-Play and other amateur/hobbies applications. One that I have been working on for a long time is a way to take a continuous real time speech stream from a microphone, and generate a number corresponding to the lip shape. From there a secondary processer can take this number and manipulate servo’s to give an approximation on an animatronic character.
For instance.. When they are making a movie where the main character is a werewolf consisting of an actor in the suit, and a team of puppeteers conforming the lips and facial expressions using remote control. All of this has to be carefully scripted or put in afterwards as CGI.
However take the same character and put them in a live performance situation i.e. a Cos-Play convention and you do not have that flexibility. I already have ways to pick up the actors facial movements underneath the costume. But getting a small self-contain Real Time Lip Sync is beyond me.
Could Rhubarb be compiled in Mono to run on an RPi or similar. The output being just a stream of numbers covering the lip shapes. The input would be a microphone worn by the actor.
Any suggestions would be greatly appreciated.
Many thanks Dave