search

morevnaproject-org / papagayo-ng

Papagayo is a lip-syncing program designed to help you line up phonemes (mouth shapes) with the actual recorded sound of actors speaking. Papagayo makes it easy to lip sync animated characters by making the process very simple - just type in the words being spoken (or copy/paste them from the animation's script), then drag the words on top of the sound's waveform until they line up with the proper sounds.
236 stars 51 forks source link

Integration with rhubarb-lip-sync? #44

Closed morevnaproject closed 3 years ago

morevnaproject commented 6 years ago

Interesting project to consider - https://github.com/DanielSWolf/rhubarb-lip-sync

morevnaproject commented 5 years ago

We've got Rhubarb feature merged just now. - #50 ^__^

towfiqi commented 5 years ago

@morevnaproject Great! Can you kindly let me know how you are converting the phonemes to rhubarb shapes? I am trying to add a function to my project to convert the cmu-sphinx phonemes to rhubarb shapes.

I can get the phonemes from the cmu dictionary along with the time for each phonemes: { word: 'THANK', value: 'TH AE NG K' } and I can get the shape set for the word:

A   0
C   0.26
A   0.46
B   0.065

What I am doing is grouping each shapes to some phonemes and look it up for each word's phoneme set. It works somewhat, but there are some issues sometimes:

    /*A*/ ["P", "B", "M"],
    /*B*/ ["K", "S", "T", "EE", "IY", "IH"],
    /*C*/ ["EH", "AE", "AH", "Schwa", "EY", "AY", "HH", "G", "CH", "JH", "R", "Y"],
    /*D*/ ["AA"],
    /*E*/ ["AO", "ER", "SH", "ZH"],
    /*F*/ ["UW", "OW", "W", "UH", "AW", "OY"],
    /*G*/ ["F", "V"],
    /*H*/ ["L", "N", "NG", "T", "D", "TH", "DH", "S", "Z", "D"],

my question now is how I can properly convert this to correct rhubarb shape set. There are some differences with Rhubarb's results and mine. I contacted Rhubarb author and he said, the algorithm is more complex than a simple lookup and suggested me to look into papagayo-ng.

It would be great if you could let me know what logic you are using for converting the phonemes to rhubarb shapes. I tried looking into the code, but I dont know python, so couldn't figure it out. :cry:

Thanks

luzpaz commented 5 years ago

nice