search

DanielSWolf / rhubarb-lip-sync

Rhubarb Lip Sync is a command-line tool that automatically creates 2D mouth animation from voice recordings. You can use it for characters in computer games, in animated cartoons, or in any other project that requires animating mouths based on existing recordings.
Other
1.81k stars 216 forks source link

Can you direct me to where the source code is... #134

Closed mjoe67886 closed 8 months ago

mjoe67886 commented 8 months ago

I have a specific character where the phonetic library works pretty good but with her specific voice the pocketsphinks library does not but I just tested it with Google speech recognition API and I got a pretty good result for this character using its transcribed feature most of the text that this character said was recognized so I was thinking it would be pretty awesome to integrate Google into your code and I was going to write that real quick but I didn't see the source code up here would it be possible for you to supply it in return i'd be happy to return the finished code so you could also supply it here for other people to try.

mjoe67886 commented 8 months ago

Also I'm getting ready to make an entire lip syncing system based off your EXE also using open AI in my blender add on so if you could supply that source code... I wouldn't mind also... Giving you whatever I come up with As far as AI integration I don't know if you're a blender User but if you are I make the biggest blender add on in the world code wise it has tons of awesome features so I wouldn't mind throwing that in to sweeten the deal...$50 value

mjoe67886 commented 8 months ago

Actually I do apologize I just realized that I wasn't signed in to Github and I think as a result it didn't show your source code I just looked back and it was in there...

mjoe67886 commented 8 months ago

But all that being said I just looked how many files that you have and this is a pretty big repo as far as what you put into it you know about the code can you let me know what file your phonetic logic is in and you created all this so you're really good at all this stuff what do you think of my approach to possibly using Google in place of phonetic do you think it might get better results I'm sure you probably use pocket sphinx so you can use exe in more situations and with more applications.... But also you know about the code if I just went in and replaced the phonetic logic would it work or is your Code all over the place meaning what I have to like search around in all these files and figure stuff out and learn or can I just link it up to whatever variable in that one file and it's done

mjoe67886 commented 8 months ago

OKI learn more about the code I just ported it over as an update to my add on for using rhubarb with shape keys and I just have a feature request kinda that may help out your end product you know what I like to do to keep a good workflow is import you know my entire character's dialog at once like I imported her saying like three different things so I had three different clips that had you know 10 to 15 seconds of silence between them that separated these clips I created them in Ableton live from replica software....but... the way you do the adding the key frames you only add a mouthe closed at the very end if you would add a mouthe clothes or a rest pose at the very beginning then this would be strengthened for people like me that do this and you know when you're doing animation in the phonetic is only one I really know cuz that's what I just used but when the character you know finished saying a word you know it's common for them to close their mouths like one to two frames after and then open their mouths a lot of times I see a little bit more Keyframes not being closed properly if you know what I'm meaning like the characters done talking and the mouse should be closed but it's just merging into the next keyframe so just saying I would look over the mouse rest I really think with this functionality if the logic is fine tuned it's possible to get a perfect result every time but thank you very much for this awesome software....

mjoe67886 commented 8 months ago

I'll send a video example if you need it

DanielSWolf commented 8 months ago

Wow, that's a lot of thoughts at once! (And very little punctuation, which I personally found tough to read. 😉) To answer your questions:

mjoe67886 commented 8 months ago

OK this kind of syncs up after I make a few sales off of my openai addition to my blender add-on including automatic code testing/fixing... I'm getting ready to hire literally the best programmer I've ever met to write An auto lip syncing solution using open ai He's charging me $200 for my error checking fixing but it's real world class code if you're interested I could provide code examples where he does some amazing node recursion in blender... Using open AI code I plan to make it world class like something you would find on one of the new ai websites that come out.... You know the ones that charge $20 a month well then this way I could bundle this type of feature with my add on and if you wanted to pitch in on this and maybe um contribute some money toward this process this guy has quite a few good reviews on fiber i've hired him anytime.... You know that's always an option and you can assist in explaining your code and what we need to do to upgrade it successfully to whisper... If you wanted to contribute on your end and you know assist... Then that could curtail some of the costs... Yeah this guy's a game engine designer but together if we team up we could probably make a world class solution and really up the value of both of our products.. And if we find it's good enough maybe upload the source code to a website and create our own automatic lip sync competitor.... You know if we could figure it out better..... I'm not certain if you're making money off your product I know it is valued worldwide so at the very least this could improve your standing in the community otherwise you could just say hey I'm releasing rhubarb two this one costs money but feel free to upgrade or use rhubarb one as the free version....

On Sun, Jan 7, 2024 at 7:03 AM Daniel Wolf @.***> wrote:

Wow, that's a lot of thoughts at once! (And very little punctuation, which I personally found tough to read. 😉) To answer your questions:

  • I agree that PocketSphinx as a speech recognizer is rather outdated, with other libraries giving much better results. However, replacing the recognizer isn't straightforward since there is some interplay between the speech recognition and alignment part. That being said, I'm actively working on Rhubarb 2.0 (#121 https://github.com/DanielSWolf/rhubarb-lip-sync/issues/121), which will be a full rewrite based on modern technology. In particular, I plan on using a modern offline STT system like Whisper to combine the quality of Google STT with the offline support of PocketSphinx. However, given my very limited time, I can't make any promises as to when (if ever) it will be done.
  • For much the same reason, I'm not planning to make any more changes to Rhubarb 1 except for bug fixes.
  • Feel free to embed Rhubarb in your Blender plugin, as long as you properly attribute it.
  • I'm not sure whether I understand you correctly regarding closed mouth shapes. Rhubarb does support X as a closed mouth, and it will use this mouth shape whenever the character isn't saying anything. Only for very short silences, it will leave the mouth open for a more natural result. If you experience a different behavior, please report it as a bug, preferably in a separate issue.

— Reply to this email directly, view it on GitHub https://github.com/DanielSWolf/rhubarb-lip-sync/issues/134#issuecomment-1880083875, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWFQGIIFVCZU3WC5ACEYCZTYNK2MNAVCNFSM6AAAAABBPX3FUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBQGA4DGOBXGU . You are receiving this because you authored the thread.Message ID: @.***>

mjoe67886 commented 8 months ago

OK Yeah I just noticed what you said about the punctuation yeah I am very very busy I have the biggest blender add on in the world code wise which I can provide to add a little power to my idea about a collaboration.... I'll try and keep these messages concise that'll make them easier to read.... But if you're interested you could look up fast animation studio toolz on blender market.... I don't have a huge amount of sales yet because I've been working on the developing for the last year but I'm just starting once I get these AI feature set up to go full boar into advertising.... This add on really is epic it's like almost two gigabytes of pure code ... It's a tool that makes television production in blender a possibility for one person... And adding the auto lip sync functionality will remove the dependency my add on has on the FACEIT addon which will make it a little bit easier for my end users to approach than having to buy another $79 add some of which which are important for my workflow because other blender add-ons like extreme P B R can't easily be replicated and are necessary...... In my vision of the end result it did include a linking up to ARKIT shape keys... I don't know if you would consider that to be on my end or if you would like to dip your toes in that section the only other issue would be to how to create the arkit shape keys like the FACEIT add-on does... As it is now it costs $79 to get their addon and create the arkit shape keys easily so we at least need to find a cheaper solution...But the end result we could share as far as the blender functionality whatever functionality that we create as a team you could have that part as your own separate blender add-on and sell it separately and I could implement it into my own add on as an extra feature... Just some ideas we both seem to be in a similar situation....

DanielSWolf commented 8 months ago

I'm sorry, but I'm busy, too, and I refuse to spend my time trying to decipher this kind of spat-out mess.

mjoe67886 commented 8 months ago

No problem I'm going to create this soon...and if it becomes necessary anywhere in the process to update your code i'll send whatever fixes I make. Don't worry I won't send anything that isn't functional well written and definite benefit... I just realized that it's probably going to be necessary to upgrade it to use open AI whisper so if you want you could just hold off on that and I'll send the end result when I get it done.