search

animate1978 / MB-Lab

MB-Lab is a character creation tool for Blender 4.0 and above, based off ManuelBastioniLAB
Other
1.85k stars 314 forks source link

Automated Lip-Sync #95

Closed amirpavlo closed 5 years ago

amirpavlo commented 5 years ago

I'm working on a lip-sync animation extension to MB-Lab. I have it working in my repository (https://github.com/amirpavlo/manuelbastionilab). Unfortunately, it's only for linux at this point. I'm posting here to get feedback on what you think of the interface, any changes? improvements on the work flow?

This is an initial look at the feature. https://www.youtube.com/watch?v=aS6Q59jU9Kk

I'll add more description on to use it in a bit.

telnetr4 commented 5 years ago

Looks good so far. Are you going to work on a Windows version?

As for the interface it's good. IMO we really need to make the rest of the MB-Lab UI better.

amirpavlo commented 5 years ago

I agree regarding the UI. I was thinking of doing a context sensitive pie menu sort of like boxcutter, but there is a bunch of polishing to do on the functionality of the lip-syncing, to end up with a better result. I think I'm gonna tackle that first.

Basically there are two work flows:

  1. Completely automatic: Add face rig. Add wave. Add transcript. Mark. Animate. With these 5 clicks you can get an initial animation.
  2. Marker by Marker. Add face rig. Add wave. Add transcript. Mark. Then you can go marker by marker, add the keyframes, then tweak, add key frames, tweak. You can also delete frames at a specific marker, and redo.
animate1978 commented 5 years ago

Oh that is cool

amirpavlo commented 5 years ago

I pushed the first draft of the Automatic lip-sync feature. Note, it only works for linux. I don't have a windows platform so I haven't tested it. But it should just print out a label under the "Speech Parser" section of the MB-Lab tab saying "Only linux support". If someone can give it a whirl on a non-linux platform to make sure it works before merging it, would be great

And if anyone finds value in porting this feature to windows or mac, it should be straightforward. All you'll need to do is:

  1. Port this to windows: https://github.com/amirpavlo/YASP -> code should work, just will need to be built for windows
  2. Add the windows libraries to the MB-Lab add-on and update the mb_yasp.py to load the windows libraries if a windows platform is detected. The code infrastructure is there.

I'm continuing to test it, so I'll probably be making further changes.

Any feedback ideas would be nice.

Here is a video of me playing around to show how lip-sync and face rig interact. It's low FPS, so the lip-sync isn't very smooth. https://www.youtube.com/watch?v=yC2CpZS1WvI&t