Update Request: Work with Blender 3.2

[gamedesign1]

This is not an issue, but an update request. Just wondering if you had plans on updating this addon for Blender 3.2 :)

[KaijuKoder]

First @scaredyfish thank you very much for blender-rhubarb-lipsync!

Second on issue #45, what gamedesign1 may be talking about is that the BlenderFoundation seem to have disabled the ability to create new pose libraries in 3.2 which /blender-rhubarb-lipsync uses. Though you can still use existing pose libraries so a workaround is to open your file with a Blender 3.1 executable - fortunately the blend files are compatible - create and enter the pose library there, then reopen it with Blender 3.2.

I checked the code and doc to see what is going on. https://docs.blender.org/api/current/bpy.ops.poselib.html

This says that bpy.ops.poselib.apply_pose() is 'legacy' and will be removed in Blender 3.3. Perhaps they disabled the panel to warn of this. The documentation implies it should be replaced with bpy.ops.poselib.apply_pose_asset() which supports the new Asset Library. Unfortunately the rest of the documentation isn't very clear about how to do this though it seems it can still be done through poselib but using newer calls. I have my own scripts to convert too so will do some research. If anyone else works this out please post.

[gamedesign1]

That's great you are looking into it. I really appreciate your work on this addon. I used it for the first time the other day and it worked so smoothly :) Great job :)

[aknittel1]

I have Blender 3.2.1 and significant work put into poses in the new pose library. I'm not willing to go back to the old pose library system.

So I have to ask as I am just about to install this add-on.: Will a new version of this add-on be available to support the new pose library and do you have an estimate when that might be available?

[scaredyfish]

No estimate, unfortunately. I have not had much time to work on this free addon lately.

[Skidborg]

Is there a work around where the visemes could be exposed on keyframes as a value that could then be hooked up to a geo nodes or driver setup?

[aknittel1]

I could fund this change if you're interested. The only alternative is Papagayo, and I don't think that's even compatible with 3.0 no less 3.2 pose libraries.

Please send me a quote. I would not restrict the release with the change to the community.

[aknittel1]

@KaijuKoder, if you're game and experienced I could fund this change to you or anyone else, but you'd have to support the change. @scaredyfish - is this open source enough to have someone else make the change and update the GIT code?

[Premik]

I've managed to get this working in Blender 3.5.0. Based on the #51. But I had to do several changes in the operator for it work (#52). But the result is pretty bad. I think there is something wrong with the baking process. Since when I tried to manually key few poses directly from the rhubharb command output the result is much better.

[gamedesign1]

Thanks for working on updating this addon. Let us know how you get on, can't wait to try it (no pressure) :)

[TwoHalfs]

Thanks for working on this. 3.4 no longer works for Rhubarb.

[Premik]

I haven't investigate in depth what is wrong with the keyframes. The timings seems correct. Bumping the fps up helps a bit but still it feels too off for simple manual corrections to be able to fix it. I've actually created short video with instruction. And the result can be seen at the end of the video. Should anybody still want to give it try there are additional steps in the video description. Describing how to update the installed plugin with the PR content.

Anyway I'm actually considering to do some major cleanup of this plugin as it is hard to test and maintain currently. And there are few things I'd like to do in the UI too. Like the baking process. It would probably make more sense to output action-clips into two NLA tracks. Without requiring the actions to be mark as an asset. This would also enable mixing with shape-keys action in the future.

[TwoHalfs]

I was looking at the results in the video you made. It is a very good rhubarb action. A small amount of editing will get a good result here. The audio may work better moved slightly to the right. Two rows of vertical keyframes should be deleted as there are double movements with the words size and on. I would be very happy with this result. I find I always have to do a bit of editing the keyframes with rhubarb. Delete double keyframes or resize length. But it creates great results quickly. Rhubarb does most the work. If you find you can get rhubarb to do a better result that would be fantastic. I am not sure what you mean with the nla action. As long as there are still key fames to work with it will be useable. Thanks for your effort it is appreciated.

[FrancisVS]

I'm using Blender Version 3.2.2 and I have done replace the new script and it work. But when I press the Rhubarb Lipsync button it came out this error like below. Wonder anyone have the same problem.

Python: Traceback (most recent call last): File "C:\Users\francis\AppData\Roaming\Blender Foundation\Blender\3.2\scripts\addons\blender-rhubarb-lipsync-master\op_blender_rhubarb.py", line 29, in poll context.object.pose_library.mouth_shapes.sound_file AttributeError: 'NoneType' object has no attribute 'mouth_shapes'

[Premik]

@FrancisVS It looks the script replacing didn't work. Perhaps your Blender is still using the old file. Try restarting the Blender if you haven't yet.

If still no luck

• Open the C:\Users\francis\AppData\Roaming\Blender Foundation\Blender\3.2\scripts\addons\blender-rhubarb-lipsync-master\op_blender_rhubarb.py file again.
• Check what you have on the line 29. There should be this: context.object.mouth_shapes.sound_file. Not context.object.pose_library.mouth_shapes.sound_file

The two scripts from the PR: op_blender_rhubarb.py and pnl_blender_rhubarb.py. Note there is no pose_library word used anywhere in the new scripts.

[FrancisVS]

@FrancisVS It looks the script replacing didn't work. Perhaps your Blender is still using the old file. Try restarting the Blender if you haven't yet.

If still no luck

• Open the C:\Users\francis\AppData\Roaming\Blender Foundation\Blender\3.2\scripts\addons\blender-rhubarb-lipsync-master\op_blender_rhubarb.py file again.
• Check what you have on the line 29. There should be this: context.object.mouth_shapes.sound_file. Not context.object.pose_library.mouth_shapes.sound_file

The two scripts from the PR: op_blender_rhubarb.py and pnl_blender_rhubarb.py. Note there is no pose_library word used anywhere in the new scripts.

Hi, thx. It work now. But when I start baking the animation it will just create 2 key frame and is not baking the whole wav file sound track voice reading. Wonder what is causing this problem.

[FrancisVS]

The error is like this

start:0.0 frame:1 value:X shape:mouth_x poseIndex:17 Traceback (most recent call last): File "C:\Users\francis\AppData\Roaming\Blender Foundation\Blender\3.2\scripts\addons\blender-rhubarb-lipsync\op_blender_rhubarb.py", line 82, in modal self.apply_pose(context, frame_num - self.hold_frame_threshold, bpy.data.actions[pose_index]) IndexError: bpy_prop_collection[index]: index 17 out of range, size 13 An exception of type IndexError occurred. Arguments: ('bpy_prop_collection[index]: index 17 out of range, size 13',)

[KaijuKoder]

Sorry, been away, but I'm about to return to this. Will try the latest version(?), but meanwhile looked into the new Asset system and found other than making the API incompatible, functionally it doesn't seem that different from poselib anyway. The Asset poses are just ordinary Actions, which you can link to using bpy.data.libraries.load(). You don't need to go through the UI or ops. Once you have your action you can apply it using Philipp-Lechner's solution here: https://devtalk.blender.org/t/keyframing-a-pose-asset-in-python/25982/2 There are a few minor bugs in Phillipp's answer: You need an extra ".w" for quaternions, and a modification to handle single-float channels.

[Premik]

My current thoughts about "baking"the result is there should eventually be multiple options how to "bake" the captured mouth Cues:

1) To a new Action. Pretty much what is in the current version. Mapping setup requires creating an Action for each Cue type. And the result is a new Action. With each mouth-shape of the caputred Cues baked to key-frames. 2) To NLA tracks. The mouth-shape Actions and/or Shapekey Actions are put into two NLA tracks as NLA strips in a zig/zag pattern. With proper strip settings they'd get blended automatically by NLA. So Mapping setup requires creating Actions and/or Shape-key Actions or both. 3) "Jump to frame" - more like a preview. Clicking on the captured Cue from the list would jump to the frame of that Cue. This is inspired by the Faceit plugin. Where Every 10 frames there is different mouth shape (and optionally shape-key) keyframed directly on the timeline.

I also realized it doesn't make sense mark the Mouth-shape Actions as Assets. Like it is currently required in the PR. They can be makred as assets but it should't be required.

So from the use-case perspective, it makes more sense to split the whole process into two stages (two panels): 1 - Sound setup and capture: Select the sound input, run the binary. Captured cues are put into an UIList but not "baked" yet. 2 - Cues mapping and baking: For each Cue type an Action or/and Shapekey Action or frame number is provided (=the Mapping setup). Then baking is ran. While the baking can be run repeatidly without re-runing the capture.

So these thought led me to start working on this plugin more. But it took me more time than I anticipated and it ended up in a rewrite :D

Currently I have the first part "Setup and Capture" pretty much done on my branch. But the Mapping setup and "baking" the Cues is still WIP:

[utiq]

Hi @Premik, great work! thanks for taking your time to make the plugin work. I managed to make it work with your PR #51 and #52. The only problem is that it only syncs the first 2 seconds, after that, nothing.

[Premik]

Hi @utiq, Good stuff. Not sure what it could be. I had bad results in general with the two PRs.

I'm actually still working on the complete rework of this plugin. Getting there slowly. But not in any usable stage yet:

[utiq]

@Premik yeah! I saw your branch, nice work!. I cloned I tried to use it but I had no luck, just errors. If you want I can help you test it out. I have computers with Windows, OSX and Linux.

[Premik]

Thanks. Some testing would be great help indeed. Especially on Windows and Mac. I only tried to run the unit tests in Wine. But was having hard times with Conda environment activation there.

I've just created 4.0.0 pre-release so you might give it try. Perhaps it would help with some of the errors you are getting. Eitherway feel free to open a ticket in my fork and add more details about the errors.

[scaredyfish]

Hi folks,

Sorry for being so quiet on this issue. Unfortunately, I haven't had much time to work on this addon, or even review the patches.

@Premik I'm happy to point people towards your repo going forward. Let me know if you want me to do that, and/or if you want to wait until your 4.0 is released.

[utiq]

Hi @Premik, I started testing it on Windows. I cannot open an issue on you repo because the tab is not visible. Maybe you have it deactivated in the config?

Meanwhile I am gonna post the error here. As soon as I select a file, it gets these errors: