search

DanielSWolf / rhubarb-lip-sync

Rhubarb Lip Sync is a command-line tool that automatically creates 2D mouth animation from voice recordings. You can use it for characters in computer games, in animated cartoons, or in any other project that requires animating mouths based on existing recordings.
Other
1.71k stars 208 forks source link

Support for Spine Skin Placholders? #108

Closed bencresty closed 2 years ago

bencresty commented 2 years ago

Hi,

First of all, thanks a lot for all the hard work and the magnificant outcome! Great tool!

I'm testing this for usage with Spine now and see everything works fine with having slots which contain region attachments/images. But some files here work with Skins, so instead of using region attachments directly into slots, they use Skin Placeholders (see: http://esotericsoftware.com/spine-skins). But unfortunately the tool doesn't recognise them so we can't use the tool on skinned files :(

Like so: image

Or like so: image

Which both don't get detected unfortunately: image

Using version: Rhubarb for Spine 1.11.0

Any chance support for skin placeholders could make it to the project so we can also use the tool for skinned projects?

Thanks in advance and keep up the great work!

DanielSWolf commented 2 years ago

I'm not an active Spine user myself and I don't have experience with skin placeholders. Could you attach a zip file with a working example setup?

bencresty commented 2 years ago

@DanielSWolf Of course. I just created two zip files for you;

  1. without-skin-folders.zip
  2. with-skin-folders.zip

1) without-skin-folders.zip

The first one has two skins which are placed directly into the skins node. Both skins have their own folder directly inside the img-folder.

2). with-skin-folders.zip

The second one has the same two skins, but now these are placed inside a skin folder in Spine. This skin folder ('characters'), which is basically a skin group, is also present as a fysical folder inside the img folder in the project. And that 'characters'-folder contains both skin-folders ('char1' and 'char2'). The concept of skin folders is created in Spine to mix-and-match different skins (of different skin folders) together at the same time, so that we can have a skin folder with mouths and a skin folder of backpacks, shoes etc. for example.

I don't think you really need to see the real image path, but above examples just in case so you know there can be different skin workflows. I personally prefer to have all skins in sub folders, but there are also people having all images of the first skin as a default in the root of the img folder and the other skins as sub folders inside the img-folder with their images.

I included a short audio file with some speech I found on youtube for you to test with. Don't pay much attention to the mouth. I just created something in a hurry and I'm new to that 6-mouths rule so not sure if I did this right. But it's only for testing. Next to the spine file I also exported the json just in case you don't have the latest spine version to export. (It's created this with Spine 4.0.47 pro btw)

Hope this makes sense and it's something you can work with!

Thanks a lot for your willing to help! Really appreciate it! Would be very cool to use your tool with the placeholders!

If you have any questions please let me know.

Have a nice day!

with-skin-folders.zip without-skin-folders.zip

DanielSWolf commented 2 years ago

Thank you for sharing your project files and for the detailed explanation! I don't have much time at the moment, so I can't promise when I'll be able to look into it. But I certainly will!

bencresty commented 2 years ago

Hi @DanielSWolf , thanks for the update. No problem and no hurries. Thanks for willing to make the adjustment later! Really appreciated! Have a nice weekend

DanielSWolf commented 2 years ago

I've attached a .zip file that attempts to add support for skinning. The content is a drop-in replacement for the original .jar file. Could you give it a try and tell me whether it works for you?

rhubarb-for-spine-1.11.0.jar.zip

bencresty commented 2 years ago

@DanielSWolf Wow! That's quick. Thanks!

I've just tested the new version with both test projects using skins (the ones I'd sent you). I see the lipsyncing working fine after conversion with the new jar-file, and the animations work on both skins, so that's cool!

However, when testing the new version with a file without using skins for mouths these mouths won't be recognized by Rhubarb anymore.

Not sure if the new version is just a quick'n'dirty test case for testing only the skinned method or should be a beta version supporting both? If the first case is going on and you need help, please let me know. Perhaps I can help you in supporting both together.

For you it might also be good to know that when we are using skins in Spine not all slots suddenly got to have skinned slots when using skins in a file. Some slots are still just 'regular' slots which don't change when switching to another skin. So for example a mouth slot can be skinned, while other slots are not, or visa versa. Or clothes and mouth might be skinned, while other slots are just 'regular' slots, meaning all skins are re-using images for those. Hope this makes sense.

[BTW] I saw I made a mistake in the events in the zip files I'd sent you. These were events, but no audio events. So just in case you'd like to test yourself with audio avents with these spine-files in Spine: 1) Open Spine 2) Open the Events node in the Tree, select the 'speech' event and hit DEL to delete it 3) Open the Audio node in the Tree, drag the speech.wav file and drop it on the Events node. This will create an Audio Event for it

DanielSWolf commented 2 years ago

Thanks for your feedback! The problem with Rhubarb for Spine is that I never took the time to write automated tests. So any change I introduce may break existing functionality, which seems to be the case here. I'll look into it and get back to you.

bencresty commented 2 years ago

@DanielSWolf no worries. I'm already glad you are taking the time to make this work! Thanks again

DanielSWolf commented 2 years ago

However, when testing the new version with a file without using skins for mouths these mouths won't be recognized by Rhubarb anymore.

I can't reproduce this behavior. I just loaded an old JSON file without skins into the new version of Rhubarb for Spine. The mouth slot and its attachments were all detected correctly.

Could you attach a project (ideally including the exported JSON file) that isn't working for you?

bencresty commented 2 years ago

Hi @DanielSWolf , I think that version is working when either having everything WITH skins or everything WITHOUT skins, but not when using both methods in the same file (so some parts skinned and other parts unskinned).

I'm having a tight deadline here, so am not able to provide you with a file which is using both methods at the same time, but if I find more time I'll provide you with one.

But perhaps and hopefully this is already useful information for you to get you started.

Thanks again for your efforts!

bencresty commented 2 years ago

@DanielSWolf Sorry for the late response. Today I finally got more time, so I did some tests again with the test version of Rhubarb you sent here and create projects for you to test with.

I've created and tested the following projects

Without even starting the Rhubarb action, when importing the json files it's clear when Rhubarb does and when it doesn't work, just by drag'n'dropping the json onto the app. These are the results:

No skins: rhubarb-import-noSkins

Full skinned: rhubarb-import-fullSkinned

Full skinned - no skin folders: rhubarb-import-fullSkinned-no-skin-folders

Skinned, except for mouth: rhubarb-import-skinnedExceptMouth

Skinned, except for mouth - no skin folders: rhubarb-import-skinnedExceptMouth-no-skin-folders

test-projects

new-rhubarb-tests.zip

conclusion

I didn't test what happens if the mouth is skinned, but the rest is not, but my guess is that that would work, because in that case Rhubarb probably detects there are skins available and that the files for the mouth are available via these skins.

For the rest the test app recognises the mouths when either everything is skinned, or nothing is skinned and there are no skins used. When everything is skinned, except for the mouth, Rhubarb can't retreive the mouth parts from the json.

Hope this helps!

DanielSWolf commented 2 years ago

Thank you so much for going to the trouble of creating all these demo files! This will make my job much easier. I'll try to get to it as soon as I can!

bencresty commented 2 years ago

Hi @DanielSWolf , by no means I want to hurry you, as we're all busy so I completely understand it if you don't have the time for it. But it's two months later so I am a little curious to see if there perhaps is any progression on this? If you need any more information let me know. Thanks and have a nice day!

DanielSWolf commented 2 years ago

Wow, I didn't realize it had been so long! I'll try and fix the problem in the next few days.

DanielSWolf commented 2 years ago

I think I got it working in all cases. I tried out your example projects and they all look good. I've attached the zipped JAR file again. Could you give it a try before I make an official release?

rhubarb-for-spine-1.11.0.jar.zip

bencresty commented 2 years ago

Wow, I didn't realize it had been so long! I'll try and fix the problem in the next few days.

Yeah, I had the exact same feeling. Time is moving way too quickly!

I think I got it working in all cases. I tried out your example projects and they all look good. I've attached the zipped JAR file again. Could you give it a try before I make an official release?

Great news! I will. Didn't have the time for it yet, but come back to you soon on this. For now, have a nice weekend!

bencresty commented 2 years ago

Hi @DanielSWolf , just did a test here with all types of projects and with audio events in them. AFAIS everything works perfectly now! So it seems like a great new version to release for everybody to enjoy!

Thanks again for adjusting Rhubarb to using skins and mixtures of skinned and non-skinned elements!

DanielSWolf commented 2 years ago

I just released v1.12 with skinning support. @bencresty Thank you once again for your help and all the example files!