search

ButzYung / SystemAnimatorOnline

XR Animator, AI-based Full Body Motion Capture and Extended Reality (XR) solution, powered by System Animator Online
https://sao.animetheme.com/XR_Animator.html
805 stars 68 forks source link

ARKit 52 blendshapes to Blender #33

Open Echolink50 opened 12 months ago

Echolink50 commented 12 months ago

I got a response on a YouTube video about vmd export not supporting ARkit blendshapes. I am trying to get the ARkit blendshapes into blender somehow. Is there any method to do this? BMC protocol or anything? Thanks

anajenna commented 12 months ago

Yes it is possible using an add on called vmc4b you can send data from xr animator to blender. https://booth.pm/ja/items/3432915.

I also found that after exporting my vroid model from vroid studio I can give it the 52 blendshapes using the Hana app https://booth.pm/ja/items/2917742.
This however only works if you export your model as vrm 0.0 NOT 1.0. You also have to before saving go to the, setting, reduce polygons, uncheck delete transparent meshes before exporting you model from vroid studio.

After adjusting the model in the Hana app I drop it into xr animator and turn on vmc protocol, camera, tracking, etc. Or if I already did in my last session I use streamer mode. Then I go to blender, import that same model, and set up the vmc4b add on using these settings Address: 127.0.0.1 Port: 39539 Armature : Armature Then I hit connect and xr animator sends real-time motion data to blender. Hope that makes sense.

Echolink50 commented 12 months ago

Thanks very much anajenna. In my research I did run across vmc4b. Since it was a paid addon and I wasn't sure if it would work I didn't purchase it. Does vmc4b transfer all 52 ARKit blendshapes/shapekeys to blender? Also, vmd export does send a few of the shapekeys on the default character. They are in Japanese but it's things like oh, grin, stomach, surprise, etc. What is the difference in these and why are they able to be exported? Thanks again

anajenna commented 12 months ago

I can't say I fully get how it works, im also not familar with vmd but if it can be converted to the vrm format that would be good. Or you can import it as is into blender if there's a vmd importer add on you can use. Not sure what will happen though, or if the blendshapes will transfer.

Anyways, it's my understanding that vmc4b recognizes the blendshapes your model has (or at least most of them), so if you have all 52 blendshapes on your model and you import it into blender you can check to see if all the expressions are there(object mode > select face> click (data) aka green triangle symbol and you'll see a list of shape keys are there.) Then when you record your motion data the values of each blendshape should change based on your expressions. For me I can see that expressions I added in actually work. And even if I wasn't using vmc4b, I could still toggle the values of my blendshapes and adjust my character's expressions. As long as you have an up to date vrm importer add on, you should be able to import your edited model into blender. Then again your using a vmd model, so I'm not sure what would happen if you imported it as a vmd model.

As I said earlier, I had to use the Hana app to add the 52 blendshapes, otherwise my model would only have the basic expressions. I don't really know what vmd export is ,but it seems like it let's you add some additional blendshapes just not all of them. Or maybe they're just custom blendshapes or the default ones, idk? In any case, my knowledge of all this is limited, I just know that it works for me when I'm using a vrm model. Good luck, I hope it works out for you.

Echolink50 commented 12 months ago

I will look into vmc4b as well as other options. Thank you for the info my friend

ButzYung commented 12 months ago

VMD is the motion format used by MikuMikuDance (MMD). I use this format because VMD support already exists in the project inherited by XR Animator. Ideally I would like to support FBX export as well, but I am not that familiar with this format, and I have yet to find any plugin that supports FBX export on web/node-based platform

anajenna commented 12 months ago

Oh okay. So if im understanding correctly it's a kind of motion capture file and its compatible with Xr animator . Okay well in that case technically you @Echolink50 could send the data to blender with vmc4b but im not sure if the facial expressions will transfer too. I've done this with fbx files from mixamo and videos. I drag and drop the file into xr animator and then stream it to my vrm model in blender. I don't have a vmd file with emoting to check if it works. All I know is that the what I can get my avatar to do in xr animator can be sent to blender.

ButzYung commented 12 months ago

If you are using a VRM model on XR Animator that supports ARKit blendshapes, all ARKit blendshapes data will be sent via VMC protocol. The export format is just for file saving and will not affect realtime VMC protocol.

Echolink50 commented 12 months ago

If you are using a VRM model on XR Animator that supports ARKit blendshapes, all ARKit blendshapes data will be sent via VMC protocol. The export format is just for file saving and will not affect realtime VMC protocol.

I will look into getting the shapekeys via VMC protocol. As for the export everything is in Japanese so it's hard for me to go through the XR Animator files, but is there some kind of config file or something? If no shapekeys got exported I would think it was not possible but the fact that about 15 or so.shapekey do get exported it seems like there is a possibility. The The program just needs to know the name and corresponding values somehow. Thanks for your help anajenna and ButzYung