Using Leap for Hand/Finger Mocap on Custom Rigs

[BaazarStudios]

Hello Getnamo,

I thought I'd compile my questions into one thread since the issues and confusion seem to be popping up across the board.

My whole goal is to use a custom character skeletal mesh with the leap motion to record finger and hand data as additive anims. (Similar to the purposes of the Leap iClone plugin but I want to do it directly in Unreal.)

Since the auto-rig bodystateanim blueprint crashes the editor, I have tried custom rigging and that works, but it doesn't allow for previewing and seeing your live control without putting it in a blueprint and and playing, (unlike the bodystateanim types.)

I did get that to rig properly but whenever I simulate/play the hands/arms are reversed upside in HMD/VR mode (I assume.) Calling "Set Leap Mode" to Desktop from the Blueprint does nothing in Unreal, but it does break the Leap Motion Control Panel, swapping the arms and making me reinstall the software again. (Most of the bodystateanim examples are also reversed if you don't want to use VR with the leap motion.)

Additionally, you can't really record and use sequences with this blueprint so it seems like the most effective method is using the BodystateAnim mode once the bugs are worked out of that. But your thoughts are appreciated.

I would just use the LeapMannequinAutoHandsOnly_Anim_Blueprint but the hands are also reversed with that.

Thanks again for your assistance and support!

[getnamo]

It is strange that the Set Leap Mode would break the control panel, sanity check: have you updated to the latest version? (https://www.leapmotion.com/setup/desktop/windows/)

There are two things that happen when you set the leap mode, 1) the leap changes optimization so that it expects hands in a certain way, 2) the orientation of the data is changes to match your setup (sensor facing up). What appears to be happening is that the leap mode is not being properly set for you or that it is being set only on begin play. By default the option is set to vr mode, it might make sense to move this setting along with other default options to a json file so they can be changed and kept throughout the editor (enhancement issue: https://github.com/leapmotion/LeapUnreal/issues/32).

I'll need to take a look at this issue with leap mode and https://github.com/leapmotion/LeapUnreal/issues/17 so that bodystate will work for your use case.

[BaazarStudios]

It is strange that the Set Leap Mode would break the control panel, sanity check: have you updated to the latest version? (https://www.leapmotion.com/setup/desktop/windows/)

There are two things that happen when you set the leap mode, 1) the leap changes optimization so that it expects hands in a certain way, 2) the orientation of the data is changes to match your setup (sensor facing up). What appears to be happening is that the leap mode is not being properly set for you or that it is being set only on begin play. By default the option is set to vr mode, it might make sense to move this setting along with other default options to a json file so they can be changed and kept throughout the editor (enhancement issue: #32).

I'll need to take a look at this issue with leap mode and #17 so that bodystate will work for your use case.

Thank you! Yes I have tested this and it is the same with both the Orion and the new Orion Beta SDK. On some tests, by closing unreal, rebooting and unplugging the leap motion would correct it.

I was able to get some temporary results by then using the control panel to reverse orientation which actually did set the hands in the proper position however the rigging seemed to warp the fingers and not map correctly, (thumb was unmovable.) I am not sure if that was a rig related issue or a false positive on orientation that cause artifacts in the anim of the hand.

[BaazarStudios]

It is strange that the Set Leap Mode would break the control panel, sanity check: have you updated to the latest version? (https://www.leapmotion.com/setup/desktop/windows/)

There are two things that happen when you set the leap mode, 1) the leap changes optimization so that it expects hands in a certain way, 2) the orientation of the data is changes to match your setup (sensor facing up). What appears to be happening is that the leap mode is not being properly set for you or that it is being set only on begin play. By default the option is set to vr mode, it might make sense to move this setting along with other default options to a json file so they can be changed and kept throughout the editor (enhancement issue: #32).

I'll need to take a look at this issue with leap mode and #17 so that bodystate will work for your use case.

Hey Jan, just checking in. Undoubtedly you're super busy and I respect your time but wanted to see if you'd have an update to these issues anytime soon. Just trying to see if I can get to use Leap Motion for my current pipeline or if it will be a while and I'll need to check back in for the future. Thank you!

[getnamo]

I'm looking to address this within a week, but its just a guideline, no promises.

[BaazarStudios]

I'm looking to address this within a week, but its just a guideline, no promises.

Thanks Jan, I appreciated the hard work you've put into this. Either solution works, if the autorig hotfix is the faster better method, then that works for my pipeline as well.

[BaazarStudios]

Hey Jan- just checking in, any progress on fixing the auto rig at all? No pressure if not, this would just be my last week for this potential workflow.

Thank you!

[maxenko]

Stuck on this as well, editor crashes every time I try to parent to BodyStateAnim, is there a fix? Thank you.

[vvlay9090]

Same as here~ Is there a fix now?