Metahuman instructions

[1OGAG]

I've been trying to get the leap motion controller to drive the hand movements for Unreal's Metahuman (for virtual productions) without any success. I am hoping one you big brained people could give me some guidance or point me to instructions.

[Ultraleap-Dan]

We have had a look at how hand tracking can be used with Unreal's metahuman and it looking like the quickest way to to do this with LiveLink. This way our hand tracking plugin is a live link source which streams bone transforms.

These can then be copied in the meta human control rig:

You would then need to map this to the skeleton bones for the hand and add IK nodes to make the arms and shoulders follow.

[1OGAG]

Awesome - thanks for the input. Let me give this a try and see how it goes. I'll report back.

[1OGAG]

I had to open an RMA for my controller. As soon as I get it back I will try out the instructions above and let you know if I encountered any issues.

[ReadyPlayerEmma]

Is it possible for this to work in the UE5 early access releases?

[1OGAG]

@ReadyPlayerEmma just wondering if you might have tried it on ue4 and gotten it to work? I have not been successful yet.

[ReadyPlayerEmma]

@1OGAG I have a project I started in UE5, so I have been working entirely in there. I did compile the plugin successfully against UE5 though, and the plugin loads fine. So now I just have to figure out the process. If I do, I will come back and post more details. Also, if you figure it out first, I wouldn't mind hearing how xD

I am relatively new to UE and am coming from the Unity world, so I am still slow to work through some of these problems, but I eventually get there. Hopefully the same will be the case with this.

edit: I don't think this actually is working in UE5 yet, even though it compiled against it and technically loaded okay. But I can't be sure since I may be doing something wrong. I am going to go back to UE4 in a test project and experiment to see if I can integrate the Metahuman avatar and Leap Motion Live Link data using a known supported environment.

[1OGAG]

@Ultraleap-Dan would you be open to publishing a step-by-step with IK nodes? @ReadyPlayerEmma , if I do get it working in UE4 I'll be sure to write up some instructions.

[JimSimons-UH]

@ReadyPlayerEmma Re UE5, the plugin will compile, but a lot of the assets/maps and blueprints are not compatible with the early access release as they were saved in UE4.27. Assets that are saved in 4.27 don't open in UE5 Early access, at least until another early access release is made by Epic.

I've tested against Unreal 5.0 built from source (i.e. the most up to date code), and this loads the Ultraleap plugin's assets fine. This means when Epic do release the next UE5 preview, the plugin's assets will load correctly.

[ocalaf]

We have had a look at how hand tracking can be used with Unreal's metahuman and it looking like the quickest way to to do this with LiveLink. This way our hand tracking plugin is a live link source which streams bone transforms.

These can then be copied in the meta human control rig:

You would then need to map this to the skeleton bones for the hand and add IK nodes to make the arms and shoulders follow.

How this mapping should be done? I don't find any examples or documentation and I'm a bit lost.

Trying directly to set a rotation or transform ends in editor not responding, hanging and not letting exiting without forcing the end of the task. I guess some remapping should be done prior to do these rotations or transforms.

Thank you!

[ocalaf]

Just trying to add more information. The Live Link Frame data (Transform) is an array of 42 elements if both hands are detected, but the Evaluate Live Link Frame (Transform) returns only one transform, so how should we proceed?

I also tried to use the Live Link Controller Preview with a Retarget Asset and the preview viewport keeps showing "Previewing reference pose", maybe the Live Link Control Rig plugin which is still in beta is not totally implemented or something like that.

[JimSimons-UH]

@ocalaf there's another node that allows you to access live link transforms:

[ocalaf]

@ocalaf there's another node that allows you to access live link transforms:

I saw that node too, but as far as I understand ultraleap transforms doesn't have name, isn't it? And at the end the get transform by name that you also use returns a single transform too, how should we access the array of transforms coming from ultraleap live link?

There is a new thing though, I don't know if it was the new ue5 preview 2 or the new plugin, but the live link controller preview and the remap asset now works under control rig.

The thing is, how the remap could be used inside the control rig, but not just in the preview controller?

[1OGAG]

@Ultraleap-Dan or @JimSimons-UH - I know it might be a big ask, but could one of you put together a step-by-step tutorial for this?

[JimSimons-UH]

Hi, Metahuman examples and support is on our roadmap within the next couple of months.

The bone names/transform names are:

[0] BONE_LOWERARM_L FName

• [1] BONE_HAND_WRIST_L FName
• [2] BONE_INDEX_0_METACARPAL_L FName
• [3] BONE_INDEX_1_PROXIMAL_L FName
• [4] BONE_INDEX_2_INTERMEDIATE_L FName
• [5] BONE_INDEX_3_DISTAL_L FName
• [6] BONE_MIDDLE_0_METACARPAL_L FName
• [7] BONE_MIDDLE_1_PROXIMAL_L FName
• [8] BONE_MIDDLE_2_INTERMEDIATE_L FName
• [9] BONE_MIDDLE_3_DISTAL_L FName
• [10] BONE_PINKY_0_METACARPAL_L FName
• [11] BONE_PINKY_1_PROXIMAL_L FName
• [12] BONE_PINKY_2_INTERMEDIATE_L FName
• [13] BONE_PINKY_3_DISTAL_L FName
• [14] BONE_RING_0_METACARPAL_L FName
• [15] BONE_RING_1_PROXIMAL_L FName
• [16] BONE_RING_2_INTERMEDIATE_L FName
• [17] BONE_RING_3_DISTAL_L FName
• [18] BONE_THUMB_0_METACARPAL_L FName
• [19] BONE_THUMB_1_PROXIMAL_L FName
• [20] BONE_THUMB_2_DISTAL_L FName
• [21] BONE_LOWERARM_R FName
• [22] BONE_HAND_WRIST_R FName
• [23] BONE_INDEX_0_METACARPAL_R FName
• [24] BONE_INDEX_1_PROXIMAL_R FName
• [25] BONE_INDEX_2_INTERMEDIATE_R FName
• [26] BONE_INDEX_3_DISTAL_R FName
• [27] BONE_MIDDLE_0_METACARPAL_R FName
• [28] BONE_MIDDLE_1_PROXIMAL_R FName
• [29] BONE_MIDDLE_2_INTERMEDIATE_R FName
• [30] BONE_MIDDLE_3_DISTAL_R FName
• [31] BONE_PINKY_0_METACARPAL_R FName
• [32] BONE_PINKY_1_PROXIMAL_R FName
• [33] BONE_PINKY_2_INTERMEDIATE_R FName
• [34] BONE_PINKY_3_DISTAL_R FName
• [35] BONE_RING_0_METACARPAL_R FName
• [36] BONE_RING_1_PROXIMAL_R FName
• [37] BONE_RING_2_INTERMEDIATE_R FName
• [38] BONE_RING_3_DISTAL_R FName
• [39] BONE_THUMB_0_METACARPAL_R FName
• [40] BONE_THUMB_1_PROXIMAL_R FName
• [41] BONE_THUMB_2_DISTAL_R FName

I grabbed these by debugging the LiveLink source. They come from EBodyStateBasicBoneType in BodyStateEnums.h in the Ultraleap Tracking Plugin.

[1OGAG]

@ocalaf (or anyone else) are you still trying to get this to work? Any insights?

[Lemondogdog]

I hope someone does it and shares UE4 files to learn

[Phonicavi]

@JimSimons-UH Hi, I followed the official instruction using "Ultraleap Modify Mapped Bones (Body State)", embedding this graph node into MetaHuman's official animation blueprint (e.g. f_med_nrw_animbp). However, only one side (the left hand) could be driven even though the Live Link Panel actually shows "LiveLinkTransformStructure Data Frame" with correct two-hands data.

After reading the source code, I noticed that each UBodyStateAnimInstance has an array of FMappedBoneAnimData called MappedBoneList, according to RigTargetType. While each AnimNode_ModifyBodyStateMappedBones instance only has a single FMappedBoneAnimData member and it was always updated with the first element of MappedBoneList. I guess that is the reason why this graph node only support one single hand.

Although it is supported to auto-map both two hands when setup the blueprint class, the data from the second hand (the right side) can not be passed correctly, with one BSAnimInstance.

Since I'm really new to Unreal Engine, and I have no idea about how to split the MetaHuman blueprint into two separate parts (also for their SkeletalMesh), like the Ultraleap official example contents (i.e. BSLowPolyLeft/RightHand ). I hope this reply could be helpful to you who would like to fix it.

Best Regards.

[JimSimons-UH]

@Phonicavi thanks for the detailed info, we are currently QA'ing metahuman support which includes auto mapping of both hands with associated fixes.

[JimSimons-UH]

The latest release supports Metahumans using templates, see the docs @ https://docs.ultraleap.com/unreal-api/unreal-guide/metahuman-support.html