Human joint skeleton to joint positions for Humanoid robot

[ShadowDawg]

Hi loco-mujoco team! I was hoping to use loco-mujoco to generate joint poitions/angles/torques for a humanoid robot given joint skeleton data of a human obtained through Mediapipe from a video(example skeleton picture given below).

I'm not really sure how to get started and I would appreciate some help or a brief overview on how to accomplish this using loco-mujoco. Thanks!

[robfiras]

Trajectory data used in LocoMuJoCo only contains kinematic data (join angles and velocities). The easiest thing to do is to choose a humanoid environment and find the joint angle pairs that match (between your model and the chosen one in LocoMuJoCo). You can also adapt a dataset as described here: https://github.com/robfiras/loco-mujoco/tree/master/loco_mujoco/datasets/data_generation

But if your model has a serious kinematic mismatch with the chosen model, the result will probably look pretty bad. In that case, you would rather match foot/hand positions and then calculate the joint angles using inverse kinematics. However, we don't have a script for this right now.

I will close this, as this is not a real issue with LocoMuJoCo.

[ShadowDawg]

Hi! Thanks for your response and I am now proceeding with matching the joint-angle pairs from my skeleton model and a humanoid model in loco-mujoco. I went through the readme here: https://github.com/robfiras/loco-mujoco/tree/master/loco_mujoco/datasets/data_generation The UnitreeH1 example provided seems to take in the following joint angle data:

['pelvis_tilt' 'pelvis_list' 'pelvis_rotation' 'pelvis_tx' 'pelvis_ty'
 'pelvis_tz' 'hip_flexion_r' 'hip_adduction_r' 'hip_rotation_r'
 'knee_angle_r' 'ankle_angle_r' 'subtalar_angle_r' 'mtp_angle_r'
 'hip_flexion_l' 'hip_adduction_l' 'hip_rotation_l' 'knee_angle_l'
 'ankle_angle_l' 'subtalar_angle_l' 'mtp_angle_l' 'lumbar_extension'
 'lumbar_bending' 'lumbar_rotation' 'arm_flex_r' 'arm_add_r' 'arm_rot_r'
 'elbow_flex_r' 'pro_sup_r' 'wrist_flex_r' 'wrist_dev_r' 'arm_flex_l'
 'arm_add_l' 'arm_rot_l' 'elbow_flex_l' 'pro_sup_l' 'wrist_flex_l'
 'wrist_dev_l']

My skeleton data is obtained from mediapipe(image attached below) and I was having a little trouble figuring out the corresponding joint in loco-mujoco. I was wondering if there was a mapping for the same? Or maybe there was a way I could figure it out myself?

[robfiras]

That should work out, even though you are missing a lumbar joint. What I recommend you is to use Mujoco's simulate tool (https://mujoco.readthedocs.io/en/stable/programming/samples.html#sasimulate) to launch the simulator with our humanoid model. Then pause and reset. With the paused simulation, you can then move the each joint with the sliders on the right side. In doing so, you can see what joint correspond to what movement. Also, pay attention to the sign of the rotation.

[ShadowDawg]

Sounds good, thanks for the help!