What is the order of Body Joints

[abhin7993]

I am trying to animate a humanoid avatar in unity using your code. Is the order of body joints as follows?

// 'Torso_Back': 1,
//'Torso_Front': 2,
//'RHand': 3,
//'LHand': 4,
//'LFoot': 5,
//'RFoot': 6,

//'R_upperLeg_back': 7,
//'L_upperLeg_back': 8,
//'R_upperLeg_front': 9,
//'L_upperLeg_front': 10,

//'R_lowerLeg_back': 11,
//'L_lowerLeg_back': 12,
//'R_lowerLeg_front': 13,
//'L_lowerLeg_front': 14,

//'L_upperArm_front': 15,
//'R_upperArm_front': 16,
//'L_upperArm_back': 17,
//'R_upperArm_back': 18,

//'L_lowerArm_back': 19,
//'R_lowerArm_back': 20,
//'L_lowerArm_front': 21,
//'R_lowerArm_front': 22,

//'RFace': 23,
//'LFace': 24

If not then could you please share the correct order?

[LucaswasTaken]

I am trying to animate a humanoid avatar in unity using your code. Is the order of body joints as follows?

// 'Torso_Back': 1,
//'Torso_Front': 2,
//'RHand': 3,
//'LHand': 4,
//'LFoot': 5,
//'RFoot': 6,

//'R_upperLeg_back': 7,
//'L_upperLeg_back': 8,
//'R_upperLeg_front': 9,
//'L_upperLeg_front': 10,

//'R_lowerLeg_back': 11,
//'L_lowerLeg_back': 12,
//'R_lowerLeg_front': 13,
//'L_lowerLeg_front': 14,

//'L_upperArm_front': 15,
//'R_upperArm_front': 16,
//'L_upperArm_back': 17,
//'R_upperArm_back': 18,

//'L_lowerArm_back': 19,
//'R_lowerArm_back': 20,
//'L_lowerArm_front': 21,
//'R_lowerArm_front': 22,

//'RFace': 23,
//'LFace': 24

If not then could you please share the correct order?

I am also curious. Could you please also share the hand joints as well?

[abhin7993]

@LucaswasTaken check this for hand joints. https://github.com/facebookresearch/frankmocap/issues/18

[lisa676]

@abhin7993 Could you share the steps that how to animate humanoid avatar in unity using this repository code? Sorry for my silly question, I'm new in Unity that's why I want to know about basic steps. Thank you

[penincillin]

@abhin7993 @Lucaswas Please refer to SMPL-X's official definition for order of joints We might make small modifications to the order of joints, but in general, this list is OKish

[abhin7993]

As mentioned in the doc https://github.com/facebookresearch/frankmocap/blob/master/docs/run_bodymocap.md#mocap-output-format-pkl There are 24 body-joints predicted in the output in angle-axis format.I am planning to use these to animate the avatar.But the SMPLX's official definition contains 140+ joints, even if i remove hand joints , still it there are many more than 24.Can you please let me know, which one are predicted in frankmocap.

[abhin7993]

@lisa676 Though I am still working on it.But I am following this repo https://github.com/keel-210/OpenPose-Rig and use the 3d pose parameters from frankmocap's output pkl for every frame.

[abhin7993]

@penincillin I am following this tree (https://github.com/YeeCY/SMPLpp/blob/master/docs/media/kinematic_tree.png). Is this correct?

[penincillin]

@abhin7993 The order of joints depicted in your shared image is correct for joints 0 to joints 21. Joints 22 and 23 are different from ours.

[abhin7993]

what are joint 22 and 23 in frankmocap?

[penincillin]

I am not quite sure for this part. It depends on which group of joints you are used. I would suggest you to visualize the joints to determine the exact meaning of each one.

[penincillin]

@abhin7993 I just noticed that the order of joint is not what I told you before. I am sorry for that ! I checked today and find that the order of body joints is: The order of hand joints is: In general, the order of joints in frank_mocap is the same as openpose in general. Again, I strongly suggest you to visualize the joints and double-check it.

[abhin7993]

@penincillin I'd request you to please check again the order of body joints.The reason I am saying this is because when I used the order of body joints as per the image I had shared previously, I was able to animate any Humanoid Avatar in unity properly. My Hand animation wasn't working properly and I will change the hand joints order as per the above image and will report the result.

[abhin7993]

(https://youtu.be/katBT50EIx8) Btw check the above video for the result of unity re-targeting.Thought, you would be happy to see it.

[stspanho]

@abhin7993 very interested to see your result, but the video is private. Can you make it public?

[abhin7993]

Oh!! My bad.I have made it public, now you should be able to watch it.

[abhin7993]

Right now, I am using only the joint rotations, So Avatar does not have any linear motion.Also, there is frame rate mismatch at Unity end which can be taken care later.

[abhin7993]

@penincillin I request you please check again the order of body joints.The reason I am saying this is because when I used the order of body joints as per the image I had shared previously, I was able to animate any Humanoid Avatar in unity properly. My Hand animation wasn't working properly and I will change the hand joints order as per the above image and will report the result.

I changed the hand joint order but results are even worse.Actually with the joint order you had suggested in #18 I was able to get quite accurate animations with Unity Chan Avatar, but when applying the same with any other avatar hand shapes are totally distorted.I think it might have something to do with me not using hand local/global orient properly.Could you please put some light on meaning of 4 different data in pkl file (1.Left_hand_local_orient_hand 2.Left_hand_global_orient_hand 3.Left_hand_local_orient_body 2.Left_hand_global_orient_body).I do understand global and local orientation but what do we mean by orient_hand/orient_body here?

[penincillin]

@abhin7993 I think we mess something up here. When you talked about "joint", you mean the joint position or joint rotations ? Actually the order of these two formats are different. If it is joint rotation, then it is the one I mentioned in issue 18. If it is joint position, then it is the one I mentioned above

[abhin7993]

That sounds like right.But in joint position ti gives data for 49 points.Should I consider only initial 24 points?

[penincillin]

@abhin7993 Please only consider the first 24 points.

[lisa676]

@abhin7993 Thanks for your wonderful track because you gave a good idea to animate avatar in Unity. As I commented above about the basics step and you recommended to check this OpenPose-Rig, I checked it as sample and as it is using 3D data available in text for every frame. So in case of this repository (Frankmocap) we can get pred_hand_pose from pkl file. So my question is that, we only need pred_hand_pose data for every frame to animate avatar or also need some other data from pkl file like pred_hand_betas, pred_camera etc? Thanks in advance.

[abhin7993]

@lisa676 openpose rig uses joint posotion to calculate rotation angle, but you can directly use relative rotation given by frankmocap and make your bones rotate with respect to its parent. I am only using pred_hand _pose, but I am geetting good result with some avatars b t not with others, which shouldn't be the issue with humanoid animation.So I think other parameters might be helpful, butI still need to test that.Let me know if you find something in that regard.

[lisa676]

@abhin7993 Thanks for your help. I'm also in starting stage hopefully I will get it with your help and if I got something I will share but seems that you are doing great in this solution.

Still I'm confused about one thing that as you said above that openpose rig uses joint posotion to calculate rotation angle, but you can directly use relative rotation given by frankmocap and make your bones rotate with respect to its parent. Could you elaborate more about relative rotation? pred_hand _pose actually is relative rotation or it is something else?

Thanks again for your help.

[abhin7993]

it is relative rotation only

[penincillin]

@lisa676 It's actually relative rotation.

[lisa676]

@abhin7993 @penincillin thanks guys for your help and support, I'm checking it

[wangtss]

@abhin7993 I'm also trying to use the data in pred_hand_pose to animate avatar in unity, but I'm having a hard time to understand the data format. The data in pred_hand_pose is in angle-axis format, so I try to transform it into Quaternion and use it in Unity, but it doesn't work. Could you please share how you did it? Thanks.

[ahsan3803]

@wangtss I think you need to use 3D joints data from model and then you may use these joints to Unity. SMPLX can output 3D joints, follow #39 for more info.

[wangtss]

Thank you for your advice. I was able to get the 3D hand joint position from model output, but position data alone is not sufficient for animating virtual hand in unity. I try to translate the angleaxis rotation to quaternion, but the result is very bad. So, it would be a huge help if @abhin7993 @ahsan3803 could share the way about how to translate the angleaxis data into a formate that can be used in Unity.

[ahsan3803]

@wangtss I didn't try it in Unity, I just draw 3D joints

[Q-Y-Yang]

@penincillin Regarding the joints order, do you mean the 3D joints rotation follows the order of smpl-x model, and 3D joints position follows the order of openpose? Is the pred_joints_img in pred_output_list of demo the 3D joints position?

Thanks!

[penincillin]

@Q-Y-Yang 3D Joints rotation (both body and hand) follows the order of SMPL-X model. For 3D Joints Position, the body part follows the order of openpose. Hand joints order differs in different mode. In hand-only mode, it follows the order of SMPL-X model. In whole-body mode, it follows the order of openpose.

[Q-Y-Yang]

@Q-Y-Yang 3D Joints rotation (both body and hand) follows the order of SMPL-X model. For 3D Joints Position, the body part follows the order of openpose. Hand joints order differs in different mode. In hand-only mode, it follows the order of SMPL-X model. In whole-body mode, it follows the order of openpose.

@penincillin Thanks for your reply! I have one more question. Is the third column of pred_img_joints the depth? Because it contains both positive and negative values. If it is depth, then where is the origin of z axis (distance)? If the origin lies in the camera, then the depth should be the same sign. Sorry I am new in the field of pose estimation.

[huanngzh]

(https://youtu.be/katBT50EIx8) Btw check the above video for the result of unity re-targeting.Thought, you would be happy to see it.

@abhin7993 Hi! I saw your use of rotations to drive avatar motion in Unity and it works fine. I'm trying to do something like this, driving the avatar using the joints' rotation matrices output by ExPose (also a SMPLX model). But my effect is wrong. I get the rotation matrix of 21 joints through ExPose, use the method scipy.spatial.transform.Rotation to convert it to the Euler angle of zxy, and assign it to joint.localEulerAngles on the corresponding joint of avatar. But after this implementation, avatar does not display the correct output. Is there any error in this step?