About local frame！

[haolyuan]

where can I find the specific defination of each local frame in this MANO code？

[lixiny]

each rotation (local frame) is represented as axis-angle w.r.t its parent frame.

The order is:

# **** axis order right hand
#         14-13-12-\
#                   \
#    2-- 1 -- 0 -----*
#   5 -- 4 -- 3 ----/
#   11 - 10 - 9 ---/
#    8-- 7 -- 6 --/

Lixin

[haolyuan]

Thanks a lot! I have another question, Why the render result of app.py is not consistent to random_pose? When random_pose is all zero, the render result is as shown below. Is T-pose of MANO not consistent to T-pose of Smplx?

[lixiny]

@Yamato-01

Sorry for the confusion.

The all zeros T-pose of MANO is consistent with T-pose of SMPL-X only when the flat_hand_mean = True. You can type --flat_hand_mean in your command line when running app.py see code here

[haolyuan]

Thanks a lot! One more question. How to avoid abnormal hand pose after designing local frame in your project? Such as setZero?

[lixiny]

If I understand your question correct, the abnormal poses can not be penalized without knowing its parent, or child.

We use a coordinate adaptation axis layer to convert the original rotational axes to twist-bend-splay (in our code: back-left-up), and then penalize the abnormal rotations w.r.t twist-bend-splay.

Thanks a lot! One more question. How to avoid abnormal hand pose after designing local frame in your project? Such as setZero?

[haolyuan]

Thanks again! I have found three loss in geo_loss.py: joint_u_axis_loss, joint_l_limit_loss and joint_b_axis_loss. For example, You calculate cos value using b_axis and axis. Do the b_axis and axis mean the twist axis direction of the same local frame with this joint's pose (include fathers' joints along kintree) and with only its father joints' pose?

[haolyuan]

If I understand your question correct, the abnormal poses can not be penalized without knowing its parent, or child.

We use a coordinate adaptation axis layer to convert the original rotational axes to twist-bend-splay (in our code: back-left-up), and then penalize the abnormal rotations w.r.t twist-bend-splay.

Thanks a lot! One more question. How to avoid abnormal hand pose after designing local frame in your project? Such as setZero?

One more question! I am confusing about abnormal pose of thumb recently. It seems that the three joints of thumb have more DOFs than other fingers. I would like to know how to penalize the abnormal rotation of thumb specifically. Could you please help me?

[lixiny]

Thanks again! I have found three loss in geo_loss.py: joint_u_axis_loss, joint_l_limit_loss and joint_b_axis_loss. For example, You calculate cos value using b_axis and axis. Do the b_axis and axis mean the twist axis direction of the same local frame with this joint's pose (include fathers' joints along kintree) and with only its father joints' pose?

Yes, the b_axis represent the back direction (aka twist in our paper), and it is calculated through: current joint's child's xyz minus current joint's xyz.

axis-angle: Since rotation can be represented as an direction (unit 3D vector, axis) and a scalar value (angle), calculating the cosine value between axis and the b_axis (unit vector along the back direction) give us the magnitude of the rotation vector along the b_axis. Intuitively, we want this magnitude to be close to zero.

[lixiny]

If I understand your question correct, the abnormal poses can not be penalized without knowing its parent, or child. We use a coordinate adaptation axis layer to convert the original rotational axes to twist-bend-splay (in our code: back-left-up), and then penalize the abnormal rotations w.r.t twist-bend-splay.

Thanks a lot! One more question. How to avoid abnormal hand pose after designing local frame in your project? Such as setZero?

One more question! I am confusing about abnormal pose of thumb recently. It seems that the three joints of thumb have more DOFs than other fingers. I would like to know how to penalize the abnormal rotation of thumb specifically. Could you please help me?

We approximate the rotation DoFs on the first joints (closest to the wrist) of the thumb as a combination of motions in two orthogonal directions: bend and splay

Hope these figures illustrate our implementation:

[haolyuan]

Thanks again! I have found three loss in geo_loss.py: joint_u_axis_loss, joint_l_limit_loss and joint_b_axis_loss. For example, You calculate cos value using b_axis and axis. Do the b_axis and axis mean the twist axis direction of the same local frame with this joint's pose (include fathers' joints along kintree) and with only its father joints' pose?

Yes, the b_axis represent the back direction (aka twist in our paper), and it is calculated through: current joint's child's xyz minus current joint's xyz.

axis-angle: Since rotation can be represented as an direction (unit 3D vector, axis) and a scalar value (angle), calculating the cosine value between axis and the b_axis (unit vector along the back direction) give us the magnitude of the rotation vector along the b_axis. Intuitively, we want this magnitude to be close to zero.

Thank you! I get it! It seems that you split the axis angle direction into three vertical direction. You restrict one direction to avoid abnormal pose so the final direction is got by suming the other two direction. But in my opinion, the axis angle is not in euclidean space. That is, "exp(aa^)exp(bb^)" is not equal to "exp((aa+bb)^)". So is it available to restrict the rotation using axis angle directly? Sorry for so many questions~I just started learing pose estimation.

[lixiny]

Thanks again! I have found three loss in geo_loss.py: joint_u_axis_loss, joint_l_limit_loss and joint_b_axis_loss. For example, You calculate cos value using b_axis and axis. Do the b_axis and axis mean the twist axis direction of the same local frame with this joint's pose (include fathers' joints along kintree) and with only its father joints' pose?

Yes, the b_axis represent the back direction (aka twist in our paper), and it is calculated through: current joint's child's xyz minus current joint's xyz. axis-angle: Since rotation can be represented as an direction (unit 3D vector, axis) and a scalar value (angle), calculating the cosine value between axis and the b_axis (unit vector along the back direction) give us the magnitude of the rotation vector along the b_axis. Intuitively, we want this magnitude to be close to zero.

Thank you! I get it! It seems that you split the axis angle direction into three vertical direction. You restrict one direction to avoid abnormal pose so the final direction is got by suming the other two direction. But in my opinion, the axis angle is not in euclidean space. That is, "exp(aa^)exp(bb^)" is not equal to "exp((aa+bb)^)". So is it available to restrict the rotation using axis angle directly? Sorry for so many questions~I just started learing pose estimation.

Wow, good question! you are almost there. Indeed the sum operation upon so3 algebra should be conducted as the multiplication in SO(3) space.

However, here we use neural network to produce rotation R , but to penalize the R with respect to those abnormal direction. It is feasible to represent the rotation as the Euler angles, and the angles can only be applied to the valid directions, in our case, bend and splay. However, we find that axis-angles (recently we have moved to the ortho6D representation) is more suitable for neural networks.

[haolyuan]

Thank a lot!