More joint angles

[J0ekr]

Hey, Found this repository while also working on calculating the joint angles of mediapipe poses. Good stuff! I'm currently trying to add more mediapipe joints, which you are currently not using. Or is there any good reason you did not calculate the angles for the index, pinky, thumb, face, and also heel and toes? Do I understand the offset_direction dictionary correctly, that the vector is the relative direction to the previous joint? In that case the offset vectors for the left fingers would be:

offset_directions['LEFT_WRIST'] = np.array([1,0,0])
offset_directions['LEFT_PINKY'] = np.array([1,0,0]) # as its one unit more to the left than the left_wrist in the Blaze_pose model? 
offset_directions['LEFT_INDEX'] = np.array([1,1,0]) # as its one unit to the left and one up from the left_wrist?
offset_directions['LEFT_THUMB'] = np.array([0,1,0]) 

Sadly the reconstructed angles and thus pose dont really look too good for these newly added joints. Im happy for any input / discussion!

[TemugeB]

Hi,

I didn't add any other joints because they were not necessary for the work I was doing at the time. The joints are all defined with respect to the parent joint in the T pose. You have to define new axes like in the picture. Then the position of each finger must be with respect to the axes defined at the parent.

Additionally, you have to define the hierarchy: how they connect to the root joint. The definition is in here: https://github.com/TemugeB/joint_angles_calculate/blob/caec22c4ede33da6b8d037e3f62a8085e6a8dc20/calculate_joint_angles.py#L62

Have you tried these already? You also have to add the new joint names in the dictionary. I assume you can get the dictionary ok?

https://github.com/TemugeB/joint_angles_calculate/blob/caec22c4ede33da6b8d037e3f62a8085e6a8dc20/calculate_joint_angles.py#L27

Finally, finger positions are not very easy to obtain and can have large position variance when you triangulate to a 3D point. This might lead to a lot of errors in the obtained 3D points. You can check if your 3D points are acceptable without going through the joint angles, just plot the 3D points as a skeleton and see if they make sense.

If you're having any specific problem, you can share it here. I can't really make suggestions without knowing what kind of problem you're having.

[TemugeB]

Also, you should use a unit vector when defining an offset direction.

[J0ekr]

Hey, thanks for the quick reply!

I have added them to my hierarchy:

    'HIP': [],
    'LEFT_HIP': ['HIP'], 'LEFT_KNEE': ['LEFT_HIP', 'HIP'], 'LEFT_ANKLE': ['LEFT_KNEE', 'LEFT_HIP', 'HIP'], 
    'LEFT_FOOT_INDEX': ['LEFT_ANKLE', 'LEFT_KNEE', 'LEFT_HIP', 'HIP'], 'LEFT_HEEL': ['LEFT_ANKLE', 'LEFT_KNEE', 'LEFT_HIP', 'HIP'], #new
    'RIGHT_HIP': ['HIP'], 'RIGHT_KNEE': ['RIGHT_HIP', 'HIP'], 'RIGHT_ANKLE': ['RIGHT_KNEE', 'RIGHT_HIP', 'HIP'],
    'RIGHT_FOOT_INDEX': ['RIGHT_ANKLE', 'RIGHT_KNEE', 'RIGHT_HIP', 'HIP'], 'RIGHT_HEEL': ['RIGHT_ANKLE', 'RIGHT_KNEE', 'RIGHT_HIP', 'HIP'], #new
    'NECK': ['HIP'], 'NOSE': ['NECK', 'HIP'], 
    'LEFT_EYE_INNER': ['NOSE', 'NECK', 'HIP'], 'LEFT_EYE': ['LEFT_EYE_INNER', 'NOSE', 'NECK', 'HIP'], 'LEFT_EYE_OUTER': ['LEFT_EYE', 'LEFT_EYE_INNER', 'NOSE', 'NECK', 'HIP'], 'LEFT_EAR': ['LEFT_EYE_OUTER', 'LEFT_EYE', 'LEFT_EYE_INNER', 'NOSE', 'NECK', 'HIP'], #new
    'RIGHT_EYE_INNER': ['NOSE', 'NECK', 'HIP'], 'RIGHT_EYE': ['RIGHT_EYE_INNER', 'NOSE', 'NECK', 'HIP'], 'RIGHT_EYE_OUTER': ['RIGHT_EYE', 'RIGHT_EYE_INNER', 'NOSE', 'NECK', 'HIP'],'RIGHT_EAR': ['RIGHT_EYE_OUTER','RIGHT_EYE', 'RIGHT_EYE_INNER', 'NOSE', 'NECK', 'HIP'], #new
    'LEFT_SHOULDER': ['NECK', 'HIP'], 'LEFT_ELBOW': ['LEFT_SHOULDER', 'NECK', 'HIP'], 'LEFT_WRIST': ['LEFT_ELBOW', 'LEFT_SHOULDER', 'NECK', 'HIP'],
    'LEFT_PINKY': ['LEFT_WRIST', 'LEFT_ELBOW', 'LEFT_SHOULDER', 'NECK', 'HIP'], 'LEFT_INDEX': ['LEFT_WRIST', 'LEFT_ELBOW', 'LEFT_SHOULDER', 'NECK', 'HIP'], 'LEFT_THUMB': ['LEFT_WRIST', 'LEFT_ELBOW', 'LEFT_SHOULDER', 'NECK', 'HIP'], #new
    'RIGHT_SHOULDER': ['NECK', 'HIP'], 'RIGHT_ELBOW': ['RIGHT_SHOULDER', 'NECK', 'HIP'], 'RIGHT_WRIST': ['RIGHT_ELBOW', 'RIGHT_SHOULDER', 'NECK', 'HIP'],  
    'RIGHT_PINKY': ['RIGHT_WRIST', 'RIGHT_ELBOW', 'RIGHT_SHOULDER', 'NECK', 'HIP'], 'RIGHT_INDEX': ['RIGHT_WRIST', 'RIGHT_ELBOW', 'RIGHT_SHOULDER', 'NECK', 'HIP'], 'RIGHT_THUMB': ['RIGHT_WRIST', 'RIGHT_ELBOW', 'RIGHT_SHOULDER', 'NECK', 'HIP'], #new

My offsets:

offset_directions = {}
offset_directions['LEFT_HIP'] = np.array([1,0,0])
offset_directions['LEFT_KNEE'] = np.array([0,-1, 0])
offset_directions['LEFT_ANKLE'] = np.array([0,-1, 0])
offset_directions['LEFT_FOOT_INDEX'] = np.array([1,0, 0]) #new
offset_directions['LEFT_HEEL'] = np.array([0,-1, 0]) #new

offset_directions['RIGHT_HIP'] = np.array([-1,0,0])
offset_directions['RIGHT_KNEE'] = np.array([0,-1, 0])
offset_directions['RIGHT_ANKLE'] = np.array([0,-1, 0])
offset_directions['RIGHT_FOOT_INDEX'] = np.array([-1,0, 0]) #new
offset_directions['RIGHT_HEEL'] = np.array([0,-1, 0]) #new

offset_directions['NECK'] = np.array([0,1,0])
offset_directions['NOSE'] = np.array([0,1,0]) #new

offset_directions['LEFT_EYE_INNER'] = np.array([1,0,0]) #new
offset_directions['LEFT_EYE'] = np.array([1,0,0]) #new
offset_directions['LEFT_EYE_OUTER'] = np.array([1,0,0]) #new
offset_directions['LEFT_EAR'] = np.array([1,0,0]) #new

offset_directions['RIGHT_EYE_INNER'] = np.array([-1,0,0]) #new
offset_directions['RIGHT_EYE'] = np.array([-1,0,0]) #new
offset_directions['RIGHT_EYE_OUTER'] = np.array([-1,0,0]) #new
offset_directions['RIGHT_EAR'] = np.array([-1,0,0]) #new

offset_directions['LEFT_SHOULDER'] = np.array([1,0,0])
offset_directions['LEFT_ELBOW'] = np.array([1,0,0])
offset_directions['LEFT_WRIST'] = np.array([1,0,0])
offset_directions['LEFT_PINKY'] = np.array([1,0,0]) #new
offset_directions['LEFT_INDEX'] = np.array([1,0,0]) #new
offset_directions['LEFT_THUMB'] = np.array([1,0,0]) #new

offset_directions['RIGHT_SHOULDER'] = np.array([-1,0,0])
offset_directions['RIGHT_ELBOW'] = np.array([-1,0,0])
offset_directions['RIGHT_WRIST'] = np.array([-1,0,0])
offset_directions['RIGHT_PINKY'] = np.array([-1,0,0]) #new
offset_directions['RIGHT_INDEX'] = np.array([-1,0,0]) #new
offset_directions['RIGHT_THUMB'] = np.array([-1,0,0]) #new

For the finger-joints I'm not sure if it's better to define them to be all 1 more to the left/right than the wrist joint, or if, e.g., the thumb joint should go up in relation to the wrist. (Taking the following picture as reference)

Finally, finger positions are not very easy to obtain and can have large position variance when you triangulate to a 3D point. This might lead to a lot of errors in the obtained 3D points. You can check if your 3D points are acceptable without going through the joint angles, just plot the 3D points as a skeleton and see if they make sense.

This might be my issue then. My reconstructed positions for the fingers & feets just seem to be a bit off (however, the original positions are fine): But I'll just keep playing around with the offsets and update where I end up.

[TemugeB]

Try inputting all zero angles. This should reproduce the T pose. Check if it looks like what you expect.

[J0ekr]

That's a good tip, cheers! Looking like this right now (funnily enough I found out that my Tpose is flipped in contrast to the regular poses)

[AJDA1992]

@J0ekr can you share the code you used to include joints for the full skeleton?

[mnauf]

@J0ekr can you share the code you used to include joints for the full skeleton?

@AJDA1992 Were you able to fix this / code this?

[J0ekr]

@mnauf @AJDA1992 I have since moved to use the join positions. I never got the angles fully to work as I wanted them to, although I am kinda interested in getting it to work again.

I quickly created this gist from my Jupiter notebooks from back when I was working on this, some imports might be missing: https://gist.github.com/J0ekr/0534d28a58b08432f36bbb13406baed9

[TemugeB]

@J0ekr If T-pose is flipped as you say, then you might be indexing the points in the flipped order.

The keypoints indices are defined here. Try flipping the new keypoints indices. https://github.com/TemugeB/joint_angles_calculate/blob/caec22c4ede33da6b8d037e3f62a8085e6a8dc20/calculate_joint_angles.py#L27