isarandi / PosePile

Dataset processing scripts for large-scale 3D human pose estimation
MIT License
46 stars 9 forks source link

RICH Dataset joint selection #4

Open mertkiray opened 7 months ago

mertkiray commented 7 months ago

Hello,

Thank you for your great work. I wanted to ask how did you decide on which joints to select from the RICH dataset. In this line you selected some joints from the available joints but I wonder how did you decide on these?

For example in SMPLX Unity project they select these joints.

Do you think how one can try to obtain these joints from your work? Is it feasible to create the datasets using those joints and finetune the model or what do you suggest?

Thanks!

isarandi commented 7 months ago

Hi! Thanks for your interest! What joints are you missing for the SMPL-X body? I aimed to include all body joints. Note though that I'm currently working on a method to estimate full SMPL/SMPL-X models, so this will not be an issue.

mertkiray commented 7 months ago

So the selected joints for metrabs are: ['pelv', 'lhip', 'rhip', 'bell', 'lkne', 'rkne', 'spin', 'lank', 'rank', 'thor', 'ltoe', 'rtoe', 'neck', 'lcla', 'rcla', 'head', 'lsho', 'rsho', 'lelb', 'relb', 'lwri', 'rwri', 'jaw', 'leyehf', 'reyehf', 'lmiddle1', 'rmiddle1', 'nose', 'reye', 'leye', 'rear', 'lear', 'lto2', 'lto3', 'lhee', 'rto2', 'rto3', 'rhee', 'lthu', 'lmiddle', 'rthu', 'rmiddle']

Which I think translates to these joints from SMPLX joints named from this link: ['pelvis', 'left_hip', 'right_hip', 'spine1', 'left_knee', 'right_knee', 'spine2', 'left_ankle', 'right_ankle', 'spine3', 'left_foot', 'right_foot', 'neck', 'left_collar', 'right_collar', 'head', 'left_shoulder', 'right_shoulder', 'left_elbow', 'right_elbow', 'left_wrist', 'right_wrist', 'jaw', 'left_eye_smplhf', 'right_eye_smplhf', 'left_middle1', 'right_middle1', 'nose', 'right_eye', 'left_eye', 'right_ear', 'left_ear', 'left_big_toe', 'left_small_toe', 'left_heel', 'right_big_toe', 'right_small_toe', 'right_heel', 'left_thumb', 'left_middle', 'right_thumb', 'right_middle']

So this translates to these joints are missing in the SMPLX Unity project:

 Not found: nose, right_eye, left_eye, right_ear, left_ear, left_big_toe, left_small_toe, left_heel, right_big_toe, right_small_toe, right_heel, left_thumb, left_middle, right_thumb, right_middle

Which is fine as this is a limitation of the SMPLX fbx model in the unity.

But metrabs also lacks these joints that is available in the SMPLX fbx model:

Not found: left_index1, left_index2, left_index3, left_middle2, left_middle3, left_pinky1, left_pinky2, left_pinky3, left_ring1, left_ring2, left_ring3, left_thumb1, left_thumb2, left_thumb3, right_index1, right_index2, right_index3, right_middle2, right_middle3, right_pinky1, right_pinky2, right_pinky3, right_ring1, right_ring2, right_ring3, right_thumb1, right_thumb2, right_thumb3

Which looks like only lacking joints for metrabs <-> Unity model is most of the hand joints.

So my questions are:

1- Does my conversion between metrabs SMPLX joint names and original SMPLX joint names looks correct? 2- Do you know the difference between right_eye and right_eye_smplhf joints for example? 3- Do you plan to support the hand joints in the metrabs or do you think this is a limitation of the skeleton models and probably won't yield good results? 4- Can you explain the reasoning behind the naming convention? For example what is the meaning behind rto2, rto3 etc. Do you think it makes sense to create a documentation that maps these into a more accustomed joint naming convention?

Thank you :) If my question is not clear please let me know I can explain further.