search

jyf588 / transformer-inertial-poser

Python implementation accompanying the Transformer Inertial Poser paper at SIGGRAPH Asia 2022
https://arxiv.org/abs/2203.15720
Other
69 stars 10 forks source link

Visualize with SMPL: The definition of 18 joints #4

Closed yukichou closed 1 year ago

yukichou commented 1 year ago

Thanks for doing such great job. The paper mentioned that the output will be18 joints angles defined in SMPL human model, but SMPL has 24 joints, so what's the difference between 18 and 24 joints? How can I use 18 joints to drive the normal SMPL model?

jyf588 commented 1 year ago

Hello! There are six joints ignored — two toes, two wrists, and two palms (i.e. “hand” per SMPL naming). One cannot reasonably estimate them from only 6 IMUs. Previous works also freeze them when using SMPL to visualize I believe.

I’m not sure about the coding details, but it shouldn’t be hard to swap the visualizer from PyBullet to SMPL. The main thing you need to figure out is the different ordering we use for the 18 SMPL joints. This joint ordering of our model’s output, unsurprisingly, is the same as the ordering during our data synthesis. So this function can help you crack out the joint correspondence.

yukichou commented 1 year ago

I see. Thank you so much!

syguan96 commented 1 year ago

Hi @jyf588 , Thanks for your great work and code! I have another related question. Hope you can help me. Previous works (DIP, TransPose) not only ignore [10,11,20,21,22,23] but also ignore [7,8]. The joint index definition is shown below. What's your motivation to additionally predict the rotation of Joint 7,8? image

jyf588 commented 1 year ago

Hi @syguan96, that's a good question. The decision was somewhat arbitrary -- in the beginning of this project we thought that ankle angles can be pretty much guessed from the full-body pose most of the time; just remember that our foot need to always make static contact with the ground (also see the SBP section of the paper) which constrains possible ankle motion. We did not revisit or change this design choice later on.

syguan96 commented 1 year ago

Make sense. Thanks! @jyf588
Last question: When you compared with previous works in terms of angle and position, did you directly ignore ankle joints?

jyf588 commented 1 year ago

No, we probably just took the SMPL-format output from previous works and did not exclude ankles. But honestly I think either way won't affect the statistics much since ankle is just a leaf joint.

SlimeVRX commented 1 year ago

I just converted the output to SMPL format

https://user-images.githubusercontent.com/99313947/232146454-75b56353-842b-4f24-bcf8-2481daceae39.mp4

jyf588 commented 1 year ago

@SlimeVRX Nice! If you are willing to post a Github gist link here sharing with others how you did this, that would be really helpful to other researchers and practitioners.

SlimeVRX commented 1 year ago

Hi @jyf588 Thanks for your work! Here is the conversion code from Pybullet to SMPL. I am working to compare TransPose, PIP, TIP articles with TotalCapture dataset.

SlimeVRX commented 1 year ago

I write a UI to control SMPL model. Code here

SMPL model:

https://user-images.githubusercontent.com/99313947/232261816-abfa7123-5d02-456b-99b8-ffd5d233cf5d.mp4

Pybullet model:

https://user-images.githubusercontent.com/99313947/232261818-5e9e763a-3641-4451-bbd1-8fd2de68e121.mp4

WingkitChou commented 1 year ago

That's soooo cool, and i think it's really helpful for others! Thank you for your work.