search

saidwivedi / TokenHMR

[CVPR 2024] TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation
https://tokenhmr.is.tue.mpg.de
Other
213 stars 18 forks source link

SKEL model instead of SMPL #7

Closed abhishekmonogram closed 2 months ago

abhishekmonogram commented 3 months ago

Hi, Thank you so much for sharing this repository. I wanted to check if a SKEL body model could be used in place of SMPL? Incase the answer is yes, what would I have to do to use SKEL here?

saidwivedi commented 3 months ago

I am not sure if I can answer that. Maybe Marilyn, lead author of SKEL, can answer.

MarilynKeller commented 3 months ago

Hi, Token HMR was trained on images with ground truth SMPL fits, so it cannot directly output SKEL parameters.

However, you can obtain SKEL parameters by fitting SKEL to the SMPL mesh output from Token HMR using this script. The idea is to optimize the SKEL pose q to minimize the vertex to vertex distances ∣∣SKEL(β,q)skin − SMPL(β,θ)∣∣.

For direct regression, you could also retrain Token HMR on SKEL. To do this, you would need to fit SKEL to the entire training dataset SMPL meshes and then use the pose parameter of SKEL (q) instead of SMPL's θ in the architecture. But this would be a very consequent work.