Can I directly generate bones with different poses and shapes by controlling shape parameters and pose parameters (like smpl)?

[zbc-l]

Thank you for your excellent work. Can I directly generate bones with different poses and shapes by controlling shape parameters and pose parameters (like smpl)? If so, is the shape parameter of the skeleton this? https://github.com/MarilynKeller/OSSO/blob/761e1abb87edb97ae80e1a68705d95b43ecd4865/osso/utils/inference.py#L31 Where are the posture parameters?

[MarilynKeller]

Hi, this model does not have a shape space, we actually create a new skeleton template given specific betas skel_betas. So you have to recreate the model everytime you change the beta parameters. This means that the output bone shapes is not differentiable wrt the skeleton betas. If you want to change the bones shape in a differentiable way, you might want to have a look at the base skeleton model by Silvia Zuffi, there you can modify the bone shapes as demonstrated in her demo code.

The bones pose are controled by two parameters. A per bone rotation sp.r_abs and a per bone translation sp.t (see here)[https://github.com/MarilynKeller/OSSO/blob/761e1abb87edb97ae80e1a68705d95b43ecd4865/osso/utils/inference.py#L60]. For each bone, the rotation center is the center of the bone, so if you change the rotation parameter alone, the bones won't be stitched together anymore. For this reason, this skeleton model can only be pose using optimization, i.e. optimizing sp.r_abs and sp.t while minimizing a stitching cost that forces the bones to stay stiched together as done here.

I hope it clarifies :)

[zbc-l]

Thank you for your patient explanation. I have understood the part about posture, but I still don't understand the part about shape. I don't really understand what it means to have to recreate the model every time you change the beta parameter. Can I understand that skel_beta is a special β computed from the β input to star, we can't change skel_beta to change the skeleton shape, every time we input a star model with a different shape, a new skel_beta is generated based on the β of star to create the model?

[zbc-l]

Is skel_beta the β in sp (β, t, r)?

[MarilynKeller]

Yes, what you wrote is correct. Let me detail what it means to have to recreate the model every time we change the beta parameter. The shape setting is done in this script: osso/utils/inference.py

So given STAR's β, we compute a skeleton beta β_B and generate a personalised mesh in T pose. https://github.com/MarilynKeller/OSSO/blob/761e1abb87edb97ae80e1a68705d95b43ecd4865/osso/utils/inference.py#L105

Then, using this personalised mesh as a template, we instanciate a new parametric skeleton model https://github.com/MarilynKeller/OSSO/blob/761e1abb87edb97ae80e1a68705d95b43ecd4865/osso/utils/inference.py#L111

In this model initialisation, we take the mesh skel_vertices just created, split it into individual bone groups:

Then we offset each bone so that the bones with the new shape for a T pose again to create the model template. Indeed If I change the shape of the bone and it gets longer, and I apply the same gv.t parameter to this bone than the initial template, the bones will interpenetrate. This offset fixes that. https://github.com/MarilynKeller/OSSO/blob/761e1abb87edb97ae80e1a68705d95b43ecd4865/osso/utils/init_skel_model.py#L52 Once we have a model gv initialised with a personal template, we can set the pose with gv.t and gv.rot_abs.

So to summarize, for each new beta, we generate a skeleton mesh, split it in parts and offset each part properly, and set the result as template of a new model. In the current implementation, this is not differenciable, I guess it could be made differentiable though. Like, you initialise the model once, and given the skin_beta parameter, compute g.v in a differentiable way.

[zbc-l]

Your reply is very detailed. Thank you for your patient reply.