search

hassony2 / manopth

MANO layer for PyTorch, generating hand meshes as a differentiable layer
GNU General Public License v3.0
601 stars 104 forks source link

Joints prediction is outside the mesh #10

Closed adrianofragomeni closed 4 years ago

adrianofragomeni commented 4 years ago

Hello,

I have a problem when I use this MANO implementation. After some epochs, some predicted joints are outside the mesh. How is it possible? I attach a picture to give an idea about the problem. Capture

hassony2 commented 4 years ago

Hi @adrianofragomeni,

Thank you for your feedback ! You mention that you are training the method, what code are you using for training ? Would it be possible that you are optimizing over buffer parameters of the mano model ?

Best,

Yana

adrianofragomeni commented 4 years ago

HI @hassony2, I am using the manobranch.py you used in obman_train, where the parameters self.lambda_shape and self.lambda_pose_reg are not None. For this reason am using two losses on these Mano parameters. Could it be this the problem?

Best, Adriano

hassony2 commented 4 years ago

Something that might produce such results would be you having switched the parameter mano_adapt_skeleton https://github.com/hassony2/obman_train/blob/83be15c342e24b332555b75b69544cb2eaa65cdb/mano_train/networks/handnet.py#L49 to True, which allows the method to also redefine hand joints during training.

hassony2 commented 4 years ago

I don't think it would have to do with self.lambda_shape or self.lambda_pose_reg, but I am not 100% sure :)

adrianofragomeni commented 4 years ago

You were right, I had the mano_adapt_skeleton parameter set to True. I will try to switch it to False and I let you know if it works.

Thank you

Adriano Fragomeni

foamliu commented 4 years ago

Thank you very much for the great work!

Just in case it helps a little bit: do we need to adjust joint locations after the vertice are conditioned on pose?

image

dimtzionas commented 4 years ago

Dear @foamliu , thank you for your comment. The use of the regressor is right as it is :)

Please have a look at the SMPL paper, the formulation of which we follow also for MANO.

Please see Eq.10 on page 5. The joint regressor of the model is applied after applying the shape blend shapes on the average template, and before posing the model.

This is a common misconception, because other people train different regressors that are applied after applying both the shape and pose blend shapes. Any regressor should be applied in the exact way it is trained for; different regressors might be trained for different ways of use :)

Regarding the SMPL regressor, if the above seems counter-intuitive to you, you can think of this as the skeleton within your own body. Your skeleton joints are a function of your body shape (e.g. height) and not your pose; the fact that you lift your arms should not be changing your joint positions with respect to your body (just an intuitive example, not necessarily anatomically correct).