search

korrawe / harp

HARP: Personalized Hand Reconstruction from a Monocular RGB Video
68 stars 10 forks source link

Unable to optimize nimble param to fit nimble meshes to mano meshes. #6

Open delaprada opened 1 year ago

delaprada commented 1 year ago

Hi, may I ask how to optimize the nimble param to fit nimble meshes to mano meshes?

I take your advice in the previous issue to use the method you provided to fit nimble meshes to mano meshes, but unfortunately it can not converge. I encounter a few problems.

The first problem is the nimble layer you used in the method seems to be different from the original NIMBLE repo? The nimble layer provided in the original NIMBLE repo doesn't have cur_rot, cur_trans, global_scale and no_tex params. Could you kindly provide your version of nimble layer?

The second problem is that it seems there are no regularizations on pose and shape parameters during the coarse and fine alignment stage. I run the method but the generated hand mesh is distorted. I guess it is the reason of not adding regularization. Did you optimize well without adding the regularization terms?

I try to add a regularization term to optimize, but the value of weighting factor is difficult to adjust in order to fit nimble meshes to mano meshes. It would be nice of you if you could provide some suggestions for optimization.

Thank you!

korrawe commented 1 year ago

Hi, The code I use to fit Nimble is the one that you link to. The cur_xxx only means xxx in the current frame. Here is my version of NIMBLELayer.py. I only changed the code a bit to make it easier to run with optimization.

For your second question, I did not use regularization because I retry the optimization when the difference between the starting vertices and the final vertices is too high. This happens when the mesh is distorted after optimization. But, you can add the regularization term to keep the pose and shape parameters close to the mean (zero values). Although, from my experience, the regularization terms do not work very well with Nimble PCA space

As this issue is not directly related to HARP, I will try to help but only if I have free time. Please expect a less frequent answer. Still, I will keep this open in case someone wants to help. Best,

delaprada commented 1 year ago

Hi, The code I use to fit Nimble is the one that you link to. The cur_xxx only means xxx in the current frame. Here is my version of NIMBLELayer.py. I only changed the code a bit to make it easier to run with optimization.

For your second question, I did not use regularization because I retry the optimization when the difference between the starting vertices and the final vertices is too high. This happens when the mesh is distorted after optimization. But, you can add the regularization term to keep the pose and shape parameters close to the mean (zero values). Although, from my experience, the regularization terms do not work very well with Nimble PCA space

As this issue is not directly related to HARP, I will try to help but only if I have free time. Please expect a less frequent answer. Still, I will keep this open in case someone wants to help. Best,

Thank you so much! I will try your NIMBLELayer.py first to see whether it can improve the optimization.