search

bharat-b7 / RVH_Mesh_Registration

Code to fit SMPL model to scans
179 stars 21 forks source link

Inquiries Regarding RVH Mesh Registration #26

Closed hwy1992129 closed 3 months ago

hwy1992129 commented 4 months ago

Hello, and thank you for your contributions to RVH. I have encountered some issues and would appreciate your insights.

When attempting to align SMPL(H) and SMPL(H)+D with scans without providing a pose file, the outcomes were unsatisfactory. Additionally, while fitting SMPL(H)+D to scans using IP-Net, the model's dimensions were inaccurate, as depicted in the attached examples. 0 Fitting SMPL to scans 1 Fitting SMPL to scan using IP-Net

My confusion arises from the differing results obtained using the SMPL fitting and IPNet code from another source. Despite not supplying a pose file here, the results were surprisingly decent. It seems that moving from kalin0.1 to pytorch3d is not reason for this issue. I want to use the one with pytorch3d, since it has better compatibility, but the RVH leads to weird registration.. 2

I compare the code of fit_SMPL in both, and it seems that there is not much different..... However, I noted a variation in the 'prior.pkl' file—RVH's version contains a 'mean' of 69, whereas IP-Net's version has 63. Could you provide any guidance on these issues? @xiexh20

xiexh20 commented 4 months ago

Hi,

  1. Fitting SMPL-H to scan without pose file is simply ICP, which can easily stuck in local minima.
  2. Fitting SMPL-D to scan using IP-Net: I guess there is a scale and translation issue here, note that IP-Net will first do a normalization to the scan, see here: https://github.com/bharat-b7/RVH_Mesh_Registration/blob/main/smpl_registration/fit_SMPLH_IPNet.py#L37 So I guess the normalization is not undone properly.
  3. The new prior.pkl is computed from AMASS dataset, which is different from the original SMPL prior, but I guess this should not be a main issue here.
  4. Regarding the difference: which pipeline are you refering to in the IPNet repo? Is it fitting SMPL without IP-Net prediction? if so then the optimization hyperparameters are also different, i.e. iterations, lr etc.

Best,

hwy1992129 commented 4 months ago

@xiexh20 fit_SMPLH in RVH Its registration result is shown below. image

fit_SMPL in IP-Net The second one is previous IP-Net project using kaolin0.1. Its registration result is shown below. image

The scan model are not the same shown in the image, but they have the similar result... I mean the first method doesn't give the correct pose. I didn't provide pose for both registrations...

xiexh20 commented 3 months ago

Hi,

Then I guess there is some difference between pytorch3d and kaolin in terms of the implementation of point to surface distances.

But in general I don't recommend registration without body pose. The old IP-net registration might be able to handle this A-pose, but it will fail once the body pose is more complex.

hwy1992129 commented 3 months ago

@xiexh20 OK, Thanks.