Quick Clarification on mv1p.py and mocap.py for SMPL

Hello @chingswy,

So, I was testing the mv1p.py and trying to generate smpl from the groundtruth keypoints.

 joint_idx = [0, 1, 2, 3, 6, 7, 8, 12, 16, 14, 15, 17, 18, 19, 25, 26, 27]

These joint idx are given in the h36m files. I am using h36m [ 17 ] ground-truth 2D keypoints, The detection was correct and I received the repro folder. But the repro-smpl and smpl were wrong

COMMAND USED

python3 apps/demo/mv1p.py ${data} --out ${data}/output/smpl --vis_repro --undis --body h36m --subs 1 2 3 4 --sub_vis 1 2  --vis_smpl

Detection

000004

Repro smpl

000004

SMPL Mesh

000004

But while optimizing the Joints I got the following error:

  -> [optimize joints     ]:  11.9ms
  Traceback (most recent call last):
    File "apps/fit/fit.py", line 45, in <module>
      fitter.fit(body_model, dataset)
    File "/home/ndip/EasyMocap/easymocap/multistage/base.py", line 298, in fit
      body_params, infos = self.fit_data(data, body_model)
    File "/home/ndip/EasyMocap/easymocap/multistage/base.py", line 276, in fit_data
      body_params = self.fit_stage(body_model, body_params, infos, stage, irepeat)
    File "/home/ndip/EasyMocap/easymocap/multistage/base.py", line 235, in fit_stage
      self.optimizer_step(optimizer, closure, weights)
    File "/home/ndip/EasyMocap/easymocap/multistage/base.py", line 168, in optimizer_step
      loss_dict = closure(debug=True)
    File "/home/ndip/EasyMocap/easymocap/multistage/base.py", line 155, in closure
      loss_dict[key] = loss_func(kpts_est=kpts_est, **new_params, **infos)
    File "/home/ndip/miniconda3/envs/humannerf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
      return forward_call(*input, **kwargs)
    File "/home/ndip/EasyMocap/easymocap/multistage/lossbase.py", line 356, in forward
      return self.loss(est, self.keypoints, self.conf)
    File "/home/ndip/EasyMocap/easymocap/multistage/lossbase.py", line 44, in loss
      square_diff = reduce((est - gt)**2, dim=-1)
  RuntimeError: The size of tensor a (25) must match the size of tensor b (17) at non-singleton dimension 1

I think this is because, 17 H36m keypoints to Body25 is not possible directly. Am I right? that's why the 2d detection is accurate but the smpl conversion is not. Body25 is based on Openpose I suppose, which is Coco17 to 25 joints. Also mapping of openpose and human3.6 joints are different too.

I tried the same on Mocap.py but the same error appeared.

https://github.com/zju3dv/EasyMocap/blob/b44fa3c90b6bbdb3b3fece87fc891cbf154c99f8/easymocap/dataset/config.py#LL626C1-L649C19

Can you tell me what should I do here, in order to avoid this issue? Or is it not possible in Easymocap except if we use Hrnet or Openpose to detect the joints? Since the J regressor used here is body25 based

What if I use human3.6 j regressor in place of body25 one. Will that provide any output difference? I have the H36m j regressor from VIBE and SPIN J_regressor_h36m and j_regressor_extra

What changes in case thats possible would be helpful to know.

Kindly let me know @chingswy

Thank you

zju3dv / EasyMocap