search

dancasas / SMPLitex

SMPLitex – Official implementation
92 stars 7 forks source link

Help please! I'm going crazy - How to get weak camera perspective working with pytorch3d #9

Closed gt732 closed 2 weeks ago

gt732 commented 2 weeks ago

Hi,

Thanks for the hard work here. I have everything working with applying the texture to the SMPL model the issue I'm facing is getting the camera perspective correct. I'm using PARE to extract SMPL data from a reference video. PARE uses weak camera perspective, I want to render the exact perspective of the original video used to get the SMPL data.

FROM PARE documentation - https://github.com/mkocabas/PARE pred_cam (n_frames, 3) # weak perspective camera parameters in cropped image space (s,tx,ty) orig_cam (n_frames, 4) # weak perspective camera parameters in original image space (sx,sy,tx,ty)

How can I convert this into pytorch3d camera? I tried for many hours/days but the camera is always incorrect.

image

Ref video frame 1920x1080

image

Any code references you can point me to? This is the last hurdle I'm facing!

Thanks!!

gt732 commented 2 weeks ago

Fixed.

dancasas commented 2 weeks ago

Sorry I couldn't jump earlier to this issue, here's how I set up the PyTorch3D cameras, which should render the SMPL meshes upright facing frontal.

https://github.com/dancasas/SMPLitex/blob/0476822a8273f96ee8ab3463560d6968c46697c7/scripts/utils/renderer/pytorch3d_renderer.py#L240-L246

How did you fix your issue?

gt732 commented 2 weeks ago

@dancasas Np, thanks for the reply! I was able to fix it by using another library mmhuman3d. They have a ton of helper function that set up the correct camera for you when you pass the cam params from VIBE.

https://github.com/open-mmlab/mmhuman3d/blob/8c95747175f8c4bd76d5e0be7be244f8a9a4a6de/mmhuman3d/core/visualization/visualize_smpl.py#L1129

I was able to use the texture provided by this library doing the rendering process in mmhuman3d.

This is the results!

https://github.com/user-attachments/assets/1cb7dd47-adf3-4146-a199-50263626f2dd

https://github.com/user-attachments/assets/dbd8fb89-487e-459a-ad60-dce6b0d3e5a3

dancasas commented 2 weeks ago

Looks great -- thanks for the pointers to mmhuman3d, very interesting! Thansk!