search

mateuszwyszynski / PoseNDF

Implementation of Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fields
Other
0 stars 0 forks source link

nerfstudio for pose visualisation #8

Closed mateuszwyszynski closed 6 months ago

mateuszwyszynski commented 6 months ago

nerfstudio offers a web-based 3D visualization tool called viser. It works with smplx, so I believe we should use it to visualize poses. It will be much more convenient than rendering pictures or videos. One will be able to, for example, look at a pose from different angles or play around with joint parameters.

mateuszwyszynski commented 6 months ago

There is a problem to run vposer with basic SMPL model. It works well with SMPLX. The problem is actually inside the smplx package itself.

Traceback (most recent call last):
  File "/home/mateusz/Programming/viser/./examples/08_smplx_visualizer.py", line 287, in <module>
    tyro.cli(main, description=__doc__)
  File "/home/mateusz/anaconda3/envs/viser-python-3.9/lib/python3.9/site-packages/tyro/_cli.py", line 187, in cli
    output = _cli_impl(
  File "/home/mateusz/anaconda3/envs/viser-python-3.9/lib/python3.9/site-packages/tyro/_cli.py", line 454, in _cli_impl
    out, consumed_keywords = _calling.call_from_args(
  File "/home/mateusz/anaconda3/envs/viser-python-3.9/lib/python3.9/site-packages/tyro/_calling.py", line 241, in call_from_args
    return unwrapped_f(*positional_args, **kwargs), consumed_keywords  # type: ignore
  File "/home/mateusz/Programming/viser/./examples/08_smplx_visualizer.py", line 96, in main
    output = model.forward(
  File "/home/mateusz/anaconda3/envs/viser-python-3.9/lib/python3.9/site-packages/smplx/body_models.py", line 360, in forward
    full_pose = torch.cat([global_orient, body_pose], dim=1)
RuntimeError: Tensors must have same number of dimensions: got 2 and 3

As described in the error, the problem is that the two tensors have different dimensions (2 and 3). I believe that when using SMPLX model the call to the torch.cat from above is simply different and this is why the code works with the SMPLX model. I have checked the smplx package code more or less and it seems to be the case. All lines where the full_pose is constructed are different and incorporate some kind of reshaping (or relate to other model parts like hand_pose)

As a side note, for some reason body_pose in the SMPL model has more elements than in the SMPLX model: the shape is torch.Size([1, 23, 3]) instead of torch.Size([1, 21, 3]). The shape of global_orient is the same for both models, i.e. torch.Size([1, 3]). This however most likely is not the cause of the issue - I tried cutting the last two rows (i.e. artificially changing 21 to 23) of body_pose and still got the error.

I think the best option is to switch to the SMPLX model for the whole repo. It is the newest and the most sophisticated model. At the same time it works smoothly with viser