SMPL fittings in motion_chunks_final looks worse than input

[xiyichen]

Hi, Nice work! I tested your method on a custom video. The 3D positions look good. However, the SMPL fittings in the final visualization video (motion_chunks_final_000260_src_cam.mp4), as shown in the first screenshot, somehow look worse than those in the input video (input_final_000000_src_cam.mp4), as shown in the second screenshot, especially the feet poses.

I also runned the optimization on the demo video you provided in the colab notebook, and the SMPL fittings do look better in the motion_chunks_final video. Do you have any idea why the SMPL fittings are worse after optimization in my custom video?

[geopavlakos]

Would it be possible to share the original video for us to inspect it?

[xiyichen]

Thanks for your quick response. This video is the bike sequence from the NeuMan dataset. It contains 104 frames. You can download the whole dataset from here.

[geopavlakos]

The framerate of the video seems very low (maybe 2-3 fps?). The motion prior we use (HuMoR) is trained with sequences of 30fps, so this is also expected at test time (based on our observation, 20-25 fps is probably ok too). The lower framerate of the video can be an issue for the optimization of the last stage of slahmr which depends on HuMoR. This can explain the lower quality of the results you observe.

[xiyichen]

Thanks for your response. I see the issue. I'll try it on videos with a higher fps.

[xiyichen]

I also have a question on the optimized camera parameters. I extracted SMPL parameters, cam_R, cam_t, and the intrinsic matrix from '_000260_world_results.npz' (the last one). However, when I convert the SMPL vertices from the world space to the camera space and project it onto the image, the calibration does not look right: I wonder if I am using the wrong camera parameters (there are some other camera parameters in _cameras_000260.json) or is there any conversion needed?

[geopavlakos]

For this, it's better to check and follow the rendering/visualization functions, to confirm that you are using the right intrinsics/extrinsics parameters and human body coordinates.