[MVP] Regarding inconsistent MVP 3d estimation

[yitongx]

Hi,

First of all, thanks for the great work of openxrlab!

I'm testing MVP estimation with default Panoptic 5 views. I'm struggling with inconsistent 3d keypoints estimation.

In the converter configuration, for bbox_detector I tried both "MMtrackDetector", and "MMdetDetector", and for kps2d_estimator I'm using default "MMposeTopDownEstimator" with pretrained "hrnet_w48_coco_wholebody". From the perception 2d view of detected keypoints, it seems that "MMdetDetector" are more sensitive to hidden bodies and produces more small noisy poses.

https://github.com/openxrlab/xrmocap/assets/49282413/7f4d12a7-2094-4526-9d9a-1ddc542351ac

https://github.com/openxrlab/xrmocap/assets/49282413/64ba8a40-9ada-4e32-b15b-734dfb8ca807

And the final result of MVP with "MMdetDetector" is as follows. "MMtrackDetector" also has similar inconsistent 3d keypoints.

I'm following xrmocap/configs/modules/core/estimation/mview_mperson_end2end_estimator.py but still got inconsistent 3d keypoints. I guess it might be due to some inappropriate hyperparameters. Would appreciate some suggestions.

https://github.com/openxrlab/xrmocap/assets/49282413/57f0c9c6-513c-492c-9f68-a8ed799f2166

Best regards.

[github-actions[bot]]

Hi @yitongx, welcome to commit your first issue! 你好 @yitongx，非常欢迎首次提交你的问题！

[wqyin]

Hello, since MVP is a single-frame method, we need to apply tracking and optimization on the raw output to get consistent results for each person. To solve inconsistent 3dkeypoints estimation as shown in the video, you may try to adjust the threshold for post-processing, especially for duplicate identity removal and tracking in the config.

[yl-1993]

This issue is closed as it has been inactive for a while. Feel free to re-open it if the problem is not solved.