Back_projection of 3D keypoints

open-mmlab / mmpose

OpenMMLab Pose Estimation Toolbox and Benchmark.

https://mmpose.readthedocs.io/en/latest/

Apache License 2.0

5.86k stars 1.25k forks source link

Back_projection of 3D keypoints #1708

Open iulsko opened 2 years ago

iulsko commented 2 years ago

hi, how can i implement back-projection of original keypoints onto the image in top_down_video_demo_with_mmdet.py ?

ly015 commented 2 years ago

Hi, the top-down video demo with mmdet performs 2D keypoint detection from the video and 3D keypoint is not involved. Could you please further explain your needs?

iulsko commented 2 years ago

@ly015 hi, i am so sorry, i meant body3d_two_stage_video_demo.py and how can i back-project keypoints_3d?

ly015 commented 2 years ago

Please refer to the camera_to_pixel method of the camera class.

iulsko commented 2 years ago

@ly015 this should be applied after line 458 right?

ly015 commented 2 years ago

Camera-to-pixel transformation should be applied to the model output at L442. There are a few technical details that you may need to pay attention to:

The predicted 3D keypoints could be reletave coordinates around the root (usually pelvis) joint (depending on the model configs)
The predicted 3D keypoint coordinates could be normalized (depending on the model configs), and you may need to restored their scale before projecting them to the image space
The argument norm_pose_2d should be False
The projected 2D coordinates are in a local coordinate space defined by the bbox

iulsko commented 2 years ago

(edited)

dear @ly015 thank you so much for your answer! i will consider these tips. i have a small "to understand" question:

do you follow the original implementation of VideoPose3D exactly?

ly015 commented 2 years ago

Yes, we followed the model details and experiment settings from the original paper and implementation. As for the demo script, we added some tricks (e.g. normalize 2D inputs, transform output coordinate space for better visualization, temporal smoothing, et al) for better results, which may differ from the original implementation.

iulsko commented 2 years ago

@ly015, I am sorry, I have another question, it is because I am still confused about 3D-back-to-2D, if i have a relative-to-root coordinate situation, how can I bring it back to the camera 3D frame?

ly015 commented 2 years ago

In this case, the model needs to further predict the absolute coordinates of the root joint in the camera space, or I am afraid it's impossible to project the 3D keypoints back to the image space and completely align with the original image.