reconstruction with more views

[LeeHY-1]

Hi, thanks for your work, I have successfully finish the 3d pose estimation with 2 views. But now I have 8 cameras, so I want to know whether I can use this work to reconstruct 3d skeleton with more than 2 views. Thank you.

[TemugeB]

Hi. Sorry for the delayed response. You can use any number of views in principle. However, it will no longer be real-time. Please read the DLT guide here: link.

To add more views, you need to update the A 4x4 matrix on line 14 of utils.py:

    A = [point1[1]*P1[2,:] - P1[1,:],
         P1[0,:] - point1[0]*P1[2,:],
         point2[1]*P2[2,:] - P2[1,:],
         P2[0,:] - point2[0]*P2[2,:]
         point3[1]*P3[2,:] - P3[1,:],
         P3[0,:] - point3[0]*P3[2,:]
         ...
        ]
        A = np.array(A).reshape((num_cam_views,4))

and so on until you have matrix elements from each of your camera view. Above, point3 is the pixel coordinates of your point in camera#3 view and P3 is the projection matrix of P3. After that, you need to multiply by the transpose matrix and run SVD decomposition (check the link for details). Don't forget that you want the last element of the SVD decomposition (Vh in my code).

If real-time work is necessary for you, you might look into running mediapipe on GPU. Also, if you have confidence scores on each of your 8 views, then you might try passing only high confidence points to the triangulation code. So that the A matrix is calculated based on the number of high confidence points.

Let me know if you have any questions.

[LeeHY-1]

Thanks for your reply, I'll try it soon. And now I want to know if this work can be used in multi person scenario. If so, what other work should I do? Thank you.

[TemugeB]

Mediapipe itself does not support multi person detection. But you can switch mediapipe with other works that do detect multi-person keypoints (e.g. openpose). However they are hard to set up and usually suffer from poor generalization.

If you can get the bounding boxes of the people in your frame using some other method, then you can crop each individual from your frame and pass that to mediapipe. However, this might not work because mediapipe keeps the bounding box of person in previous frame and uses it to crop out the person in the next frame. So you have to be careful not to move your person crop too much between frames. I'm not sure how this will work in practice.

[LeeHY-1]

Thanks you for the answer, I will try some other methods as you said.

[TemugeB]

I assume you have no more issues. I'm closing this issue.