search

CMU-Perceptual-Computing-Lab / openpose

OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation
https://cmu-perceptual-computing-lab.github.io/openpose
Other
31.28k stars 7.87k forks source link

Multiple definitions of the camera matrix are confusing #1060

Closed ravijo closed 5 years ago

ravijo commented 5 years ago

Issue Summary

Multiple definitions of the camera matrix are confusing

The camera matrix is defined in the following two ways:

  1. CameraMatrix: The extrinsic parameters of the camera (Please see 2(ii) inside Camera Calibration)
  2. cameraMatrix: 3x4 camera matrix of the camera which is equivalent to cameraIntrinsics * cameraExtrinsics (Please see line 196 datum.hpp)

    Posting rules

Type of Issue

System Configuration

OpenPose version: Latest GitHub code

Questions

Due to multiple definitions of the camera matrix, I got confused. Following are my questions-

  1. What exactly is the camera matrix? Is it the dot product of camera Intrinsics and camera extrinsics?
    • If yes, then why do we need it? since it can be derived anytime by applying dot product.
  2. In OpenPose, the camera extrinsics matrices for all cameras are relative to camera 1. Hence, for camera 1, the camera extrinsics matrix shoud be an identity matrix. Am I thinking right?
    • If yes, then instead of defining camera extrinsics matrices w.r.t. camera 1, is it okay if we define cameraExtrinsics w.r.t. any other stationary base such as floor (I do have those transformations with me)?
gineshidalgo99 commented 5 years ago

The doc is not defined differently, it is referring to different places of the code. I believe you are confusing the XML file (where only CameraMatrix exists, being the equivalent of Datum::cameraExtrinsics), with Datum::cameraMatrix and Datum::cameraExtrinsics.

  1. Yes, it is the dot product (definition 2, the one in Datum). The code uses it several times, I just did it to avoid some computation, but it is not the bottleneck so the opposite could have been done instead (but wasn't).
  2. Yes, our calibration toolbox does it w.r.t. camera 0, but you are free to chose as long as your extrinsics are good.
ravijo commented 5 years ago

Okay. Just to confirm, CameraMatrix has two definitions as mentioned below:

  1. In xml file: CameraMatrix = CameraExtrinsics
  2. In datum.hpp file: CameraMatrix = CameraIntrinsics * CameraExtrinsics (where * denotes the matrix dot operator )

Hence for origin camera, i.e., cam0, the CameraMatrix in XML file should be an identity matrix of dimension 3x4. However, for 3-D triangulation from multiple single views it should be the following:

auto cameraIntrinsics = mCams[i]->getCameraIntrinsics();
auto cameraExtrinsics = mCams[i]->getCameraExtrinsics();
auto cameraMatrix = cvMatDot(cameraIntrinsics, cameraExtrinsics);
// fill datum
datum->cameraIntrinsics = cameraIntrinsics;
datum->cameraExtrinsics = cameraExtrinsics;
datum->cameraMatrix = cameraMatrix;

Please correct if it's wrong!

pritorius commented 5 years ago

auto cameraMatrix = cvMatDot(cameraIntrinsics, cameraExtrinsics); is it dot product? Or just matrix multiplication?

ravijo commented 5 years ago

@pritorius: I apologize for the confusion! I actually calculated ‘matrix dot‘ operation outside the code. In the above pseudo code ‘cvMatDot’ refers to ‘matrix dot‘ operation.

hadi1376tm commented 3 years ago

Hi, i need to use a different camera brand but doc is confusing, can you please give me a example if you have done it?

pritorius commented 3 years ago

Hi, i need to use a different camera brand but doc is confusing, can you please give me a example if you have done it?

Quick clarification - are you trying to do 3D openpose? Or 2D?

Regards,

hadi1376tm commented 3 years ago

Hi, i need to use a different camera brand but doc is confusing, can you please give me a example if you have done it?

Quick clarification - are you trying to do 3D openpose? Or 2D?

Regards,

hi, i need to do this for 3D Pose Reconstruction and Estimation.

hadi1376tm commented 3 years ago

update: so i found out where to paste my custom file, mentioned here ,but yet i don't know exact changes that i should make to set my camera parameters.

pritorius commented 3 years ago

II will put together a document and send it to you. It might take 1-2 days as the commits are mixed and not serial in time. Please let me know if that helps.

Regards,

On Thu, Aug 12, 2021 at 2:02 PM Hadi Tamimi @.***> wrote:

update: so i found out where to paste my custom file, mentioned here https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/04_cpp_api.md ,but yet i don't know exact changes that i should make to set my camera parameters.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/1060#issuecomment-897965076, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOIEHRM3SAOD6LH43ZYGRDT4QZE7ANCNFSM4GUDPK2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

hadi1376tm commented 3 years ago

II will put together a document and send it to you. It might take 1-2 days as the commits are mixed and not serial in time. Please let me know if that helps. Regards, … On Thu, Aug 12, 2021 at 2:02 PM Hadi Tamimi @.***> wrote: update: so i found out where to paste my custom file, mentioned here https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/04_cpp_api.md ,but yet i don't know exact changes that i should make to set my camera parameters. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1060 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOIEHRM3SAOD6LH43ZYGRDT4QZE7ANCNFSM4GUDPK2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

Thank you it will help a lot ,i will be so grateful.

you can send it to my email: hadi1376tm@gmail.com

hadi1376tm commented 3 years ago

II will put together a document and send it to you. It might take 1-2 days as the commits are mixed and not serial in time. Please let me know if that helps. Regards, … On Thu, Aug 12, 2021 at 2:02 PM Hadi Tamimi @.***> wrote: update: so i found out where to paste my custom file, mentioned here https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/04_cpp_api.md ,but yet i don't know exact changes that i should make to set my camera parameters. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1060 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOIEHRM3SAOD6LH43ZYGRDT4QZE7ANCNFSM4GUDPK2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

Hi sorry to bother, do you have any updates?

pritorius commented 3 years ago

Sorry for being late. I will send you tomorrow morning.sorry about that.

On Wed, Aug 18, 2021 at 10:45 PM Hadi Tamimi @.***> wrote:

II will put together a document and send it to you. It might take 1-2 days as the commits are mixed and not serial in time. Please let me know if that helps. Regards, … <#m1194322510709808680> On Thu, Aug 12, 2021 at 2:02 PM Hadi Tamimi @.***> wrote: update: so i found out where to paste my custom file, mentioned here https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/04_cpp_api.md ,but yet i don't know exact changes that i should make to set my camera parameters. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1060 (comment) https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/1060#issuecomment-897965076>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOIEHRM3SAOD6LH43ZYGRDT4QZE7ANCNFSM4GUDPK2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

Hi sorry to bother, do you have any updates?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/1060#issuecomment-901628008, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOIEHRM3B3MNTUVXPRIASLT5SLA7ANCNFSM4GUDPK2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

hadi1376tm commented 3 years ago

Sorry for being late. I will send you tomorrow morning.sorry about that. On Wed, Aug 18, 2021 at 10:45 PM Hadi Tamimi @.> wrote: … II will put together a document and send it to you. It might take 1-2 days as the commits are mixed and not serial in time. Please let me know if that helps. Regards, … <#m1194322510709808680> On Thu, Aug 12, 2021 at 2:02 PM Hadi Tamimi @.> wrote: update: so i found out where to paste my custom file, mentioned here https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/04_cpp_api.md ,but yet i don't know exact changes that i should make to set my camera parameters. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1060 (comment) <#1060 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOIEHRM3SAOD6LH43ZYGRDT4QZE7ANCNFSM4GUDPK2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email . Hi sorry to bother, do you have any updates? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1060 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOIEHRM3B3MNTUVXPRIASLT5SLA7ANCNFSM4GUDPK2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

Nothing to be sorry about, thank you very much for your kindness.