Pose / Gaze direction

[shivenmian]

The keypoints that Openpose generates are 2D. Is there a way to get direction vectors through OpenPose itself (for cases like getting face / gaze direction)?

Or is the recommended way for this something like setting up a stereo rig and do a 3D visualisation of the keypoints and then get the vectors from there?

[tsimk]

The keypoints detected on the pupils are currently very unreliable, I would not try to use them to estimate gaze direction. As a coarse approximation, fitting a 3D face model to the 2D keypoints to get rough head orientation usually works (even from a single camera view).

[realtimecrime]

@tsimk Regarding the head pose estimation, do you think this approach (mapping the 2d keypoints to a 3d model) would work even for extreme angles where only the profile is shown and less keypoints are detected?

[tsimk]

I don't have an evaluation, but it will likely be noisier. Because the jaw keypoints are on the silhouette rather than the jaw, I'd try something like Section 6 in Chen Cao's DDE paper. Using the COCO keypoints on the ears should improve stability.

[costef]

Hi everybody. I'm also interested about getting the face/gaze direction. I've a test application based on the example OpenPose example "1_extract_from_image.cpp". I can successfully obtain the keypoints as in the example

const auto poseKeypoints = poseExtractorCaffe.getPoseKeypoints();

Now I would like to get the face/gaze direction from the poseKeypoints. Can anybody help me ? A piece of code in C/C++ would be very very helpful and appreciated. Thanks in advance

[toinsson]

for reference, I created a related issue (#428) which touch on the same question (albeit a bit more generic): getting 3d models from keypoints. In my case, I'll be more interested in pose, but hands or face should be the same.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.