search

tensorlayer / HyperPose

Library for Fast and Flexible Human Pose Estimation
https://hyperpose.readthedocs.io
1.24k stars 275 forks source link

Determining self occluded joints #328

Open korejan opened 3 years ago

korejan commented 3 years ago

Hi, is there a reliable method to determine self occluded joints? I'm converting 2D joints to 3D via an RGB-D camera, there are cases when pose inference (correctly) predicts a joint that is self-occluded but of-course there are no depth values for an the occluded joint so I'm going to apply some hueristics to estimate the depth (such as extrapolating on previous frames).

In order to do this I need a reliable method to know when the inferred joint is an (self) occluded one. I did some tests to check the confidence scores that hyperpose outputs and it seems like I can use this because the scores are typically < 0.6 but there are cases when the score is below 0.6, joint is correctly predicated but not occluded. I guess I will need to use some extra rules such as the orientation of the person in relation to the camera. Anyway I just wanted to get your thoughts on this, maybe there's a better way I can know this from hyperpose?

stubbb commented 3 years ago

Hello, The confidence scores will not tell you the information you want to know, I have found background noise produce >0.6 confidence score occasionally. You need to go one step back. Hyperpose devs are currently working on the new method of feature aggregation: PifPaf. It uses a Part Intensity Field (PIF) to localize body parts and a Part Association Field (PAF) to associate body parts with each other to form full human poses. This would likely be where you can get an accurate idea of feature occlusion. To learn more, I suggest looking up OpenPifPaf project on github.

korejan commented 3 years ago

Hyperpose devs are currently working on the new method of feature aggregation: PifPaf...

@stubbb Yes I've seen the PR but this version implements inference for Python only atm, I'm using C++ API.

stubbb commented 3 years ago

@korejan As far as I have figured, feature inference and pose estimation are logically separated in the C++ API. In fact you can choose the kind of pose estimation you want to employ - currently paf or pose proposal. I never had a reason to see what is passed between the infer and paf but I am guessing checking that out would likely help your development.