search

yohanshin / WHAM

MIT License
719 stars 78 forks source link

Body out of frame #79

Open ReetinavDas opened 7 months ago

ReetinavDas commented 7 months ago

Hello @yohanshin. Great work on this project! Me and my team were thinking of using this for a research project of ours, specifically patients and diagnosing them. However, in a lot of shots, parts of their bodies are complete out of frame, (typically the lower half of their body). I was wondering if you had any tips or ideas on how to get around this issue? I would also love insight into how the model deals with the situations where a person's body is out of frame.

yohanshin commented 7 months ago

Hi @ReetinavDas ,

Thank you for your interest in our work! Yes, one of the major limitations of the current version is that it assumes the full-body is visible (although allowing small occlusions) in the image frame.

There are two simple ways to resolve this.

  1. Train stage 1 with better occlusion modeling. This may require some implementation + experiments, but I observed that by adding "occlusion" modeling to stage 1 improves the robustness of the severe occlusion scenarios.

  2. First run WHAM and perform optimization as a post-processing. During the optimization, you can reset WHAM's prediction on those joints which are occluded. For example, if the entire lower body is occluded, just take WHAM's upper joints estimation and optimize the lower body using motion priors.

Hope this helps your project!

AlecDusheck commented 2 weeks ago

I’m also interested in this. Is there anything as accurate as WHAM but functions better in environments with occlusions?