search

OpenGVLab / UniHCP

Official PyTorch implementation of UniHCP
MIT License
151 stars 9 forks source link

Question about instance-level prediction #8

Open JosonChan1998 opened 1 year ago

JosonChan1998 commented 1 year ago

Thanks for your nice work.

I have a question about the method: How should the query be defined for multi-person human pose estimation and human parsing? Does it need an extra detector to crop the human instance from the original image?

orashi commented 1 year ago

For UniHCP as described in the paper and implemented in the repo, a Top-down approach would be needed: First detects humans via Pedestrian detection queries, cropping out each identified individual by the bounding box, and then do pose estimation separately. There might ways to incorporate bottom-up to UniHCP during training/inferencing but they are not discussed in this version.

JosonChan1998 commented 1 year ago

Thanks for your reply.

JosonChan1998 commented 1 year ago

Sorry for reopening the issue. Recently, I have a idea for creating a unified instance-level human-centric perception model, including human parsing, pose estimation and part detection. In other words, it is an end-to-end paradigm without the extra person detector and support multi-task dataset training. I would like to sincerely ask you what you think about this idea? Looking forward to your reply!

orashi commented 1 year ago

That's a cool idea and something we thought of the same but were unable to easily achieve under the current UniHCP structure, our thoughts were: Fine-grained instance level semantics identification would make it hard to let semantics self-contained in queries, e.g., either we have a lot of semantic-self-contained queries like what we do for detection (but more kind of things to detect, with separate set of queries), which is computation-inefficient; or a mask2former style bipartisan matching for this purpose, which would require classification heads on top of decoder, and that classification head is dataset-dependent (class definition varies across different datasets) thus intruducing unsharable parameters.

JosonChan1998 commented 1 year ago

Thanks for your reply! As you said, the most difficult thing is how to unify instance-aware and part-aware queries. I am still exploring it now. Thanks again.