search

huawei-noah / noah-research

Noah Research
857 stars 160 forks source link

[CLIFF] upper body reconstruction #151

Open sangdv opened 1 year ago

sangdv commented 1 year ago

We have a research project related to 3D reconstruction of upper body part to estimate the intubation difficulty of patients. However, when we applied your model on images that contain only upper body part, the model yielded very bad results since it tried to reconstruct the whole human body. Could you please advise us how to deal with that? Here is an example of an input image with only upper human body part: https://upload.wikimedia.org/wikipedia/commons/f/f8/Upper_body_front.png

zhihaolee commented 1 year ago

I test the image, and the result looks fine to me. Sorry that it is not convenient for me to upload the result image, could you upload your result image, then we can see what the problem is.

sangdv commented 1 year ago

Hi, here are my results. I also attach some test images so that you can check on your side. Thanks. https://drive.google.com/drive/folders/1_Dt7nGL0-lBzG59z82-x1LkcOfjmm95U?usp=sharing

zhihaolee commented 1 year ago

These are head images, not upper body images. I guess what you want is 3D head orientation estimation.

sangdv commented 1 year ago

No, we want to reconstruct the 3D upper body including the head, neck and maybe shoulders as well. We then detect some 2D keypoints like the eyes, chin, thyroid notch, sternal notch... and lift them to the 3D keypoints using reconstructed mesh. The purpose is to calculate some distances between them as clinical features to estimate difficult intubation.

sangdv commented 1 year ago

Here are some other examples: https://drive.google.com/drive/folders/14HFcTCRSFMLeuMXYvTJ4Hds6HDzhccbD Can you let me know how to fit the 3D shape better? The 3D reconstructed shape seems to be quite small compared to the original one.

zhihaolee commented 1 year ago

Please refer to this repo: https://github.com/akashsengupta1997/STRAPS-3DHumanShapePose.

sangdv commented 1 year ago

Thank you for your suggestion. I tried their improved method at https://github.com/akashsengupta1997/HierarchicalProbabilistic3DHuman but the results are still the same as you can see here: https://drive.google.com/drive/folders/1_Dt7nGL0-lBzG59z82-x1LkcOfjmm95U?usp=sharing

zhihaolee commented 1 year ago

I don't have the right to access the google drive data. Maybe you can try https://github.com/muelea/shapy, and see how it goes.