search

shubham-goel / 4D-Humans

4DHumans: Reconstructing and Tracking Humans with Transformers
https://shubham-goel.github.io/4dhumans/
MIT License
1.24k stars 118 forks source link

How to improve estimation of arm and hand angles in extreme scenarios? #93

Closed mrEuler closed 8 months ago

mrEuler commented 9 months ago

As I already checked many repos and papers, that work with SMPL fitting, I didn't get good apropriate result in hand/arm/forearm tracking in negative extreme angles. Here are few screenshots (left -- 4d humans, middle and right WHAM w/o and w postprocessing). Is there any way how to improve this or finetune the model? Overall 4d humans have generally best results IMHO, except these issues with extreme angles during f.e. tennis serve: Screenshot 2024-02-08 144628 Screenshot 2024-02-08 144502 Screenshot 2024-02-08 144516 Screenshot 2024-02-08 144541

geopavlakos commented 9 months ago

If you want to potentially refine the output from 4D Humans, you could use our slahmr work, which adds some overhead in the runtime but it typically leads to more precise results.

Could you also share the original RGB video you are using for this inference?

mrEuler commented 9 months ago

@geopavlakos thank you! Btw I just found slahmr 30 mins ago. So will try and let you know how it'll work! Here is the RGB video: https://drive.google.com/file/d/11I6hfR0BkOGnqS1wc0hnQAngtTZsQTEM/view?usp=sharing

mrEuler commented 9 months ago

@geopavlakos I also would say not to refine result itself, but to refine model with additional dataset, that can be captured and transformed to SMPL type.

geopavlakos commented 9 months ago

In terms of finetuning the model, that is definitely possible, if you have this type of data (i.e., images and corresponding 3D ground truth in SMPL format). We have not tested finetuning HMR2.0 with data from a specific activity, but I expect it would help reconstructing poses from that particular activity more accurately.