search

yohanshin / WHAM

MIT License
698 stars 75 forks source link

How to improve estimation of arm and hand angles in extreme scenarios? #58

Open mrEuler opened 9 months ago

mrEuler commented 9 months ago

As I already checked many repos and papers, that work with SMPL fitting, I didn't get good apropriate result in hand/arm/forearm tracking in negative extreme angles. Here are few screenshots (left 4d humans, middle WHAM w/o new flag, right WHAM with 'run_smplify' flag. Is there any way how to improve this or finetune the model? image image image image

yohanshin commented 9 months ago

Hi @mrEuler

First of all, it's really a cool visualization! What I can suggest you to improve the results --especially hand and arm-- is:

  1. Prepare whole-body 2D keypoints detection results (you can find a whole-body model from ViTPose) and save the results somewhere.
  2. Run WHAM demo code and save the results.
  3. Use WHAM's results to build SMPL+H model (which also articulates hand poses).
  4. Build your own TemporalSMPLify code that uses whole body keypoints --> This will fix hand and foot orientation.

However, I doubt this will not resolve all issues in arm poses because tennis players' arms move very fast and 2D keypoints detection often fails with blur (see the right arm of the 3rd photo).

mrEuler commented 9 months ago

@yohanshin Thank you for an advice. I will try this and let you know how it'll work. Actually, I want to use primaraly slow-motion frames, because yes, with 30-60 FPS shutterspeed, the motions are blurry.

mrEuler commented 9 months ago

@yohanshin btw here is the RGB video I am using to test, just FYI: https://drive.google.com/file/d/11I6hfR0BkOGnqS1wc0hnQAngtTZsQTEM/view?usp=sharing