search

benjiebob / SMALify

This repository contains an implementation for performing 3D animal (quadruped) reconstruction from a monocular image or video. The system adapts the pose (limb positions) and shape (animal type/height/weight) parameters for the SMAL deformable quadruped model, as well as camera parameters until the projected SMAL model aligns with 2D keypoints and silhouette segmentations extracted from the input frame(s).
106 stars 18 forks source link

suggestion for documentation/guidance for another animal species #11

Open monajalal opened 3 years ago

monajalal commented 3 years ago

Would it be possible to add instruction or guidance on how to reproduce this job for another animal species?

benjiebob commented 3 years ago

Thanks for the feedback. In theory you should just need to change the SHAPE_FAMILY. Should have docs updated in next couple of days...

zshyang commented 10 months ago

hi, which family should bear belong to?

OllieBoyne commented 10 months ago

Hi, the 5 animal classes, defined by the SMAL model (https://smal.is.tue.mpg.de/toys.html), are cats, dogs, cows, horses and hippos. I'd think dogs have the most similar shape, but you can look through the toys in the link above and see if there is one you think is particularly close.

zshyang commented 10 months ago

Hi, thanks for the reply.

  1. It appears that the "dog" category only features thin animals. I also tried "cats" and "hippos" but not work. MicrosoftTeams-image
  2. In your paper, you showed results related to bears. I've been unable to generate reasonable results for bears and hypothesize that this may be due to the model being SMAL rather than SMALR. Could you please confirm if my understanding is correct? Thank you.
  3. If my assumption about the use of SMAL rather than SMALR is correct, would it be possible to implement SMALR in either this repository or a new one? Thank you.
benjiebob commented 10 months ago

We do use the SMAL model in the Creatures Great and SMAL paper, and initialize bears from the dog category.

In our paper, we mention the refinement step from SMALR and that we don't use it.

This work was recently extended 12 with a refinement step that optimizes over model vertex positions. This can be considered independent to the initial SMAL model fit and would be trivial to add to our method.

In your examples, it looks like either your shape prior weight is too high or your silhouette loss weight is too low. If you play with these weights, you should be able to get a 3D fit that matches the outline of the bear. You can see my attempt in our video.

zshyang commented 10 months ago

I see. Thank you very much for your reply and suggestions. I'll try your advice about adjusting the weights and will create a new closed issue in the future if I achieve success.