search

benjiebob / SMALify

This repository contains an implementation for performing 3D animal (quadruped) reconstruction from a monocular image or video. The system adapts the pose (limb positions) and shape (animal type/height/weight) parameters for the SMAL deformable quadruped model, as well as camera parameters until the projected SMAL model aligns with 2D keypoints and silhouette segmentations extracted from the input frame(s).
MIT License
120 stars 18 forks source link

How to SMALify another animal's 3D model? #13

Closed monajalal closed 4 years ago

monajalal commented 4 years ago

Hi Ben,

Thanks a lot for your great research work. I had a couple of questions that would be great if I could get some instructions on them.

  1. I have this 3D squirrel model that I am interested in inputting to SMALify. How can I do that? https://www.turbosquid.com/3d-models/squirrel-3d-model-1372687
  2. I know in our discussion you wanted the animal to be rigged, but what if the animal is not fully rigged and only partially rigged? Many of the available models in 3D model Websites are generally just partially animable/rigged (e.g. they can perform only 3-10 actions).
  3. Why would I need 10-15 different 3D models of a squirrel if I am interested in modeling and SMALifying a squirrel?
  4. The model of squirrel I shared is quite expensive, high-poly, and detailed. What is the level-of-detail (LOD) and number of poly required for a model to be SMALifiable?
  5. If I am trying to SMALify an animal, does it matter if the 3D model has skin/fur/texture or not? I am noticing some 3D models on 3D websites only have mesh and some have such details as in 1 like fur and texture.
benjiebob commented 4 years ago

It's probably worth saying that the functionality you're requesting is a bit beyond the scope/intent of this repository. The process for building a deformable 3D template model is described for animals here and for humans here.

However, there is a cheap and cheerful trick you can use:

The basic idea of SMALify is to fit a mesh, parameterized in shape (blend shapes) and pose (joint angles), to an image / video. If you have only one 3D squirrel model, you cannot learn to represent different squirrel shapes -- so you can't model fatter/thinner, taller/shorter squirrels or ones with different body proportions. However, maybe this is acceptable to you.

In order to adapt the limbs to match your squirrel images, you will need the important parts of the squirrel (probably at least legs, tail) to be rigged with a kinematic tree (skeleton). It is not important whether they have animations.

What you can do (rather than building a new model from scratch), is to run an optimizer to adapt the SMAL model to your 3D squirrel using 3D-3D losses. Once completed, you will have a set of shape parameters (and optionally vertex offsets) for the SMAL model so that it looks very similar to your downloaded 3D model. You can just then fix the betas to this, and use the SMALify code as normal.

If there is interest, we could tidy up and release some code we've used for this in the past. My guess is the range of animals represented by SMAL is probably enough for most users however.

The above should answer Q1-Q3.

Q4: Level of poly-detail is up to you and doesn't matter to SMALify.

Q5: Skin is important for the silhouette loss - but you could in theory turn this off. Fur/Texture are not important.

benjiebob commented 4 years ago

Going to close for now.