search

PeizhuoLi / neural-blend-shapes

An end-to-end library for automatic character rigging, skinning, and blend shapes generation, as well as a visualization tool [SIGGRAPH 2021]
Other
637 stars 92 forks source link

Weird artifacts while running on custom mesh #13

Open codesavory opened 3 years ago

codesavory commented 3 years ago

I am trying to run this on custom meshes and I am getting this weird artifact. I am using this model from SketchFab - https://sketchfab.com/3d-models/trackerman-5b04147173fd41f19ccd4fe42f549a15 and also with a custom mesh. I have triangulated the mesh and calling is using normalize - python demo.py --pose_file=.\eval_constant\sequences\greeting.npy --obj_path=.\eval_constant\meshes\trackerman\trackerman.obj --animated_bvh=1 --obj_output=0 --normalize=1. Any possible reason for why could be happening and any suggestions/filters to avoid this, would be greatly appreciated. artifacts_error1 artifacts_error2

PeizhuoLi commented 3 years ago

Hi, the automatic normalization does not always work. In such a case you may need to manually scale and translate to spatially align the model to the reference SMPL standard model. For example, the trackerman in the attached file should work (I simply manually align it to the reference model) trackerman.obj.zip

fogside commented 3 years ago

Hi! I aligned my example mesh with these trackerman.obj file manually and triangulated my mesh at the end. But I still have these artifacts. Could you please take a look. I'm attaching my aligned mesh. girl_wolf3d_2.zip Screenshot from 2021-07-18 22-00-56

Here you can see my mesh aligned with the trackerman. I don't know how precise the alignment should be. But I guess it's not always possible to make it very precise as the character may vary in shape a lot. Screenshot from 2021-07-18 22-11-49

The command I run with my character:

python demo.py --pose_file=./eval_constant/sequences/greeting.npy --obj_path=girl_wolf3d_2.obj --animated_bvh=1
PeizhuoLi commented 3 years ago

Hi, this model contains many non-manifold edges while our MeshCNN model relies on "good" local connectivity. This can be one of the causes of this failure.

fogside commented 3 years ago

Hi, this model contains many non-manifold edges while our MeshCNN model relies on "good" local connectivity. This can be one of the causes of this failure.

Thank you for this information. I was trying to use another model instead, that I made from scratch with just meta-balls that I converted later to mesh. I think the geometry shouldn't be non-manifold there (Blender does not select any non-manifold pieces). I'm also curious, does this work only with humanoid characters with proportions that are very close to humanoid? Screenshot from 2021-08-01 22-23-34 Screenshot from 2021-08-01 22-22-11

In case you are interested in the model file, it's here. u.zip

PeizhuoLi commented 3 years ago

Hi, thanks for your question. Since our model is trained with SMPL model, it is expected to work with humanoid-like model. So yes, the input model should be similar to humanoid-like proportion. Though SMPL does cover some extreme case for humanoid, but it doesn't cover this case you showed.

If you simply want the automatically generated rigging without blend shapes, I would recommend RigNet to you: it is designed to work with all kinds of model.