search

DeepMotionEditing / deep-motion-editing

An end-to-end library for editing and rendering motion of 3D characters with deep learning [SIGGRAPH 2020]
BSD 2-Clause "Simplified" License
1.58k stars 256 forks source link

Do I need to train new models for new characters in retargeting? #43

Open DetectiveLYJ opened 4 years ago

DetectiveLYJ commented 4 years ago

Hi, all

I read your papers carefully, you mentioned that "since the domains have different dimensions, the two networks (A → B, A → B) cannot share weights, so they had to be trained separately." in the paper.

Does this mean I need to train a new model if I want to use a character with a different number of bones from the training set?

Thanks!

PeizhuoLi commented 4 years ago

Yes, you'll need to train new models for new character.

DetectiveLYJ commented 4 years ago

@PeizhuoLi Well, I see that "We use group A to train the network on 20 characters and test it on 4 (unseen) ones." in the paper. I have the following questions:

  1. If I use the same number of bones for all the characters in Group A, and the number of bones in group B is all the same. But the number of bones in A and B is different. Is it possible to use characters not seen in the training set for testing? (the number of bones is consistent with the training set, but the body proportions are not consistent)

  2. While I'm training, can each character in Group B just input one stationary pose? And can each character in Group A just input one motion(this motion is for testing later)?

  3. When groups A and B have the same number of bones for each character, can we do retarget on characters(have the same number of bones with A and B) that are not in the training sets?

Looking forward to your reply!

PeizhuoLi commented 4 years ago

  1. It's possible to test on unseen characters, like our experiments. But our experiments also suggests even with same topology, when the rest poses are too different it could fail (see the money and dog example in our paper / video).

  2. No. You'll need motion for both characters to train the network.

  3. I think the answer is almost the same as (1). If the gap between test set and training set is not very large, it should work.