Remove virutal joints from retargeted skeleton

[rainer7]

Anybody that knows how to remove the virtual joints of the retargeted skeleton without corrupting the rotations of the remaining joints?

Since the Neural Network only works with skeletons, that can be represented as homeomorphic graphs, some skeletons need to be extended by virutal joints, before the Neural Network can process them. In my case, the original source skeleton (see green skeleton) is extended by 6 virtual joints (see blue skeleton).

The virtual joints still remain in the retargeted bvh file, but I need it without the virutal joints. My approach of removing them and adjusting the rotations failed, as you can see here (the pose of the blue skeleton with virtual joints is correct - but after removing the virutal joints the rotations of the skeleton are off (see green skeleton)).

The conversion of the t-pose itself is straight forward. But applying the rotations of the virtual joints onto their children does not seem to work for me.

My approach is the following:

1. convert euler angles of bvh-joint to quaternion
2. the rotation of the virtual joint's parent stays the same
3. the rotation of the virtual joint is "skipped" (removed joint from t-pose)
4. the virtual joint's child is multiplied with the virtual joint itself: *q_child = q_virtual q_child**

Since my approach didn't work, I tried all kinds of "combinations" - multiplying with the inverse and so on- nothing seems to work. Any ideas? Thank youu

[PeizhuoLi]

Hi, all the virtual joints are introduced because of the edge representation of animation, which is not compatible with bvh file's joint representation. To solve this problem completely from the cause, you'll need to change the code to joint representation and re-train everything.

However, I guess there is a trick that can bypass the whole re-training stuff. Virtual joints are introduced only when a joint has more than one child (let's call that joint p):

1. Average the rotation of p's child virtual joints, denoted as q_avg.
2. Replace p's rotation by q_avg. (p's original rotation should be zero)
3. Remove p's child virtual joints, connect the child of each virtual joint directly to p. There's no need to touch their rotation.

Since all the q_virtual should be similar, the average step should be easy. As I imagine simply arithmetic average should be fine. Or even randomly pick one instead average could work.

[rainer7]

Thanks Peizhou for the quick response!!

I tried your approach with averaging the virtual joints and it works perfectly for the input.bvh with the virtual joints before it is processed and retargeted by the neural network, as you can see here:

However, when I try the same for the retargeted result.bvh, the averaging approach does not work. I think that is due to the fact, that the NN modifies the virtual joints individually so that they are not similar anymore? Also the joint p's original rotation (as you mentioned) is only zero in the input.bvh. In the result.bvh it has actual values.

Is there anything else I could try without changing the code to joint representation? Thank you!

[PeizhuoLi]

I thought in the result the rotation for the virtual joints should be similar to input: since it's trained on these data. Could you check if p's children's rotations are similar? If it's similar you may still you the approach, but remember also add p's rotation.

If it's not true (it could happen since the network might learn to use the "flexibility" it for further adjustment), another alternative can be use IK. You can use one of the approaches above as the initialization for the joint rotation. Then use the joint location of output (the skeleton with virtual joints) as constrain (similar to this one) to optimize the joint rotations.

[rainer7]

Hi Peizhou, thanks for the quick reply! I actually just noticed why your approach didn't work with the result BVH: my BVH parser only works with the rotation order XYZ. The rotation channels of the input BVH are XYZ, whereas the result BVH is ZYX. A stupid mistake, my bad! I adjusted the parser and now your first approach works well - it's still not the exact same as with virtual joints, but it is really close and it will do! Maybe adding p's rotation to q_avg could help too, I will try that! Thank you and sorry again for my silly mistake - I hope I didn't waste your time!