The bvh file generated by style transfer cannot be bound to the human body model

[zanxiR]

Hi, first of all thank you very much for your contribution, this is an excellent work! I am an animator and have been studying this project for some time. Recently, I wanted to try to bind the generated bvh file bones with the human body model, and found that the human body model would be distorted to a large extent. But I use the bvh file provided by CMU, and it can be successfully bound with the mannequin without distortion. I am a little puzzled. After all, the bvh files generated by your network in Blender can be successfully visualized and run smoothly. Today, when I imported the bvh file generated by the network into Blender, I chose to shrink evenly, and the parameter was 0.01. Then I might find the problem. This is a bvh file made with a motion capture device. You can see that the position of his bone points is relatively correct.

This is the skeleton of the bvh file generated by the network. The Hips joints and Spine1 joints are marked by the blue border. They appear in positions that should not appear, and they remain in this state during the movement. This explains why there are distortions in these parts when I bind the model with this file.

This skeleton cannot be bound manually, because Hips is the root node and Spine1 is also the parent node of some joints. Due to the hierarchical limitation of the bvh file, they cannot be adjusted separately.

Now I don’t know why the network generates these two misaligned joints, so I want to raise the problem here to discuss with you. Thank you very much for reading this bunch of words!

[zanxiR]

@HalfSummer11 Looking forward to your response, thanks again. :)

[HalfSummer11]

Hi, thank you for your interest in our project and your detailed description of the problem! The misaligned joints are in deed problematic - we didn't realize the problem because we to visualize our BVH outputs using cylinder-shaped primitives as bones and the two joints don't correspond to any visible bones. Our network is trained with two main types of losses on the output skeleton. One is joint position loss that disregards joint rotation (e.g. for these two joints' rotations, as long as their compositions with children joint rotations are correct - so that , this loss will not penalize them), the other is joint rotation loss. Although it seems natural to only apply joint rotation loss on our output rotations, position loss is much more effective during training since it enforces a stronger constraint on end-effectors. Otherwise, the rotation error along the kinematic chain will accumulate and result in a bad motion. I think we didn't balance the two losses carefully enough, such that the position loss becomes predominant and the rotation loss is sacrificed, which resulted in those strange joints. I think a different set of hyperparameters, even a different training curriculum where rotation loss is given increasing weight, may help solve the problem.