About Dataset Pre-Processing

[zwithz]

Hi,

Thanks for making this awesome project open source. I encountered some questions about training this model on another dataset.

1. Could you share more details on how you pre-processed the LAFAN1 (*.bvh) into the deeppose_lafan_v1_fps? I found the offset and localOffset in the data_settings file are not the same as the offset in the original BVH data.
2. Does the data in .csv files consist of global positions and global quaternions?

Thanks~

[boreshkinai]

@zwithz Thanks a lot for you positive feedback on our work and for your questoins. Here are some answers:

• "Offset" should be ignored, what really matters is the "LocalOffset" which defines the offset of the joint in the space of its parent
• Positions are global
• Rotation are local, except for the root joint (Hips) which is Global. We recover global rotation during training using Forward Kinematics
• Regarding how we imported the data, I think we imported BVH in Blender then exported to FBX which where then imported to Unity. Maybe the bind pose (T-Pose) is different which might explain the difference for offsets.

I will dig for more info about the conversion process and will try to post it soon. Please let me know if you have additional questions.

[bigpo]

@boreshkinai could you please give more detiles about how to play with the generated fbx file? I used this one and it's preety weired

[boreshkinai]

@boreshkinai could you please give more detiles about how to play with the generated fbx file? I used this one and it's preety weired

We have a different skeleton, I'm working on providing more detail about it

[yuyujunjun]

Hi, I have questions about pre-processing of the LAFAN1 (*.bvh) , too.

1. The unit of positions (offset of bvh) in Lafan1 seems to be centimeter, but the csv file in deeppose_lafan_v1_fps have a different unit. How do you normalize them?
2. Lafan1 have multiple subjects and each subject has slightly different bone length, but the dataset_settings.json only has a bone length setting. Do you average their bone length? Btw, I've learned so much from your projects. Thank you very much!

[boreshkinai]

Hello everyone, I just added the most up to date puppet that we tested against our version of the dataset. https://github.com/boreshkinai/delta-interpolator/tree/master/tools/puppet Please check it out and flag any issues you might have, plus additional questions I also included the blender and unity export/import settings

[AntonValk]

Hi @yuyujunjun,

Hopefully the additional details Boris provided should be good enough.

Just to add some context since I worked decent bit on the data preprocessing for this project, we did not use Ubisoft's skeletons for this project. Rather, we projected the animation data onto our own custom skeleton for all the animations. Naturally this creates small distortions but it allows us to more easily integrate our model into Unity's engine.

Another reason why our the .bvh, .fbx and .csv files are not identical is the different handedness of the coordinate systems that we use (Unity vs. Blender). For more details see supplementary of the paper. We also have a flat scaling by 100 so we go from centimeters to meters unit scale (otherwise the skeletons inside the Unity engine would be interpreted as being 190m rather than 190cm tall). The things I mentioned can been in this Blender export setting screenshot @boreshkinai posted: https://github.com/boreshkinai/delta-interpolator/blob/master/tools/puppet/blender_export_settings.png (scaled down to 0.01) and axis flip to account for coordinate system change.

Hopefully this clears up your questions a bit :)

[yuyujunjun]

Thank you very much!