search

ai4r / Gesture-Generation-from-Trimodal-Context

Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity (SIGGRAPH Asia 2020)
Other
245 stars 35 forks source link

About dataset #27

Closed Woolseyyy closed 2 years ago

Woolseyyy commented 2 years ago

Appreciate for your work! I am trying to figure out the content of the datasets and how to generate it. I am confused with the meaning of "pose_seq" and "vec_seq" extracted from the ted_extend dataset. It seems that "pose_seq" means the 3d joint positions of each frame and "vec_seq" means the vectors of bones. However, the lengthes of each vector of "vec_seq" are various while the paper describes that "all directional vectors were normalized to the unit length".

What's the meaning of "pose_seq" and "vec_seq"? How to generate these two variables from 3d joints positions based on videopose3d output?

Woolseyyy commented 2 years ago

I found that it is implemented when generating chache at utils/data_utils.py. Why do you normalize directional vectors again by "dir_vec - mean_dir_vec"?

youngwoo-yoon commented 2 years ago

Hello, I had two normalizations for dir_vec. The first one is the unit length normalization using the normalize function in sklearn library. https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context/blob/16dacc92fac3ac02957773c5a448ff6b5b202619/scripts/utils/data_utils.py#L109 The second one is "dir_vec - mean_dir_vec" which makes training easier (because the model only needs to predict the differences).

Woolseyyy commented 2 years ago

Thank you!