YoungSeng / DiffuseStyleGesture

DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models (IJCAI 2023) | The DiffuseStyleGesture+ entry to the GENEA Challenge 2023 (ICMI 2023, Reproducibility Award)
MIT License
147 stars 21 forks source link

Question of motion representation #7

Closed cyk990422 closed 1 year ago

cyk990422 commented 1 year ago

Sorry, I still have a doubt, why not learn the Euler angles of the local rotation in the motion representation, is it because this representation is difficult to learn?

cyk990422 commented 1 year ago

Another problem is that I noticed that when you save bvh, in the code of converting quaternion to Euler angle, the prompt can only be converted to Euler angle in zyx and xzy order, whether it can not be converted to Euler in xyz order horn? I tried to write a conversion process myself (convert the Euler angles in the xyz order to quaternions and then back to the Euler angles in the same order), but it seems that the angle value of the Euler angles has changed? Thank you very much for your patient reply!

YoungSeng commented 1 year ago

Hey cyk,

I think the representations of motion are all right!

For examples, End2End used rotation matrix and Audio2Repr2Gesture used position or exponential maps, QuaterNet used quaternions...

Although these can switch to each other, however, rotation is easy to calculate rotational velocity and acceleration.

Although position, positional velocity, and positional acceleration or rotation and rotational velocity and acceleration are contained to each other, more modalities are better, see https://github.com/sebastianstarke/AI4Animation/issues/47#issuecomment-945189038

That is why Zeroeggs used complex motion representation.

All in all, I think motion representation is not key in gesture generation. You may take some experiments on them.

Wish it will help you : )

YoungSeng commented 1 year ago

The bvh processing pipeline and writing pipeline are the same as Zeroeggs, which is not familiar to me. You may check the source code in Zeroeggs.

cyk990422 commented 1 year ago

Ok, I see, thank you very much for your reply, it helped me a lot.