YoungSeng / FreeTalker

Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness (ICASSP 2024)
https://youngseng.github.io/FreeTalker/
63 stars 10 forks source link

A question about generating SMPLX motion #2

Closed RobinWitch closed 8 months ago

RobinWitch commented 8 months ago

Thanks for your nice work ! While I have a question, we can't use python ../human_body_prior/tutorials/mdm_motion2smpl.py --input ./save/my_v3_0/model001000000/result_rec_1.0.npy --output ./save/my_v3_0/model001000000/result_rec_1.0_smplx.npz to generate smplx motion . As you mentioned that we should need to modify mdm_motion2smpl.py and the environment of human_body_prior ,how we do this and what change should we make ?

YoungSeng commented 8 months ago

Just run the code of this (maybe you need a new environment) and refer the mdm_motion2smpl.py to modify according code

RobinWitch commented 8 months ago

Thanks a lot! I forgot the code mdm_motion2smpl.py in the project. Now I work successfully! While I have a question , how can we get hands(fingers) in the smpl? Now we just set hands as zero. emm, how we can get hands?

YoungSeng commented 8 months ago

The results generated in the paper are fingers, such as those visualised using python, but not after conversion to smpl visualisation.

RobinWitch commented 8 months ago

Hello, perhaps my question was not clearly articulated, leading to a misunderstanding. I am aware that the paper indeed generated fingers in your paper, and they can also be visualized as sticks. However, I would like to know how to generate an SMPL-X model with fingers, as the currently generated SMPL-X models with fingers have their finger settings all set to zero.

Do you know how to generate it in smplx? Now we use vpose engine to fit pose in smplx but seems that it can't generate fingers?

Thanks you a lot!

YoungSeng commented 8 months ago

Yeah, I exactly know what you mean. But unfortunately, I only use the position to extract the motion feature and transfer them back to position, I also want to convert them to SMPLX to visualize them, such as in Blender, but it seems very difficult and I only use Vposer to convert them to SMPL to visualize but without fingers. I think it would be very helpful if someone could help convert the position to pose and trans in SMPLX format.

RobinWitch commented 8 months ago

Thank you a lot ,I will try to solve this problem!