search

ChenFengYe / motion-latent-diffusion

[CVPR 2023] Executing your Commands via Motion Diffusion in Latent Space, a fast and high-quality motion diffusion model
https://chenxin.tech/mld/
MIT License
579 stars 51 forks source link

How to fit 3d joints of KIT dataset to smpl mesh #10

Open qiqiApink opened 1 year ago

qiqiApink commented 1 year ago

I have generated motions in the format of [nframes, njoints, xyz], especially njoints=21. And I have no idea about fitting them to smpl mesh. Can you give me some advice?

ChenFengYe commented 1 year ago

Hi qiqiApink, I think there is no official code for the KIT format fitting if you download KIT from this website.

We have provided the fitting script for HumanML3D and KIT. The script should support njoints=21. We do not check this part. If you find any error please let us know. https://github.com/ChenFengYe/motion-latent-diffusion/blob/17ba9e3ca881cee0c107c4ab02a56f8d751cd717/scripts/fit_motion.sh#L1

https://github.com/ChenFengYe/motion-latent-diffusion/blob/17ba9e3ca881cee0c107c4ab02a56f8d751cd717/fit.py#L170

https://github.com/ChenFengYe/motion-latent-diffusion/blob/17ba9e3ca881cee0c107c4ab02a56f8d751cd717/fit.py#L225

https://github.com/ChenFengYe/motion-latent-diffusion/blob/17ba9e3ca881cee0c107c4ab02a56f8d751cd717/mld/transforms/joints2rots/smplify.py#L92

If you want to write you own fitting code, we suggest you to match each joint like:

https://github.com/ChenFengYe/motion-latent-diffusion/blob/7db8623e61e2b6818e724a0a9a1fc999f5054327/mld/transforms/joints2rots/config.py#L44

I notice your issues in TEMOS. I also did not find the fitting code in their project. They use two formats, 3d joint and SMPL parameters, for two network models. For this part, you can wait for their answer.

p.s. There are two formats you can download. The kit format and the kit in humanml3d format (search KIT-ML Dataset)