EricGuo5513 / momask-codes

Official implementation of "MoMask: Generative Masked Modeling of 3D Human Motions (CVPR2024)"
https://ericguo5513.github.io/momask/
MIT License
859 stars 73 forks source link

questions about the joints2bvh converter #76

Closed Italian-PAO closed 2 months ago

Italian-PAO commented 2 months ago

Hello, thank you very much for momask's contribution to this field!

I am using the action results (3d pose) generated by interGen(A model for generating two-person actions from text)reasoning to redirect some fbx models. My approach is to use the conversion tool provided in momask (https://github.com/EricGuo5513/momask-codes/blob/main/visualization/joints2bvh.py) to convert the 3d pose into a bvh format action, and then use bvh for redirection.

However, I found some problems in the process of converting bvh: When one person in a two-person action is facing forward, the converted BVH action is very correct; but when one person is facing away/sideways, the converted BVH action is very strange, but the visualization of 3D pose is normal (see the two videos in the attachment).

I suspect that there is a problem with the IK solution during the conversion process, but I am not an expert in this area, so I would like to ask you: What might be the problem in this case? If it is really a problem with the IK solution, how to solve this problem?

Maybe this issue is not appropriate to raise under your repo, but I still want to ask for your help, thank you very much!

Looking forward to your reply

大佬你好,非常感谢momask为这个领域做出的贡献!

我正在用interGen(一个文本生成双人动作的模型)推理生成的动作结果(3d pose)来重定向一些fbx模型,我的做法是使用momask中提供的转换工具 (https://github.com/EricGuo5513/momask-codes/blob/main/visualization/joints2bvh.py) 将3d pose转换成bvh格式的动作,再用bvh来做重定向。

但是在转换bvh的过程中我发现了一些问题: 当双人动作中某一个人的朝向是正对着的时候,转换出的bvh动作是非常正确的;但当某个人是背对/侧对着的时候,转换出的bvh动作就非常奇怪,但3d pose的可视化却是正常的(可以查看附件的两个视频)。

我怀疑是转换的过程中IK解算出现了问题,但我并不是这方面的专家,所以想要请教您一下: 这种情况可能是什么问题?如果真的是IK解算的问题,该如何解决这个问题?

可能这个issue在您的repo下面提并不合适,但还是想请求您的帮助,非常感谢!

期待您的回复

https://github.com/user-attachments/assets/155e7ff3-62ea-4be0-9cf3-1d0e62c65387

https://github.com/user-attachments/assets/6576b3f6-111d-4f88-a78e-ae810fb1567a

Murrol commented 2 months ago

You are right. It's hard to get a very stable and robust IK solver. You may try to tune the loss weights or regularization term to fix the specific case. https://github.com/EricGuo5513/momask-codes/blob/7ac5c2784dee403461558a722f0fdfdf36a3a849/visualization/InverseKinematics.py#L526