search

JeremyCJM / DiffSHEG

[CVPR'24] DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
https://jeremycjm.github.io/proj/DiffSHEG/
BSD 3-Clause "New" or "Revised" License
112 stars 9 forks source link

Convert SMPLX Expressions to ARKit Blendshapes for Visualization in Unreal Engine #18

Closed SlimeVRX closed 2 months ago

SlimeVRX commented 2 months ago

Hi!

The process of visualizing SMPLX in Unreal Engine

https://github.com/JeremyCJM/DiffSHEG/assets/99313947/aa1809ef-f818-4eaa-adb6-d41e3c401f99

I need face data in a format similar to ARKit, but I am unsure how to convert face data from SMPLX to ARKit.

Is there a possible way to convert SMPLX expressions to ARKit blendshapes?

Thank you!

JeremyCJM commented 2 months ago

Hi SlimeVRX, thanks for your question. We only did Metahuman visualization in Unreal Engine for results on BEAT dataset. Regarding visualizing SMPLX in Unreal Engine, we have not tried yet.

SlimeVRX commented 2 months ago

Hi,

Thank you for your response. I appreciate your clarification regarding the Metahuman visualization in Unreal Engine for the BEAT dataset. It's good to know the scope of your current work.

JeremyCJM commented 2 months ago

Thanks for your understanding! I heard that the Blendshape in SMPLX might not be the same as that of ARKit. I am not sure, but you can keep an eye on that:)