facebookresearch / audio2photoreal

Code and dataset for photorealistic Codec Avatars driven from audio
Other
2.7k stars 254 forks source link

How can I manually rotate an avatar's head? #46

Closed rosscutler closed 7 months ago

rosscutler commented 8 months ago

I would like to create a scenario with 3 avatars talking with each other. The avatars would all be voice driven, but I'd like the avatars that are not speaking to be looking at the one that is speaking. Ideally there would be a way to specify a head azimuth and elevation offset for each avatar, which can be programmatically controlled. Is this possible?

evonneng commented 8 months ago

Thanks for your interest in our work! Hmm it might be a bit difficult to manually rotate the neck joint and have the rest of the texture look correct (e.g. gaze) But in theory it is totally possible to rotate just the head. I need to ask around to see what the skeletal parameterization is to isolate which joint it is in the 104-d vector, but ill reply once I find it.

evonneng commented 8 months ago

From our resident skeleton expert Javier:

The neck joints are the following: neck twist: 12 (azimuth) neck lean: 13 neck bend: 14 (neck elevation) head rz: 15 (head elevation)

That said, turning only the neck to make the avatars look each other is going to make it look pretty weird. It's pretty uncomfortable to turn your neck a lot, so people typically rotate their torso and/or distribute that rotation across the whole kinematic change from pelvis to neck.

Hope this helps!