search

tijiang13 / InstantAvatar

InstantAvatar: Learning Avatars from Monocular Video in 60 Seconds (CVPR 2023)
365 stars 31 forks source link

What is the coordinate system for rotation (global_orient)? #67

Closed muyangye closed 5 months ago

muyangye commented 6 months ago

Hi folks,

Thanks for the nice project! It is very usable and extensible. I am currently building a live streaming pipeline and which will take Unity's recorded translation and rotation as the input to InstantAvatar. I have verified the axis and positions are equivalent in Unity and InstantAvatar. However, Unity's rotation ranges from 0 to 2pi whereas InstantAvatar's ranges from -pi to pi. I have tried to map Unity's rotation to InstantAvatar's by assuming negative starts at where positive starts (e.g. 300 degrees in Unity = -60 degrees in InstantAvatar) and by assuming negative starts at where positive ends (e.g. 300 degrees in Unity = -120 degrees in InstantAvatar), but both can only inference/synthesize an image from the back and front of the synthesized person (when the viewer moves to the side, synthesized image is empty).

I have also experimented whether Unity and InstantAvatar's axis are inverted (clockwise vs. counter-clockwise), meaning if Unity's degree x > 180, set it to 360 - x, if x < 180, set it to -x but found no luck. I read through the paper again and found no documentation on the coordinate system used in the dataset. Do you have any knowledge on this or can you point me to the documentation of coordinate system in your dataset (i.e. global_orient in batch in animate.py)? Thank you very much for your help!

Best, Muyang

tijiang13 commented 6 months ago

Hi Muyang,

  1. the global_orient is the relative rotation between the rest-pose SMPL mesh (which is in SMPL's coordinate system) and the camera's coordinate system. So you will need to consider the orientation of the rest-pose SMPL mesh as well.
  2. the global_orient is represented with axis-angle format, which can be represented as "θv" where θ is the angle of rotation and v the axis around which the rotation occurs.

You can give this visualization script a look. It visualises the SMPL meshes in both camera coordinate system and world coordinate system (when it starts you are in the camera view but you can switch to 3D view by dragging your camera using the mouse). Since the animated avatars and their corresponding SMPL meshes are aligned, this might be helpful reference if you want to import them into Unity.

Best, Tianjian

muyangye commented 5 months ago

Thanks for the detailed response. I have run the visualizer script and found out that since it's a NeRF-like method, all rays must hit the SMPL model before rendering. So after syncing the SMPL model's position with the camera + offset and setting camera's c2w matrix everything works. Closing this issue now.