search

UPC-ViRVIG / MMVR

Repository for the SCA 2022 paper "Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer-Grade VR Devices"
https://upc-virvig.github.io/MMVR/
Other
61 stars 7 forks source link

Few questions (only works with SteamVR, height calibration, use RGBD camera to improve the results) #9

Closed sh-taheri closed 1 year ago

sh-taheri commented 1 year ago

Thanks a lot for the great work! I tried out your project with quest 2, and I had some questions I am hoping to get answers for. (btw I am not familiar with Unity or any other game engines)

1) When I play the unity editor, and I connect the quest 2 (link or air) to the PC, I am not able to see the Game unless I open the "Steam VR" application. Do you have any thoughts on that?

2) Eventhough I press the B button (which is supposed to calibrate the height) sometimes I notice the avatar is bigger than me? Is there a minimum height setting?

I know that the body orientation prediction module is independent of the position (because it is using the velocity etc.), therefore does not depend on the size of the user, but pose information are also used in constructing the query vector, is the "Current Pose" normalized according to the height of the user? (because if not, the motion matching algorithm depends on the size of the user, and I think the motion matching database is created by one user, right?) Can you please clarify that for me?

3) Do you have any suggestions on how to incorporate an RGBD camera to further improve the accuracy of the method? Do you think the "Body direction prediction" and "current pose" blocks can be replaced with the pose estimation output of an RGBD camera? (Example: https://www.stereolabs.com/docs/unity/body-tracking/) image

Thanks a lot!

JLPM22 commented 1 year ago

Hello! Thank you very much for your interest in our work! :)

For the first issue, it looks like SteamVR is set as your first VR environment, I think you may change it in the Oculus Software following the instructions in this image I found: image

Do other applications run in Unity work in your Quest by link or air? Just to see where the problem may come from.

For 2. I will take a look into the code, there may be some bug in the Calibrator.cs script.

The current pose features come from the motion matching skeleton before the skeleton is retargeted to unity for its rendering. Therefore, the current pose features actually come from the pose in the motion matching database. As you said it is created by one user. Let me know if you need more clarification on this!

    • For the body direction, I think it will be really useful to completely replace the module with a body direction tracking using the RGBD camera (or use both trackers + RGBD camera info). With the RGBD camera, you may track the body direction and have much better accuracy, then use it to construct the trajectory features.
    • Also, another of the problems of the current method is that we do not really track which foot the user is moving (i.e., left or right). With cameras it should be possible to capture this information and incorporate it somehow as a trajectory feature or extended pose feature, therefore, with adequate weights, motion matching could try to find a pose moving the same foot as the user. For example, the foot could be in root space and normalized by height as you pointed.
    • Although the current pose is extracted from the motion matching database, as you noted, it may be worth it to try to normalize it by height, and then fill it with actual user pose information retrieved from the camera, which could help with the problem of the feet I was mentioning. I am not sure if it's gonna work tho, because the current pose should normally be as close as possible to the skeleton used by Motion Matching. But it may work :)

Hope I answer your questions!

sh-taheri commented 1 year ago

Hi :)

  1. Actually you were right, the OpenXR Runtime was set to Steam VR, but somehow by changing it to Oculus, the issue still exists, I am assuming some of my other configurations are swrong but can't manage to fix them. (but it is not a big issue for me atm)

  2. Great, thanks!

  3. Thanks a lot for the insights!