search

uhhhci / immersive-ngp

We present the first open-source VR NERF Unity package that brings high resolution, low-latency, 6-DOF NERF rendering to VR. This work is based on Nvidia's ground breaking instant-ngp technique.
Other
294 stars 25 forks source link

VrRig and display area #16

Open szawel opened 1 year ago

szawel commented 1 year ago

Hi, really great job with the Unity plugin.

However, I'm currently encountering an issue when trying to synchronize the camera in Unity with the plugin, my setup with the Oculus Quest 2 on Win10 RTX 4090 I'm experiencing a "swimming" effect, as if the NeRF does not stick to space, when I move around the stage. It appears as though the texture from the plugin is being rendered by a stereo camera with settings that differ from those of the Unity rig. Could you advise me on how to approach this issue or suggest ways to minimize or eliminate this problem? or where to modify the code.

keli95566 commented 10 months ago

Hi there! Thank you for the great question, and good catch!

We have noticed this issue and the "swimming" effect is due to the latency between the time when the camera pose is updated and the time the textures are finished rendering.

I have recently figured out a potential solution to solve this issue as I was developing another render plugin for Unity/Cuda/OpenGL.

Basically, we need to use a callback event ( Camrea.onPreCull or Camera.onPreRender) to draw the textures before it appears to user's eyes. In this callback event, we call the plugin event function: GL.IssuePluginEvent(NerfRendererPlugin.GetRenderEventFunc(), DRAW_EVENT); rather than calling this draw event in the update() loop.

I will try to implement this and push the update as soon as possible.

I can will try to integrate the solution in the next next weeks or so.