search

vuer-ai / vuer

Vuer is a 3D visualization tool for robotics and VR applications.
https://docs.vuer.ai
MIT License
121 stars 3 forks source link

webrtc streaming choppy in VR #27

Open tonyfd opened 2 weeks ago

tonyfd commented 2 weeks ago

I'm using Quest 3 to replicate the OpenTeleVision project, utilizing WebRTC to transmit the camera feed. As in OpenTeleVision's work, I used WebRTCStereoVideoPlane and set the source to my WebRTC server address.

When I access the Vuer page on port 8012 in the Quest3's browser, after establishing the socket connection, the camera feed displayed on the plane is very smooth before entering VR mode. However, once I enter VR mode, the feed becomes very choppy, with significant latency. What could be the reason for this?

Similarly, when I use the image mode in OpenTeleVision, i.e., not using WebRTCStereoVideoPlane but instead using ImageBackGround to set and transmit images, there is not such a large difference between entering or not entering VR mode, though the overall transmission quality is worse than WebRTC. Therefore, I still prefer to use WebRTC, but it’s strange that it becomes choppy as soon as I enter VR mode.

My Quest3 and the machine running the Vue server are on the same local network. The server machine is connected to the router via a wired connection, while the Quest3 is connected wirelessly.

lmj250 commented 2 weeks ago

Are you connected through adb? Does Quest 3 need a network to open the Vuer page?

tonyfd commented 2 weeks ago

Are you connected through adb? Does Quest 3 need a network to open the Vuer page?

I tried connecting directly with adb reverse, as well as connecting by WiFi, both encountered the same issue:

  1. When accessing the Vue page on port 8012, after connecting the socket, the WebRTC stream displayed on the plane in the middle of the webpage is smooth and has no delay before entering VR mode.
  2. However, when I click "Enter VR," the image immediately becomes noticeably choppy.
  3. Interestingly, when you press the home button on the Quest3's controller and bring up the browser window again, the VR mode's image continues to run behind the browser window. If you do not click the "Continue" button to return to VR mode, but instead move the browser window aside so it doesn't block your view, you will notice that the VR mode image running behind becomes smooth again. However, once you click the "Continue" button to return to VR mode (causing the browser window to disappear), the image becomes choppy again.
lmj250 commented 2 weeks ago

Is your socket automatically connected by running the teleop_hand.py？