search

OpenTeleVision / TeleVision

[CoRL 2024] Open-TeleVision: Teleoperation with Immersive Active Visual Feedback
https://robot-tv.github.io/
Other
585 stars 56 forks source link

Stereo pair streaming #4

Closed nataliya-dev closed 2 months ago

nataliya-dev commented 3 months ago

Hello! In the paper you mentioned that the robot streams a video at a resolution of 480x640 for each eye. How do you align the left and right image from the camera for the operator? Does this value change depending on the person and their measurements, such as pupillary distance? In the example code, teleop_active_cam.py, you send the images to the VR device by vertically stacking them, which does not seem accurate. Also, you reduce the resolution significantly in that example. Is that to get the streaming speed to 60 fps?

chengxuxin commented 3 months ago

  1. Currently, there are no specific adjustments for different people such as pupillary distance. We found humans are pretty adaptable to similar stereo-image pairs. Additionally zed mini's PD is similar to that of humans (63mm).

  2. Thanks for pointing out the bug of vstack. We will update it asap.

  3. Depending on your network conditions, the max resolution could vary. We tested dual 720p streaming with local network without any problems. Reducing the resolution can 1. reduce network load if you don't have a fast and stable network, 2. ensure humans see the same image as the robot for better training consistency.

You can set 

crop_size_w = 1
crop_size_h = 0

for dual 720p streaming, and change aspect ratio for right eye and aspect ratio for left eye respectively.