search

Improbable-AI / VisionProTeleop

VisionOS App + Python Library to stream head / wrist / finger tracking data from Vision Pro to any robots.
MIT License
308 stars 17 forks source link

Visual feedback in the headset #2

Open Xingyu-Lin opened 4 months ago

Xingyu-Lin commented 4 months ago

Hi,

Thanks for the great work!

Is there a way to stream videos back into the headset? Currently, all the windows inside the headset are closed when the teleoperation app starts tracking, so it becomes hard to stream the visual observation back. Would love to see if there is a simple way around this. Would it be possible to allow other windows to be kept open?

StoneT2000 commented 3 months ago

Would also be very interested in this! Our side tried to tackle it by using ALVR (and we stream visuals via ALVR) but ALVR has only just started some experimental support for apple vision pro and it does not quite work just yet.

younghyopark commented 3 months ago

Hi @Xingyu-Lin, apologies for the delayed response, and thank you for your interest in our repository!

We are working on an updated version where you can easily stream visual observations to Vision Pro from any client machine using Python. If stereo images are provided, it feels like a spatial video stream as well. I am currently in the process of refining the code and preparing the necessary documentation, planning to release this update sometime next week! Hope this update is helpful for your application!

Xingyu-Lin commented 3 months ago

Looking forward to the update!

Hao-Starrr commented 1 month ago

Amazing work! @younghyopark Thanks for the great work! Is there any updated version released?