search

aadhithya14 / Open-Teach

A Versatile Teleoperation framework for Robotic Manipulation using Meta Quest3
MIT License
125 stars 23 forks source link

Questions about VR Headset #1

Closed Nimolty closed 5 months ago

Nimolty commented 7 months ago

Hello! Thanks for making this great works open-source.

I have a question. Is VR Headset Meta Quest 2 suitable for this system?

Thank you so much!

aadhithya14 commented 7 months ago

Yes this codebase works with all Meta Quest devices. It supports Meta Quest 2 as well.

krishpop commented 7 months ago

Hi! thanks for opensourcing this codebase, it seems really useful! When I load the apk on the quest 3 for the first time, the menu page is static and doesn't seem to respond to any controller/hand inputs. Is there a specific order in which the server and app have to start, or is it possibly a versioning-related issue. Any debugging support/advice would be appreciated!

aadhithya14 commented 7 months ago

Hi Krishnan, Thanks for raising this issue. If you don't see black color hand tracking with a red pointer, check if the oculus is connected to internet. As soon as you start the APK, you will see see black color hand tracking with a red pointer and you have to click the menu icon and there will be an option to change IP. . Ensure to add your IP address of the network your robot/simulation is connected to using the drop downs. As soon as the IP is entered, enter go back and click Stream. Similarly on the codebase side Link add the IP address. You should be able to use the robot/simulation using your hand(If you have the wrappers for it). Feel free to email at aadhithya.iyer@nyu.edu for any more queries.

krishpop commented 7 months ago

Thanks for the help! It looks like hand-tracking was disabled and there was no warning/error in place to notice this.

aadhithya14 commented 7 months ago

No problem. The APK expects the user to have hand tracking enabled since the app is fully controlled by hand. Feel free to contact me in case of any issues in the future.

steb6 commented 6 months ago

Hi @aadhithya14 , I am trying to use your system on a Quest 2 as well. I managed to install everything and I was trying to launch the Libero simulation, anyway I have some questions:

Thank you for releasing your code! :)

aadhithya14 commented 6 months ago

1) The APK client and the server runs together. Yes, this is the behaviour when closed loop control between the client and server us not established. So once you start the APK, start the server teleop as well using Link 2) There are two keypoint ports, one for the left hand and one for the right hand, the one displayed on the app is only for the right hand. There is no need to change ports. The teleoperation works of the shelf. 3) The host address is the only thing you need to change and this is the machine running the server. Enter the same IP in the APK as well. The left_xarm_ip is if you want to use any xarm robot. Ensure the VR and the machine are under the same network provider (Like central university or company network).

steb6 commented 6 months ago

Thank you for your answer! I am sorry, but I still need a bit of clarification. When I launch python teleop.py robot=libero_sim sim_env=True on a machine, that machine is the server and it launches the simulation as well, right? The host ip is the ip of the Quest, since I see from the code that it publishes the simulation images on that address, right?

steb6 commented 6 months ago

I would like to add that I have installed the repository inside my WSL2 on a Windows 11 host. I am setting te host_address in network.yaml to 0.0.0.0 and in the VR I put the IP of the Windows 11 host. Anyway, it seems that I can't read the pose of the hands. I noticed that sometimes the panel in the VR app becomes blu, orange, red, green, etc.

aadhithya14 commented 6 months ago

Hi , Thanks for using Openteach. The host ip is the IP of the machine. You need to enter the machine IP on the server side as well as inside the APK. We used Ubuntu, but it should straightforward in WSL2 as well. Set your host_address in the configs to be your machine IP as well as set the IP in the VR to be your machine IP. Feel free to email me aadhithya.iyer@nyu.edu for any questions you have. Looking forward to helping you more with the code base.