Added VR and XR support

[WawasCode]

The intention is to enable VR in drake. For that multiple functions have been added to the View class.

enable_vr() : Enables VR for the current scene. enable_xr(): Enables AR/ Mixed Reality for the current scene. build_vr_controllers(): Enables the VR controllers in the scene.

---

This change is 

[RussTedrake]

@SeanCurtis-TRI -- might you be able to take a look?

[SeanCurtis-TRI]

Yeah. I'll take a look. I obviously need to subscribe to notifications for this repo. Thanks for flagging it for me.

[jwnimmer-tri]

What does "take a look" mean. Are there open questions that we need Sean to answer?

If you mean a code review, I see several important blockers before TRI could approve this -- no instructions on how to run it, nor any Drake meshcat_manual_test coverage.

[SeanCurtis-TRI]

I wasn't even able to build this PR.

I ran yarn and npm run build and got all sorts of complaints about not being able to resolve three/examples/jsm/webxr/XRButton.js (see below).

Did you need to update package.json or yarn.lock?

It'd be helpful if you could post instructions for testing the code as is.

[SeanCurtis-TRI]

Don't worry, I haven't forgotten you. I've been trying to track down a VR headset...I'll tackle this first thing Wednesday morning.

[SeanCurtis-TRI]

Previously, SeanCurtis-TRI (Sean Curtis) wrote…

I'm working on your implementation of this. As I look at the current functionality, I have several thoughts: 1. The first thing I observed is that it didn't seem that the viewpoint camera in VR is where I would expect based on the camera view when I hit the button. My expectation is that if something is visible before I hit the button, it should be more or less visible after I hit the button. This isn't the case. 2. We need to incorporate orientation as well. - By "incorporate" I don't mean literally offset the orientation of the non-VR camera with the headset. But we should incorporate heading. For example, if I've rotated the camera around to look at the scene from "the back", then when I hit the VR button, I should be behind the scene and looking towards it. So, the teleport should incorporate some rotation around the world's vertical axis (but not any of the others). 3. In principle, one thing that *seems* like it should be true is that if I hit `Start XR` and then immediately disconnect without moving, my non-VR camera should be restored to its original position. That isn't currently true. I think we need to account for the offset transform in the teleport reference space when the session ends. I'm tinkering with this, but wanted to publish this comment to give you the chance to consider yourself simultaneously.

I've attempted to address all of these issues in https://github.com/WawasCode/meshcat/pull/1.

Please take a look. If you agree, you can simply merge it into your branch and things will update here automatically.

Note: There's one thing that I didn't have a chance to figure out. The perspective camera changes in some appreciable way after first entering a VR session. When you come back out of the session, the camera seems to have a slightly different focal length. If you repeatedly enter and exit a VR session more, it doesn't seem to drift any furhter. It's just the initial session that changes the perspective. That's not entirely surprising as VR can use an arbitrarily different focal length. However, it's something we might consider addressing.

[SeanCurtis-TRI]

Previously, WawasCode (Watsche Khalatyan) wrote…

I'm unclear on what you mean by "It ruins everything." I changed it such that when VR is enabled, the camera projection matrix can't be changed.

I like what you've done with the current camera's projection matrix update. That's a good start. But it's not enough.

The user can actually send a command to change the camera to be an OrthographicCamera via calls to set_camera(). I think you need to put a guard in that function rejecting camera changes while VR is active.

The OrthographicCamera does not play nicely with the WebXR at all. As soon as it changes to an orthographic projection, the VR renderings show nothing. The same is true even if you start with an orthographic camera and start a XR session. So, here's what I propose:

1. In set_camera we should only take action if we're not already in a session. (So, we need some flag in viewer that gets toggled when sessions start and stop. This is finer granularity than enabled or not.)
2. If set_camera actually does work (because we don't have an active session), then if the camera is orthographic, we should disable the WebXR button indicating it can't be used with an orthographic camera.

[WawasCode]

Sorry I think i squasht to mutch :/.