Use XRInputSources without being in VR mode

[greggman]

Use case:

I have a web site that would benefit from controller input when I'm viewing the site in 2D but in VR via apps like BigScreen, Oculus Dash, Virtual Desktop.

I can get game controller input via the gamepad API but I can't get VR controller input without starting a VR session which I don't want.

Example: Imagine I just wanted to be able to flick left/right on the thumbstick to go through a photo list on a website.

It seems like controllers should be separated from VR/XR or some option to just get the controller input without going into VR mode would help helpful.

I see this issue

https://github.com/immersive-web/webxr/issues/350

But I'm not looking for DOM interaction. For me, I just want the gamepad data without presenting. I kind of wish the gamepads just showed up on the Gamepad API. If you want to know which gamepads go with which HMD then the XR api could tell you some id but you'd still get your gamepad data via the Gamepad API.

Or, if "inline" still gave me the gamepad data that might work too.

[Manishearth]

As I understand it, browsers are allowed to expose VR controllers as gamepads (there are even extensions for getting pose info, though idk if anyone implements those). It's up to individual user agents to choose to do this.

[toji]

@Manishearth: I don't think those gamepad extensions are ones that anyone is pursuing at this point. I suppose you could expose just the button/axis inputs via the gamepad API, but that would lead to some fairly confusing edge cases when an XR session is started.

In terms of what we could do to allow this, inline sessions are, by default, intended as a convenience for developers that allows them to re-use their XR code paths for a 2D page if they wish but aren't intended to expose any information that wasn't already available to the page. This is primarily so that a baseline inline session can be created without triggering any permission prompts. If any features are requested at session creation time via the optionalFeatures or requiredFeatures then the UA is free to prompt for user consent.

This means we could probably add a feature descriptor along the lines of inline-tracked-inputs (ugh, someone suggest a better name!) that would indicate to the page that the inline session in question wanted access to the same XRInputSources that an immersive session would have access to. Depending on the platform I have concerns about the feasibility of exposing that without spinning up a full presentation loop, but for in-VR browsers like the Oculus Browser it should be relatively simple.

[greggman]

Ok well let me add that it seems like WebXR could also cover stuff like Leap Motion. Leap Motion seems similar to a Valve Index and a Oculus Quest with the beta hand recongition. You don't get an HMD to render to nor an HMD position/orientation but you do get 2 hands and possibly multiple fingers.

Whether or not it will go anywhere I don't know but Dreams on PS4 uses 6DOF controllers outside of VR so it feels like that usecase loosely fits here.

[Maksims]

I was browsing in Oculus Quest, and opened PlayCanvas Editor, and though - would be cool to add controllers input here, with new added multi-window feature in Quest, you could have Editor on left and Launcher on right, so simple operations of editing scene could be done from Headset. Very niche, but "cool" indeed. Those controllers are essentially gamepads, exposing their gamepad values (buttons and axes), seems reasonable from GamePad API point of view. I would even suggest this to be GamePad API related, and nothing to do with WebXR.

[greggman]

I can see that the GamePad API won't work in VR for this usecase (as is) because of privacy issues. It's supposed to be you don't get to see GamePads until the gamepad is used but if you're already in VR then you're using the controllers so there is no moment when the user gets to opt in to having the controllers read like they do out of VR.

[papuSpartan]

Is there any way to do this now? I would like to do exactly what he describes as an example for an image viewer.

[greggman]

So on Oculus Rift S (sorry, don't have a Quest), when I'm viewing the windows desktop, the thumbpads show up as wheel events in the browser. So you can look at deltaX for left right and deltaY for up/down.

It's not perfect though, after about 3-5 seconds of no input they stop responding and you have to wave your hand (or wave the pointer your hand represents) for them to start giving input again.

And, AFAICT, there's no way to get different inputs for left hand vs right hand. But, it did allow me to do the use case above which is flipping through images.