Define how input is handled

[cabanier]

Only the UA really knows where layers are placed in 3D space. The author (or JS framework) can guess where they are but this might be brittle.

What is the best way to provide input events? Should non-projection layers become event targets?

[Manishearth]

I feel like we could trigger select events with xy coordinates?

Having xy coordinates would be useful for screen-space select events as well.

[cabanier]

I feel like we could trigger select events with xy coordinates?

Would it return a x/y coordinate + layer?

[Manishearth]

Oh, I was suggesting that we trigger select events on the layer, and we extend select events to also contain optional xy parameters when found on a layer. Or have an XRInputSourceLayerEvent: XRInputSourceEvent with xy coordinates. I like the former solution because we can use the same thing with screen space input.

[avaer]

What about projection layers? Would that be a special UA case, or left for the page to handle?

[cabanier]

Oh, I was suggesting that we trigger select events on the layer, and we extend select events to also contain optional xy parameters when found on a layer. Or have an XRInputSourceLayerEvent: XRInputSourceEvent with xy coordinates. I like the former solution because we can use the same thing with screen space input.

I agree. I think this means that we should make layers event targets.

[cabanier]

What about projection layers? Would that be a special UA case, or left for the page to handle?

I think they could handle events like in regular WebXR,

[Manishearth]

Perhaps we should open an issue on the core spec about adding xy coordinates to input source events, and perhaps extending the targetraymode to include something new for layers (we could also just use screen again).

[Manishearth]

So while discussing this with @asajeffrey we realized this might be trickier than "slap an onselect event on XRLayer and call it a day".

Basically, WebXR doesn't really strongly define a pointing ray. targetRayMode is the origin of the "preferred pointing ray", but applications are free to add an offset to it (for whatever reason). So we don't have enough information to actually run a hit test, unless we provide the user a way to specify an offset for certain.

Perhaps the shape of this API needs to be similar to the hit test API, using an XRRay.

Some ideas:

• You can register an XRRay on the input source, and it's used to trigger select events on the layer.
• You can create LayerHitTestSources with XRRays and input sources, and events fire on that instead.

• The input source has an attribute XRRay pointerRay, initialized to XRRay(), that we can modify.

  I like this last one the best because it still preserves the original API we were discussing here for the simple case, with the option to tweak it if you need to (which is easily done by constructing an XRRay with the origin offset used on the target ray space).

[Manishearth]

It also occurs to me that we may not just want select events, but also pointerOver, pointerOut, pointerMove events so that applications can render a reticle. Though for a reticle the x,y coordinates in https://github.com/immersive-web/webxr/issues/978 aren't enough, if you also want to truncate the pointer you need world space coordinates of the intersection point, or a ray distance.

This may be a separate issue, just wanted to mention it.

[cabanier]

It also occurs to me that we may not just want select events, but also pointerOver, pointerOut, pointerMove events so that applications can render a reticle.

Yes.

Though for a reticle the x,y coordinates in immersive-web/webxr#978 aren't enough, if you also want to truncate the pointer you need world space coordinates of the intersection point, or a ray distance.

I think that depends where you want to render the reticle. If it's in the layer that is hit, x/y would be enough. If you want to do it in a projectionlayer on top, yes, you'd also need the position in 3D. Maybe we need to provide both...

[cabanier]

• You can register an XRRay on the input source, and it's used to trigger select events on the layer.
• You can create LayerHitTestSources with XRRays and input sources, and events fire on that instead.
• The input source has an attribute XRRay pointerRay, initialized to XRRay(), that we can modify.

I like this last one the best because it still preserves the original API we were discussing here for the simple case, with the option to tweak it if you need to (which is easily done by constructing an XRRay with the origin offset used on the target ray space).

This XRRay would have the pointing ray as preferred by the author? If so, yes I agree.

[Manishearth]

If you want to do it in a projectionlayer on top, yes, you'd also need the position in 3D. Maybe we need to provide both...

Well, this is more so that you can truncate a pointing ray. so that it's not poking through the layer.

We can subclass the event to add a rayDistance, though.

This XRRay would have the pointing ray as preferred by the author?

Yep. Initialized to the default XRRay, but can be set to anything else.

[Manishearth]

Supporting pointer move events might have a problem: that seems more of a performance pitfall (since you're hit testing every frame) compared to select events. I wonder if it should be opt in somehow.

(UAs could theoretically run it only when there are events attached, but idk if that's easy to practically implement)

[cabanier]

Supporting pointer move events might have a problem: that seems more of a performance pitfall (since you're hit testing every frame) compared to select events. I wonder if it should be opt in somehow.

I don't think it will be that expensive. There aren't that many layers, we know where the layers are and the math is not super complex.

(UAs could theoretically run it only when there are events attached, but idk if that's easy to practically implement)

True.

[Manishearth]

There aren't that many layers, we know where the layers are and the math is not super complex.

Seems fair. Computers are fast.

[Manishearth]

It seems like the dom overlays spec will have the same issue, btw, see https://immersive-web.github.io/dom-overlays/#onbeforexrselect .

It would be nice if our solution here could apply there as well.

Filed https://github.com/immersive-web/dom-overlays/issues/29

[Manishearth]

Annoyingly, if we solve this issue in this spec, we'll need to have a dom-overlay -> layers -> hit-test -> core dependency chain :upside_down_face: . But we can move stuff around if we need.

[cabanier]

@Manishearth maybe all we need is a method on XRSession that takes an XRRay (or XRRigidTransform) and synchronously returns what non-projection layer was hit along with the locations in 3d space and the layer space. It's a simple solution that's easy to implement for authors and UAs. Thoughts?

[Manishearth]

Hmm, that's an interesting proposal! Something like the hit test API, but for layers?

I still prefer the event based API, but we can still make the event API subscription-based.

I do feel that as time passes there will be an increasing need for the content to tell the UA what its preferred pointing ray is, e.g. for DOM overlay

[cabanier]

Yes, we'll need it for HTML layers as well. I believe the intent is that the UA will draw the controller with pointing rays to make sure that it can't be spoofed.

[Manishearth]

Hmm, that still forces a choice of pointing ray, however. But I guess that's somewhat inevitable? I was wondering if we could "lock in" the choice of pointing ray so content cannot rapidly change it, but that might not be sufficient as a security boundary.

[cabanier]

Yes, but that choice will be made by the UA. Events for HTML layers will be the regular 2d events but delivered to the elements.

[cabanier]

During today's call, we agreed that because this can all be figured out on the client side, there's no pressing need to add this as a javascript API. I will keep the issue open but mark it as a future enhancement and will close the PR for now.

[digisomni]

In the application I mainly develop for based on C++, the general idea is to emulate a touch device on the web engine, essentially act like a VR user is an Android user so websites cooperate.

I take it this is not yet a strictly defined thing for WebXR so we'd have to try and manually implement such a thing if we wanted web pages/apps to be interactive in VR?

[cabanier]

@digisomni are you talking about a web browser that is in VR? If so, we will have to define hit testing for WebXR DOM layers.

[digisomni]

@cabanier Yeppers, I have Vue based web apps that users interact with in VR. This works because touch events are emulated to the QtWebEngine. It's my hope that we can get support for this in the web sooner than later! :) That way apps and UI can be developed for both desktop and VR simultaneously much more quickly.

I see it's not been discussed for over 6 months. What are current prevailing thoughts on the matter?

[avaer]

Is there currently any path for WebXR DOM layers for VR, which is presumably the main use case for such a hit test feature? It is less useful for window AR DOM, which has ways to do hit detection/events handling already.

I remember this was discussed at past F2F's.

(c.f. https://github.com/immersive-web/dom-overlays/blob/master/explainer.md)

[cabanier]

Is there currently any path for WebXR DOM layers for VR, which is presumably the main use case for such a hit test feature? It is less useful for window AR DOM, which has ways to do hit detection/events handling already.

I made a proposal at the last face-to-face but I can't find the presentation on the immersive web github. I will write an explainer soon and post it to the layers repo.

The basic idea is to allow layers to point to a same-origin URL.

[cabanier]

@cabanier Yeppers, I have Vue based web apps that users interact with in VR. This works because touch events are emulated to the QtWebEngine. It's my hope that we can get support for this in the web sooner than later! :) That way apps and UI can be developed for both desktop and VR simultaneously much more quickly.

Having content from arbitrary websites in a vr session will likely not be allowed but we hope that same origin content will work. As part of this work, the UA will send events from the controller when it hovers over the layer.

[digisomni]

It's a start! I mean all the apps are running on the same origin, but even if they weren't wouldn't setting up CORS allow it to work pretty well? I'm hoping to allow people to make apps and content, push it to a central repo (that's vetted) and then have that be shared to various clients as add-ons if they'd like.

But of course, my priority is to get any web surface working within VR, really.... because making UI in web surfaces is a massive time saver. Offloading the heavy lifting of general UI management to the creators of Vue and then components to the creators of Vuetify for example frees up resources to work on other complex needs for expansive multiplayer worlds.

[cabanier]

It's a start! I mean all the apps are running on the same origin, but even if they weren't wouldn't setting up CORS allow it to work pretty well? I'm hoping to allow people to make apps and content, push it to a central repo (that's vetted) and then have that be shared to various clients as add-ons if they'd like.

The first proposal will only work with same origin content. Does CORS work on web content? I thought it only applied to resources.

[digisomni]

Well, technically everything is a "resource" right? I can download a a bunch of content and then import that with local scripts to display as maybe an idea.

But overall, I'll be happy with getting iframes in 3D to start, even if that's just same domain. Web UI is infinitely valuable due to its massive library of components from every corner of the internet. There should be no need to invent a new UI framework to use in 3D applications, y'know?

Currently I believe the only way to get web pages in 3D is through illusion but that won't do for a proper and simple integration pipeline.