At what distance should this overlay be placed w/ Head Mounted MR devices

[raviramachandra]

There is ambiguity with respect at what distance should we place this. Head locked is fine. For placing the DOM overlay, can each UA make the best fit (best experience) decision ? On Magic Leap, lets say we find out that at distance 'x', the overlay has the best visual experience and we place it there. 'x' is what the UA/platform decides ?

[raviramachandra]

After some more thinking and discussion: The idea of DOM overlay being head locked seems consistent with how this feature is expected to work on Mobile phones/hand held AR devices. There are still open questions w/ implementation and user experience on head mounted devices like Magic Leap One. We can continue our collaboration here.

[klausw]

The intent is that the UA is free to pick an appropriate option here. See https://immersive-web.github.io/dom-overlays/#dictdef-xrdomoverlaystate :

An overlay type of floating indicates that the DOM overlay appears as a floating rectangle in space. The initial location of this rectangle in world space is up to the UA, and the UA MAY move it during the session to keep it in view, or support user-initiated manual placement.

NOTE: Future versions of this spec may add additional attributes to the overlay state, for example the current location in world space for a floating overlay, or additional modes such as head-locked.

Technically, head-locked is also a valid implementation of "floating", this note was intended to express that it may be useful to tell the app that it's head-locked as opposed to world-placed. If that's your preferred option, we can easily add it now.

The depth is up to the UA, but note that the spec says that DOM content must not be occluded by scene objects, so there may be Z conflicts if it's placed far from the user. On the other hand, if it's very close to the user, it would be difficult to interact with it using a 6DoF tracked controller's pointer ray.

I had made a proof-of-concept implementation on Daydream using a world-static floating quad, including support for user repositioning, but I didn't pursue other options in detail. The explainer has a section on Display modes with more examples.

This area is definitely open for suggestions and feedback. If a UA supports multiple modes, we could also consider adding a preferred display mode hint that the app can pass in the init dictionary, and implementations could use that to respect the preference if possible. (I used dictionaries for XRDOMOverlayInit and XRDOMOverlayState to make future extensions easier, even though they currently only contain a single element each.)

[raviramachandra]

Technically, head-locked is also a valid implementation of "floating", this note was intended to express that it may be useful to tell the app that it's head-locked as opposed to world-placed. If that's your preferred option, we can easily add it now.

The depth is up to the UA, but note that the spec says that DOM content must not be occluded by scene objects, so there may be Z conflicts if it's placed far from the user. On the other hand, if it's very close to the user, it would be difficult to interact with it using a 6DoF tracked controller's pointer ray.

From the reading of spec it seems for AR/MR headsets, "floating" could be either a world-static floating quad or it could also be "floating and following user" which implies head locked. Would this not cause inconsistent behavior when same content is rendered on two different AR/MR headsets ? Such ambiguity doesn't exist with "screen" state, where is it consistently locked and always visible on top of XR content.

The depth is up to the UA, but note that the spec says that DOM content must not be occluded by scene objects, so there may be Z conflicts if it's placed far from the user. On the other hand, if it's very close to the user, it would be difficult to interact with it using a 6DoF tracked controller's pointer ray.

Ok. It looks like we should experiment a bit to find out a good depth on our device. One another clarification need to figure out the depth would be the size of the DOM Content. Based on the spec:

While active, the DOM overlay element is automatically resized to fill the dimensions of the UA-provided DOM overlay rectangle.

I did not quite follow what this means. Should we treat the DOM rectangle akin to "viewport" ? For example, lets say on our platform the DOM rectangle is W x H meters and can legibly display 1024 x 800 pixels at Y depth (converted from physical size at some pixels per meter ratio), but the DOM Content is smaller, should we just make resize it to fit and thus stretch the content ? Also should we shrink if the DOM Content needs more than 1024 x 800 to show without scrolls ? Need some more clarity on this area. Because "shrinking" can cause loss of legibility.

[klausw]

From the reading of spec it seems for AR/MR headsets, "floating" could be either a world-static floating quad or it could also be "floating and following user" which implies head locked. Would this not cause inconsistent behavior when same content is rendered on two different AR/MR headsets ? Such ambiguity doesn't exist with "screen" state, where is it consistently locked and always visible on top of XR content.

"floating and following user" doesn't necessarily mean head-locked. A popular method for dialogs on VR headsets is to place them world-static in the user's field of view, but then move them back into view using a jump or rapid transition if the user has moved or rotated enough to make them not visible anymore. This is often more comfortable than strict head locking on VR headsets. and makes it easier to read text by letting the user center specific points in their field of view.

I'm in favor of adding "head-locked" as a distinct type since that does have different semantics from the floating variants. For those, would it be useful to expose the current world placement of the overlay in the state?

One another clarification need to figure out the depth would be the size of the DOM Content. Based on the spec:

While active, the DOM overlay element is automatically resized to fill the dimensions of the UA-provided DOM overlay rectangle.

I did not quite follow what this means. Should we treat the DOM rectangle akin to "viewport" ? For example, lets say on our platform the DOM rectangle is W x H meters and can legibly display 1024 x 800 pixels at Y depth (converted from physical size at some pixels per meter ratio), but the DOM Content is smaller, should we just make resize it to fit and thus stretch the content ? Also should we shrink if the DOM Content needs more than 1024 x 800 to show without scrolls ? Need some more clarity on this area. Because "shrinking" can cause loss of legibility.

The intention was that this acts similar to fullscreen mode. The effective displayPixelRatio and size in pixels is up to the user agent, but there shouldn't be dynamic resizing.

Would it help to add a section with user agent style sheet defaults? Currently, that would be https://fullscreen.spec.whatwg.org/#user-agent-level-style-sheet-defaults plus this in Chrome:

 /* WebXR DOM Overlay for handheld AR needs a transparent background for both the
   fullscreened element and its backdrop. */
:-internal-xr-immersive-dom-overlay :fullscreen,
:-internal-xr-immersive-dom-overlay :fullscreen::backdrop {
  background-color: rgba(0,0,0,0) !important;
}

[raviramachandra]

I'm in favor of adding "head-locked" as a distinct type since that does have different semantics from the floating variants. For those, would it be useful to expose the current world placement of the overlay in the state?

Now I understand better "floating and following" means. And yes, they are semantically different from head locked and if you think "head locked" would be another option, it would make the spec more consistent.

[klausw]

https://github.com/immersive-web/dom-overlays/pull/10 adds "head-locked", what do you think?

[raviramachandra]

@klausw with a minor nit comment, this looks good to me.

[raviramachandra]

 /* WebXR DOM Overlay for handheld AR needs a transparent background for both the
   fullscreened element and its backdrop. */
:-internal-xr-immersive-dom-overlay :fullscreen,
:-internal-xr-immersive-dom-overlay :fullscreen::backdrop {
  background-color: rgba(0,0,0,0) !important;
}

Got it, thanks for the clarification. Yes, the transparent background CSS will help clarify.

[klausw]

FYI, I created https://github.com/immersive-web/dom-overlays/pull/14 about the user agent styling and documenting a pseudoclass.

[klausw]

Closing now that the followup in #14 was addressed.