DOM Overlays (for Phone AR support and more)

[toji]

As described in immersive-web/webxr#394, when we move away from allowing AR content inline there's a desire to still enable 2D UI to be built using DOM for cases like phone AR, where support for displaying DOM elements and the AR stream together should be pretty trivial. (Previously the same effect would have been supported by using the fullscreen API on a parent element that contained both the AR canvas and some overlaid DOM elements.) Given that we want to utilize the advantages of the web as our platform whenever possible, it would be incredibly unfortunate to lose this ability in pursuit of an explicit AR mode.

For reference, consider this image of Pokemon Go:

If created with WebXR, ideally the animal and likely the ball at the bottom of the screen (because it has an associated throwing animation) would be rendered with WebGL as part of the core session rAF loop, but the other UI elements would ideally be handled as standard DOM elements that are simply composited over the AR content by the UA. (The name of the animal floating over it's head is a special case that I'll address more in a second.)

However, while the core need is to retain DOM support for phone AR there's also a desire to potentially enable that DOM content to be surfaced on headsets, with the idea being that the AR content would be fully immersive while the DOM portion is composited in by the UA somehow to ensure that the user can still access it. Exactly how the would appear is an open question, and one that we probably would want to leave up to the UA to avoid prescribing unproven UX patterns. Some possibilities I could see are:

• Having the overlay DOM be a floating, moveable window in space
• Attaching it to your wrist
• Pinning it to a wall
• Showing/hiding it with the push of a button

Doing so would likely be considered a "compatibility" mode, and would definitely not be a path we'd encourage for developers explicitly targeting headset AR. The big benefit being that it would enable AR content built for the more common devices (phone AR) to still be accessible on more advanced devices, thus immediately increasing the content that's accessible to them.

That said, there's also been some concerns voiced that supporting DOM like this in headsets could be difficult for some platforms, or would be hard to make a good user experience. As such, I'm reluctant to say that supporting a DOM overlay should be required for all devices that support AR. And certainly I believe that we should offer the right signals and tools in all cases to allow developers to explicitly create experiences optimized for any given devices they choose to support.

So this issue is simply to talk about how we should go about supporting those overlays and what guarantees of availability that mode should have.

Some other considerations:

• In my mind this is explicitly different than a DOM-to-texture or DOM layer solution, which would primarily be aimed at enabling DOM content to be shown in a developer-controlled way in 3D space. While you certainly could use such a mechanism to achieve the same effect it has a lot more technical, security, and ergonomics issues involved, doesn't feel like the right fit for a simple "I just want a couple of DOM buttons" UI cases, and the complexity would likely prevent us from shipping any time soon.
• As pointed out by @blairmacintyre at the AR F2F, it would be ideal if we ensured that cases where DOM UI isn't desired could be optimized by the UA, which would no longer have to do some processing on the DOM tree/compositor.
• We expect that users will attempt to do alignment of DOM elements and 3D rendered elements, probably involving gratuitous amounts of matrix math and CSS 3D transforms. See, for example, the name floating above the animal's head in the image above. Problematically, these types of alignment would probably function OK for phone AR but be next-to-useless in headset AR, especially if the developer has no sense of where the DOM content is relative to the AR content. It's an open question for me if we want to accept this as inevitable and encourage this type of use by providing spatial mapping functions, or if we should discourage it for the sake of headset AR by explicitly making it difficult to reverse-engineer the spatial mapping between DOM and AR content.

[ddorwin]

Other possibilities for headsets might include head-locked / waist-locked / viewport-aware positioning. In particular, the HUD-like UI in this image seems like something the user might always want in their view.

Speaking of which, how best to render the content in a headset may depend on the use case. It might make sense to have the author specify the intended use of the overlay as a hint to the UA. Requiring such a parameter may also reinforce the idea that the author should consider headsets and that this might not always line up with the real world.

Regardless of whether we say implementations MUST or SHOULD support such an overlay, all implementations will probably want to provide some baseline support because otherwise many experiences may be unavailable and/or broken on such clients.

We'll also need to address hit-testing, both from a spec and implementation perspective (only hit things in the DOM layer that are visible) as well as how authors build applications. For example, "throwing" the ball could be implemented entirely in the DOM layer on a smartphone, but that wouldn't work well on a headset.

[blairmacintyre]

Regardless of whether we say implementations MUST or SHOULD support such an overlay, all implementations will probably want to provide some baseline support because otherwise many experiences may be unavailable and/or broken on such clients.

One take-away I had from yesterday's conversation was that we should really separate the discussion of this feature from "how would you use this in an HMD" and focus on how would this look as an optional feature.

So, I STRONGLY VOTE that this be a feature that can be unavailable, and in the spec it initially says something to the effect that a browser may present this DOM overlay in any way they want, or not present it at all.

It seems to me that phone-based AR (and perhaps "VR") would have high-motivation to implement it. HMD-based browsers may or may not want to fiddle with it, ESPECIALLY if it's hard with whatever underlying browser tech they are using.

For example, the Servo team demonstrated high performance DOM-2-Texture, which could be the basis of this for a browser based on Servo. But other browser engines (especially on lower end devices) might not be able to render this efficiently.

By making this explicitly optional, we may reduce the number of experiences that just fail in HMDs.

[ddorwin]

By making this explicitly optional, we may reduce the number of experiences that just fail in HMDs.

Does this assume that making the overlay explicitly optional will result in authors adding equivalent controls in 3D space? In other words, because the overlay is not guaranteed to work on all user agents, authors will dutifully add a 3D version?

[blairmacintyre]

Does this assume that making the overlay explicitly optional will result in authors adding equivalent controls in 3D space? In other words, because the overlay is not guaranteed to work on all user agents, authors will dutifully add a 3D version?

Obviously not. But, it does assume that most developers do not code against the lowest level APIs, and so by making it optional, my assumption is that it would encourage framework and toolset authors to come up with more interesting and elaborate tools to fill this void.

[ddorwin]

That seems like a lot to expect from frameworks since the UI needs can vary greatly. Also, developers don't need a framework to do 2D overlays today and WebXR framework don't need to support them. Thus, we'd be moving something new into frameworks.

At least in the near future, I think it's more likely that authors of AR applications will still only implement UI for phones, especially since the author may not even have an HMD for testing. While such apps may render in "compliant" HMDs, many would be unusable because they are missing critical UI.

I think a car configurator is a good test case. It would be a lot of extra work to implement the configuration UI in 3D space, and this UI isn't something that a framework could generically handle (well). If the author does not implement the 3D UI, the result in such HMDs may just be rendering the default configuration. That may be somewhat useful for walking around the car. On the other hand, if the car is not rendered until the user pushes a button, the application just fails in an HMD.

[blairmacintyre]

Reviving this after chats with @lincolnfrog this week.

It seems like there are two basic approaches to DOM, corresponding to the two major platforms:

• "Smarphone overlay". On smartphones, we would like an actual DOM overlay that looks and behaves like a web page, receives touch events, etc. Given that most instances of phone-based XR will be embedded in a web page, dealing with this sort of thing will be easy for apps.
• “In world geometry". For HMDs (and possibly even supported on phones), the UA would put some geometry (quad, or a cylindrical quad, etc) in the world.

The former is easy to envision. Some sort of capability test would see if that's possible, and if so, interaction could be handled by normal DOM events. Interaction events on the 3D world would only happen when the user touches a places that is transparent in the DOM element.

For the later, it occurs to us that interaction could be handled by piggy backing on the world hit testing proposal.

If we treat the DOM geometry as "real world geometry", then when one of the configured hit rays intersects it, it can return it as a hit in the same way as the world hits are returned. At that point, the app could decide if it wants that hit to be "used" in the DOM (e.g., an app might notice there is virtual content between the user and the hit point on the DOM geometry, and thus not want the DOM to react to the ray), and if so, call some sort of method with the hit result as a parameter, telling the UA to have the DOM react. The app could also pass along button presses / releases.

The open question is how / where to position the DOM element. We probably want some sort of show/hide capability. Perhaps the DOM element could be positioned in the world (where the original 2D page was? attached to some well defined coordinate frame?) or attached near a controller or other known location.

Perhaps there might be some pre-defined behaviors, like the "follow behavior" that Hololens uses for the system menu?

[NellWaliczek]

Potentially related to the work now happening in https://github.com/immersive-web/layers

[ddorwin]

For those following this issue, I made a proposal to incubate DOM layers: https://github.com/immersive-web/proposals/issues/50

[TrevorFSmith]

I agree that this seems like a good feature for an incubation repo.

The usual procedure is:

• use Issues to get a general agreement about the approach from multiple orgs
• one or two people step up to be feature champions
• I create the repo
• the champions write the initial explainer
• iterate until it's done or dead

@ddorwin Are you up for championing this feature?

[johnpallett]

@TrevorFSmith I spoke with David, I can be a 'feature champion' for this one.

[TrevorFSmith]

I have created a feature incubation repo for DOM overlays.

I will close this Issue and future conversations should happen in Issues and PRs in the new repo.

[cwilso]

Reopening to keep this as a milestone marker. Discussion on this topic should occur the new repo.

[NellWaliczek]

Moving to AR module while we figure out the correct final location for this issue

[NellWaliczek]

Moving to the already existing dom-overlay repo

[klausw]

/agenda Status update on DOM Overlays experiment, now live in Chrome Canary as described in https://github.com/immersive-web/dom-overlays/blob/master/explainer.md#sample-implementation

[klausw]

Closing, this was a process issue related to creating the incubation repo which now exists, and development has been ongoing here including using the https://github.com/immersive-web/dom-overlays/issues issue tracker.