Support DOM overlay API

[takahirox]

https://github.com/immersive-web/dom-overlays

[takahirox]

Experimenting for VR so far. Some notes to my self.

1. How to render dom elements to texture?

To render dom elements in VR, we first need to render them to texture. There seems no standard way now. Two wordarounds may exist.

• Using SVG
• Using external library like html2canvas

First rendering to Canvas and then rendering the poly with the canvas as texture in VR?

2. How to trigger dom element event from input event?

Getting the point position crossing the poly and getting the top most dom element, and call dom element .click() (and other functions)?

3. How to inject the poly in VR?

• Rendering the poly before xr frame animation frame? But it disappears if the user code calls gl.clear() in the animation frame. And if the dom element is transparent, it can be overridden by even farther opaque objects in VR.
• Rendering the poly after xr frame animation frame? But it can override closer transparent objects in VR. And if the user code apply post-processing in the animation frame, the poly won't be applied.

[takahirox]

Thinking of another approach. Only DOM overlay elements keep displaying even in immersive mode. (Assuming #239 and #240 will get in.) This is not exactly emulating real behavior but easy to implement.

Coming to think the idea in https://github.com/MozillaReality/WebXR-emulator-extension/issues/222#issuecomment-623801084 may be difficult to implement. For example, even with SVG or external library, simulating perfectly the original dom elements may be hard especially if the elements are complex (event listeners are set, including JavaScript, layout is defined with CSS file, or so on).

[brunobg]

Hey, I saw the commit. So, should be working at this point? Perhaps not with the released addon, but with the master version (and how could I run the master code instead of the release)? Any idea if there will be a version 0.4.0?

I might be able to help with this project.

[takahirox]

Yes, I added an initial DOM overlay support. You need to clone and manually install from dev branch for now.

We needed to reschedule due to the lot of stuffs, but hopefully I want to release the new revision at least once by the end of this year.