blairmacintyre
commented
5 years ago One alternative way of looking at this might be this.
- stand alone AR/VR displays are creating ideas like "prism" or "holograms", regions of space controlled by the UA into which small-ish apps run and render. (Hololens holograms are not so flexible, but could be; various VR "homes" have similar concepts, or could)
- if we extended the way WebXR is initialized, we could allow a web context (page, extension) to create and render into one, and the UA can mix and control the content as it sees fit. The current UA "shells" on ML1 and Hololens are pretty ill-suited to keep long running things following you around, but they could (and probably will) evolve in this way
- assuming these things, we could allow build on the ideas presented in the context of "dioramas" (divs within pages) to include the idea that a diorama could be in a "shell area". WebXR would have to report the bounds (sounds like stage bounds), and would report the views relative to whereever the shell puts the thing.
- it could / should be possible to then initialize multiple WebXR sessions per page (which will be needed if we ever support dioramas within a page, anyway), if we wanted a single page or extension to support multiple prisms/holograms/etc
I wasn't keen on this path when folks suggested it to me in the context of "webxr extensions on a stand-alone AR display", but it's grown on me. While the current top-level shells of these displays (ML1, Hololens) are pretty ill-suited for it (placement of prisms/holograms is manual and tedious, and fixed in space, for example), it's incredibly easy to see that MSFT and ML will improve these, and the idea that a web page (or web extension) can create content bits that mix into the top level shell with other native apps is really appealing in the long run.
So, all this is to say that if WebXR supported creating these sorts of sessions, the webxr extensions simply need to be able to access webxr, and be limited to creating such sessions. Perhaps?
asajeffrey
jdm
AlbertoElias
cwilso
TrevorFSmith
avaer
AdaRoseCannon
Basic problem:
The WebXR Device API supports exclusive XR sessions created on demand from UAs' tabs or windows but there is another important use case for headset-accessed XR: long-lived and simultaneously running applications.
Examples of such applications include:
There is an existing API for hosting simultaneously running and long-lived applications outside of the context of a single web view: WebExtensions.
This is a proposal that we work toward defining and eventually standardizing extensions to the WebExtension API to support long-lived XR applications.
I tend to envision these applications as living around users as a sort of personal "flock" that appears whenever they're in the "home environment" of their headset web browsers (so, not in a WebXR session) but as with the WebXR Device API and WebExtensions there will be a lot of room for UAs (or XR OS creators) to choose different uses of the technology.
To avoid locking this discussion into the "flock" concept this Issue will use the term "WebExtension XR" (WEXR) apps.
Open questions to be addressed:
What types of sensor and UA data are necessary for WEXR apps to address common use cases?
How can UAs balance that access with security and safety expectations?
How can UAs render multiple WEXR apps while maintaining acceptable performance?
What information passing between WEXR apps and origins is necessary and acceptable?
How can UAs route user input (hand gesture, voice command, etc) to the appropriate WEXR app and prevent inappropriate access to input data by other WEXR apps?
Straw dog proposal
For the purpose of spawning discussion, here's a quick and incomplete idea for one way that these extensions could be defined and used:
manifest permissions are extended to include 'xr-forward-camera' and 'xr-input' values, indicating that the WEXR app requests access to a headset's forward facing camera and to the UA managed stream of input events.
manifest background is extended to include 'xr-script' values that indicate that the WEXR app would like to spawn a persistent background script for XR.
A message protocol and new methods in the WebExtension runtime API are available to xr-scripts in order to provide environment data, input events, life-cycle events, and inter-app messaging.
A new API is created that allows the WEXR app to give the UA a reference to a glTF file that the UA loads and renders in the UA's own graphics context. The WEXR app can use runtime messaging to request changes to the rendered data such as asking that a sub-mesh's position be translated, that a texture be swapped, or that a new uniform be passed to a shader.
The UA provides the user with ways to manage which WEXR apps are active, how and where they appear, and how to route their input. For example, a UA may only allow WEXR apps to appear inside a finite space in a known position with markers indicating the edges of the space. The UA might only pass on input events to a WEXR app if the user is looking directly at the app and makes a "finger-gun" gesture to activate the app.
Possible next steps
It would be possible to experiment with these ideas using only WebVR (eventually WebXR) by building a prototype environment that emulates a UA's "flat" user interface elements and home environment and writing a mock WebExtensions API. WEXR apps could then be written to run inside this test environment, enabling quick discovery of at least the basic developer experience and expectations.