Diorama-style content

[NellWaliczek]

Now that immersive-web/webxr#409 has been merged, I wanted to get a conversation going about how we think about supporting "diorama" style content in WebXR. In the past, we’ve called this “punch through”, but the rough idea is that it looks like a diorama box that hangs out of the backside of a 2D webpage. The primary usage would be on devices like zspace or for 2D browsing within an immersive headset (either AR or VR hardware). And in fact, Magic Leap shows this sort of experience in some of their Helio examples, though not through WebXR.

I've filed this issue in the proposals repo, because it's a bit unclear to me if this ought to be part of the WebXR 1.0, and there's a fair number of open issues that would need to be resolved before we'd have a specific design. Perhaps @rcabanier and others may have an opinion about the urgency of this?

Ok, on to the technical. When contemplating this functionality, there are a number of things i've been considering:

• The content placement relative to the user is fundamentally different than what is traditionally thought of as “magic window”. In “magic window” the content is centered around the user. However, with “diorama”, content is limited to a specific bounding volume that will be viewed from a number of different positions and angles. It’s like looking into a keyhole.
• Dioramas would really require stereo rendering when being viewed from within a headset.
• The scale of content in a diorama may not be 1 unit = 1 meter. Not sure how I feel about that, tbh.
• Would dioramas have the same refresh rate as the page they are embedded into? In other words, could the window RAF be 60hz on a headset that otherwise refreshes at 90hz? If so, what does that mean for the refresh rate of the output context and XRSession RAF?
• User position drives changes in view matrices, but maybe also projection matrices? This raises a handful of questions:
  • On systems which do not have pose data to support this behavior (ex. Handheld mobile phones), I assume we want to allow the fallback of developers to using touch/mouse input to change the viewing angle. With the current designs, developers would need to toss out the projection matrices supplied by WebXR and author their own. That seems… not good.
  • Another related problem is that when a user is wearing a headset, we definitely don’t want developers to use their own projection matrices, which ends up bifurcating the rendering path.
  • I'm not 100% clear on the math here, but it seems like the near clipping plane isn't something that the webxr developer is going to be able to set. Perhaps we just ignore the requested values if that's true?
  • Lastly, there are still open issues about how developers get some form of permission from users to have access to pose data. For “magic window”, the bar may be lower if we restrict it to 3DOF data. However, “dioramas” by their nature will require 6DOF pose data. As a result there are open issues about what is the right time/way to request that permission in the flow of session creation and frame of reference creation.

My current thinking is roughly as follows:

• Resolve immersive-web/webxr#412 in such a way that a XRFrameOfReference must be selected before WebXR can return projection matrices
• Add an XRDioramaFrameOfReference that will cause the views array to be stereoscopic. On the one hand, it would be a bit peculiar with the current API shape to have the usage of a specific frame of reference change the number of views. However, maybe that becomes less of an issue depending on how we solve issue immersive-web/webxr#412
• Provide a mechanism by which developers can explicitly request pose data for inline sessions include 6DOF data and that user’s have an opportunity to consent. If we think it’s ok to make this unique to XRDioramaFrameOfReference, then it may be as simple as requiring developers to pass that as the requiredFrameOfReference property. That said, I’m not sure we want to make that restriction, or even if using requiredFrameOfReference is the right design if we do.
• Modify the design to support URP to have developers pass in a transform they request be applied to the view/projection matrices. (This is the part I’m the most iffy about)

Thoughts? Anyone interested on working on this together?

[rcabanier]

Thanks for kicking this off Nell!

As you know, we currently have a high level API to mix javascript generated content with other running applications so it can be offloaded to a central location that knows where everything is. It is not like a diorama as each prism is 3d in a 3d world. I'm doubtful if this can be mixed with WebXR.

What you are proposing is certainly an interesting way to show 3D. It is used in multiple locations in our product (ie the MLOne's intro screen has a portal diorama and the Dr Grordbort game has diorama's of the robot planet). I think we'd be happy to work on this.

Is there any interest from other vendors? I suspect that this will work nicely on handheld as well. "views" would just return a single element. You'd also be able to layer HTML elements on top which some people expressed interest in.

I agree that the only required permission would be "pose" as these dioramas would not interact with the real world.

[Artyom17]

Definitely an interesting thing to implement in Oculus Browser.

[Artyom17]

BTW, walking on Lyon after the first day of the conference, I found a perfect example of diorama. Watch the video: https://youtu.be/Q7SNN30scSs

[toji]

That's great! Thanks for the video @Artyom17!

FWIW I think that this also helps demonstrate the non-1:1 scale issue too. Setting aside physical limitations, this is fun to view as miniature, and has been designed to fit it's viewport well. If the scene was full-size this little window wouldn't be terribly effective. You'd only see a small chunk of the scene at a time unless you're physically right next to the window. (Plus, miniatures are just cute!) However, it seems like a natural thing to say that if I want to jump from viewing this as a diorama to a fully immersive scene, I'd probably want everything to be 1:1 scale. You can easily imagine the same principle for, say, a car customization site.

So the point about dioramas not forcing 1:1 scale is simply to allow content to fit the the size viewport, which is unpredictable. (I may be viewing content on a phone, a 30 inch monitor, or a virtual 6 foot tall wall of browser content in a headset.) The content should be well-framed in all cases. We can achieve that either by having the diorama frame of reference automatically provide scaled matricies, or by providing the developer with enough information about the viewport that they can compute the transform themselves.

[jgwinner]

BTW, walking on Lyon after the first day of the conference, I found a perfect example of diorama. Watch the video: https://youtu.be/Q7SNN30scSs

@Artyom17 Artem, is it OK if I give a reference to this video? It's an amazing demonstration of what we're talking about here.

== John ==

[TrevorFSmith]

Hey, folks. It seems like there is interest in this topic but it's stalled. Is anyone interested in working on it in the next few weeks? If not, I'll close it and we can re-open it when we have time to work on it.