WebXR Dynamic Viewport Scaling

[klausw]

HIQaH! QaH! TAG!

I'm requesting a TAG review of the dynamic viewport scaling feature that's part of WebXR.

Applications can optionally render to a subset of the WebXR viewport, using a scale factor that can be changed every animation frame. This is intended to be more efficient than resizing the full framebuffer which requires reallocation, and the UA can supply a recommended scale factor based on internal heuristics.

• Explainer¹ (minimally containing user needs and example code): https://github.com/immersive-web/webxr/blob/master/explainer.md#dynamic-viewport-scaling
• Specification URL: This is part of the WebXR API. Relevant sections include requestViewportScale(), the internal attributes and the obtain a scaled viewport algorithm.
• Tests: https://github.com/web-platform-tests/wpt/blob/master/webxr/xr_viewport_scale.https.html and results
• Security and Privacy self-review²: https://github.com/klausw/webxr/blob/master/designdocs/dynamic-viewport-scaling-privacy-security.md
• GitHub repo (if you prefer feedback filed there): https://github.com/immersive-web/webxr/issues
• Primary contacts (and their relationship to the specification):
  • Brandon Jones (@toji) Google, Editor
  • Manish Goregaokar (@manishearth), Invited Expert (Mozilla until 2020), Editor
  • Klaus Weidner (@klausw), Google
• Organization(s)/project(s) driving the specification: Immersive-Web WG
• Key pieces of existing multi-stakeholder review or discussion of this specification: https://github.com/immersive-web/webxr/issues/1091
• External status/issue trackers for this specification (publicly visible, e.g. Chrome Status): https://www.chromestatus.com/feature/5640976515203072

Further details:

• [X] I have reviewed the TAG's API Design Principles
• Relevant time constraints or deadlines: n/a
• The group where the work on this specification is currently being done: W3C Immersive Web
• The group where standardization of this work is intended to be done (if current group is a community group or other incubation venue): same
• Major unresolved issues with or opposition to this specification: n/a
• This work is being funded by: n/a

You should also know that...

This is an addition to the pre-existing WebXR specification editor's draft. TAG review was requested as part of blink-dev feedback.

There's strong developer interest in this feature including from (see blink-dev thread above).

We'd prefer the TAG provide feedback as (please delete all but the desired option):

☂️ open a single issue in our GitHub repo for the entire review

[yoavweiss]

Friendly ping! :) Will this be reviewed in this F2F?

[torgo]

Thanks for bearing with us here. We haven't managed to move this along, unfortunately. We will try to get this back to you shortly.

[torgo]

Hi @klausw - Some questions: have you got developer feedback - e.g. from library developers - on this approach? Can you flesh out the use cases a bit? Since the explainer link points to a subsection of the WebXR explainer can this be considered to be a "small delta" on an already existing spec, rather than a new thing? If so, are there key questions or issues you would like our feedback on? Thanks!

[klausw]

@torgo - this feature was initially requested by the <model-viewer> developers. @elalish had provided feedback and a use case description on the blink-dev thread where we were discussing the feature. @elalish wrote:

Yes, I've been pressing hard for this viewport scaling feature as currently has significantly lagging performance in WebXR as compared to both its on-page rendering and as compared to the native Scene Viewer app. It has been fairly standard practice for some time to dynamically reduce resolution to maintain frame rate, as GPUs only spend time on pixel blocks that have things to render, which in AR is usually not most of them. Only when you have a huge object or get very close to something are you having to run your shaders on all the pixels. Decreased resolution (especially while moving) is much less noticeable than decreased frame rate, and dynamic scaling is the only reasonable way to solve this, since performance varies dramatically due to camera position.

We have already implemented and tested support for this API by enabling it with the Chrome Canary flag. It works great and exactly as expected. Gives about a 4x increase in frame rate in the worst cases.

@torgo wrote:

Since the explainer link points to a subsection of the WebXR explainer can this be considered to be a "small delta" on an already existing spec, rather than a new thing? If so, are there key questions or issues you would like our feedback on? Thanks!

In short, if you feel comfortable that this is a small-enough delta and you trust that the immersive-web group handled this with appropriate due diligence, we'd be fine with just a high-level sanity check to make sure that there isn't anything in this change that seems surprising or concerning.

While I'd personally consider it to be a small delta, the Chrome feature reviewers had pointed out that even small changes should go through the process to ensure that there aren't any potential surprises or issues that could cause long-term headaches for the web platform. Even if it had consensus within the immersive-web group, it hadn't gotten feedback or review from outside that group yet.

Alex Russell @slightlyoff had requested a TAG review in the blink-dev thread:

So, there isn't a way in our process today to skip filing for TAG review for new features, small or not, and in particular not when we're being asked to ship something first. We've been discussing for some time what such an exception might look like and we're actually working through a potential test case with a separate feature right now. I'm hopeful that we'll have some sort of text available in the next month that brings some clarity.

For background, a roughly equivalent feature had been part of the predecessor WebVR API, but it was removed from the initial WebXR specification to simplify it for the first version of that (see https://github.com/immersive-web/webxr/issues/617), with the intent being to bring it back later. It's not all that complex, but it's important to get the details right to ensure consistent behavior across implementations. We had discussions about that in https://github.com/immersive-web/webxr/issues/1091.

As far as specific feedback is concerned, https://github.com/immersive-web/webxr/issues/1091#issuecomment-663778266 and the following comments were discussing some of the subtleties of the API, and I think we got them all resolved to the satisfaction of the immersive-web group. I don't know how deeply you want to dive into this topic, but if you want more detailed information, or if anything jumps out at you as an obviously bad idea, please let me know.

[torgo]

Ok got it @klausw thanks for the clarification. I've put it on for next week and hopefully we can close it then and get you some relevant feedback.

[torgo]

Question from @atanassov in today's call: What is the event mode for when a scale changes for a particular view - like a DPI change on a particular display? Not apparent that the view object has an event for it?

[klausw]

@atanassov asked:

What is the event mode for when a scale changes for a particular view - like a DPI change on a particular display? Not apparent that the view object has an event for it?

I'm not familiar with "event mode" in this context - are you referring to MediaQueryList change events?

In short, I don't think this is applicable. XR views don't have associated DPI values, and the closest equivalent (angular resolution in pixels per degree) is a fixed property of the display device that doesn't change during the session. WebXR applications typically scale objects in real-world sizes, not based on pixel counts.

The details get a bit complicated. XR views are used for WebGL drawing which doesn't have any text output operations. Applications can get an angular resolution estimate (pixels per degree) by calculating an overall field-of-view angle from the view's associated projectionMatrix combined with the corresponding viewport pixel size. This current angular resolution can change as a result of applying a framebufferScaleFactor or a dynamic viewport scale, but these changes are all fully under the application's control and won't change unless the application changes the scale actively. The actual angular resolution of a headset is fixed in hardware as a property of its lenses and physical display and doesn't change during an XR session, so there are no associated events. The application can get baseline values for scaling if needed by referring to the native WebGL framebuffer resolution via getNativeFramebufferScaleFactor.

For background, WebXR view objects basically correspond to rectangular subsections of the opaque framebuffer object associated with the current session's XRWebGLLayer. For example, a typical VR headset would split the framebuffer into one half for the left eye, and the other half for the right eye. The framebuffer is passed to the XR device for display, and device-specific code is then responsible for presenting the appropriate content to each eye.

When dynamic viewport scaling is in use, the application uses a smaller subsection of the overall framebuffer for rendering which is then scaled to fit the final view, effectively reducing resolution while keeping angular sizes constant. This is under the application's control. An application could choose to adjust font sizes in response to a scaling change, but this would likely appear jarring and unexpected.

[atanassov]

Thank you for the detailed explanation @klausw, this was exactly the kind of answer I was trying to find. Given that the rest of the reviewers are also satisfied with your proposal I'm closing this review. Thank you for working with TAG, we look forward to see more progress made with Immersive-Web WG.