Consistent presentation/rendering of model resources

[DRx3D]

I agree that it is very good to make it easy for people to display 3D content in a web page. I completely disagree with the methods and processes described in this proposal to make it an HTML element. HTML elements need to be fully defined so that they can be similarly implemented across browsers and reflect what people would see in applications outside of browsers. The process of rendering a high-quality model requires proper handling and rendering of the model's geometry, appearance, animation, and interaction.

My knowledge is in glTF (and glTF binary) so these comments may or may not reflect on the capabilities of USDZ. I will address the topics as separate issues: Appearance and Animation / interactivity; with respect to 3D models in glTF format. Static geometry is pretty straight-forward and not subject to much interpretation.

The really difficult part is appearance. The document states that "it is impractical to define a pixel accurate rendering..." for models. However, this is really important. Khronos has done extensive work in the 3D Commerce Working Group towards pixel accurate rendering across multiple 3D viewers (https://www.khronos.org/3dcommerce/certification/). The accuracy was demanded by retailers so their products would appear visually identical across different web sites. There were so many factors that mattered in producing acceptable renderings that include lighting, rendering calculations (including equation approximations), conversion from GPU to display, and tone mapping.

The component that caused the most issues and difficulties is lighting. A model built for physically-based rendering looks best in a complex lighting environment. This is usually done with image based lighting, but punctual plus area lights will also work. The statement that "A future version ... will describe the lighting model and environment .... Both items will require community collaboration and some consensus." makes the process sound much easier that Khronos found it to be.

Some issues that came from the Certification work. Note that the Certification program did not solve all of these in the initial release.

• Is lighting done as an 8-bit RGB or 16-bit HDR image
• Is lighting done with many point and area lights?
• How does the content creator provide for different lighting?
• How does the user adjust the lighting to match a particular environment?
• What background is used for the model display?
• How is the (floating-point) rendering converted to an 8-bit RGB display?
• How is the rendering adjusted depending on user environment?
• How is the rendering adjusted depending on device (hardware, OS, browser)
• How is time-dependent display degradation handled?

It may be possible to construct an initial release without resolving all of these items.

[cabanier]

The really difficult part is appearance. The document states that "it is impractical to define a pixel accurate rendering..." for models.

When I drag a regular browser window from one monitor to another, the pixels look different. So even 2D web browsing is suffering from a lack of pixel accurate rendering. I believe browsers and the CSS WG are still working on making this a reality.

Given this and the state of VR/AR headsets I think it's reasonable not to require that models are rendered identically across UAs. Moreover, we want to give systems the ability to have these models become part of the native UI so they can have a similare look and participate in receiving/generating shadows and reflections. On headsets with limited resources, UAs should be allowed to ignore certain materials.

[toji]

(Late reply to this, but it seems like the best place to capture what I've been thinking about lately.)

I agree with Rik that requiring pixel-exact presentation between browsers/platforms is probably not realistic or even useful. There's a LOT of wiggle room in the rendering of common PBR materials that allows for minor differences in the final pixel values while still being "correct". This is especially true when discussing the difference between mobile and desktop systems, where it's generally seen as acceptable to take shortcuts when rendering on less powerful GPUs for the sake of performance or battery as long as the fundamentals of the materials and models are being respected. (As an extreme example: We may not want to forbid high-end systems from using ray tracing, but we definitely don't want to require mobile systems to do so.)

However that only really applies to the shading of the model. We should absolutely go out of our way to ensure that the contents of the model (the mesh, animations, textures, etc) can be loaded and presented consistently across all browsers. It will not be acceptable to most users if a model loaded on one platform/browser completely lacks some aspect of its appearance or structure that is present on another platform/browser. (As another extreme example: I wouldn't view it as acceptable for a model to use a hair simulation system that wasn't available universally, as that fundamentally changes the way the model is displayed.)

Falling back to analogies to HTML/CSS, it's probably acceptable if the background color of a div is a slightly different shade of blue on device A vs. device B. (This could come down to differences in the display that have nothing to do with the browser.) But it's a lot more problematic if the div is positioned or sized wildly differently across browsers.

Exactly where that line gets drawn is likely to be a topic of debate, but I hope we can come to an agreement on a couple of high-level points:

• Whatever formats we use must be well specced and the spec must be public. Using an unspecified format or an opaque proprietary one opens the door for more differences in display and behavior than I think anybody wants.
• Similarly, what parts of the format the <model> element will support should be evaluated separately and deviations (if any are required) should be specified.
• It seems like any of the formats being considered support "extensions" of various sorts, and supporting extensible behavior is good for the APIs longevity. Any extensions that are used when loading/rendering should be tightly controlled, though.
  • At the very least developers should be able to query which extensions are available, and it may be ideal to have them opt-in to any given extension being active for a particular <model> element.
  • It's probably a good idea for the group to allow extensions individually so that things like cross-platform implementability can be evaluated, rather than allowing browsers to enable support for an extension by default on a whim.

[klausw]

Can we please also add a requirement for clearly specified dimensions corresponding to a real-world metric? A popular AR model use case is to view furniture or other items in your environment, and that fails dismally if they get rescaled unpredictably. If I remember right, many 3D file formats use arbitrary or underspecified units.

(Of course it should be possible to support resizable models, but I think it needs to be possible to opt out of resizing to get a consistent result.)

[cabanier]

Can we please also add a requirement for clearly specified dimensions corresponding to a real-world metric? A popular AR model use case is to view furniture or other items in your environment, and that fails dismally if they get rescaled unpredictably. If I remember right, many 3D file formats use arbitrary or underspecified units.

If the model is displayed by itself (like in AR quicklook), I agree that there should be a way to set the actual dimensions. For models that are part of the page, I'm not so sure if this is desirable. Today, users can control the size of elements with user zoom or pinch zoom and it would be strange if those didn't affect the size of the model.