Multisampled multiview support

[kdashg]

We need to support multisampled multview rendering, but how to do so is tricky.

Since we are using texture arrays for multiview, there are two options:

• Multisampled framebuffers: OVR_multiview_multisampled_render_to_texture
• Multisampled texture arrays: EXT_multiview_texture_multisample

EXT_multiview_texture_multisample has a good brief on the two:

This is one of two extensions that allow multisampling when using OVR_multiview. Each supports one of the two different approaches to multisampling in OpenGL and OpenGL ES:

Core OpenGL and OpenGL ES 3.1+ have explicit support for multisample texture types, such as TEXTURE_2D_MULTISAMPLE. Applications can access the values of individual samples and can explicitly "resolve" the samples of each pixel down to a single color.

The extension EXT_multisampled_render_to_texture provides support for multisampled rendering to non-multisample texture types, such as TEXTURE_2D. The individual samples for each pixel are maintained internally by the implementation and can not be accessed directly by applications. These samples are eventually resolved implicitly to a single color for each pixel.

This extension supports the first multisampling style with multiview rendering; the OVR_multiview_multisampled_render_to_texture extension supports the second style. Note that support for one of these multiview extensions does not imply support for the other.

OVR_multiview_multisampled_render_to_texture was first-to-market, and is generally better supported on ES, particularly ES3.0. OVR_multiview_multisampled_render_to_texture is possible to implement on top of EXT_multisampled_render_to_texture, but not, strictly speaking, vice-versa.

The tricky part is the behavior of implicit resolve in OVR_multiview_multisampled_render_to_texture, as described in EXT_multisampled_render_to_texture:

When the texture level is used as a source or destination for any operation, the attachment is flushed, or when the attachment is broken, the GL implementation may discard the contents of the implicit multisample buffer. If the contents are discarded, the subsequent operations on the multisample buffer will behave as if all samples within a pixel have the value most recently written to the color buffer for that pixel.

The operations which may cause the contents of the implicit multisample buffer to be discarded include:

• Drawing with the texture bound to an active texture unit
• ReadPixels or CopyTex[Sub]Image* while the texture is attached to the framebuffer
• CopyTex[Sub]Image, Tex[Sub]Image, CompressedTex[Sub]Image* with the specified level as destination
• GenerateMipmap
• Flush or Finish while the texture is attached to the framebuffer
• BindFramebuffer while the texture is attached to the currently bound framebuffer.

Can we as the browser tell when this happens or prevent it? No and no:

2. Should there be a way for applications to query if the multisample buffer has been resolved - and therefore is undefined?

   This may be useful if the operations that cause the multisample buffer to be resolved is allowed to vary between implementations.

RESOLVED: No, for two reasons: 1) This extension aims to be backwards compatible with the IMG_multisampled_render_to_texture extension, which did not include such a query, and 2) Given the resolution of issue 3 this is not very useful as the application cannot control whether multisample information is preserved or not.

3. Should there be a way for applications to preserve the multisample buffer after a resolve?

This would be similar in spirit to the EGL_BUFFER_PRESERVED options in EGL 1.4. Applications could - at a performance and memory cost - choose to make the multisample buffer not undefined after a resolve.

RESOLVED: No. The purpose of this extension is to support multisampled rendering in a lightway manner. Preserving the multisample buffer goes against this intent.

Mid-render resolves cause shimmering artifacts between frames that are interrupted in different places. These artifacts can be acceptable for occasional frames, but it does violate portability of results. (It's these same artifacts that make it inviable to implement EXT_multiview_texture_multisample in terms of OVR_multiview_multisampled_render_to_texture)

It'll take some work in Firefox's WebGL implementation to ensure that we aren't incurring more implicit resolves than we have to, and even then, there will be times we can't protect from implicit resolves. (there are other OpenGL contexts rendering, after all!)

Can we use OVR_multiview_multisampled_render_to_texture to support multisampled multiview? Yes, but imperfectly, with important caveats. Do we have much of a choice? Probably not, unless we're willing to forgo support on a lot of (most GLES) devices.

@RafaelCintron @kenrussell @grorg

[Artyom17]

Honestly, I think the fact that OVR_multiview_multisampled_render_to_texture drops unfinished buffers is actually a good thing: it clearly will indicate abuse of mid-frame flushes and will force devs to design more optimal experiences. The way how it is implemented in https://www.khronos.org/registry/OpenGL/extensions/EXT/EXT_multisampled_render_to_texture.txt is harmful, since it hides such design flaws as mid-frame flushes and produces sub-par quality of rendering.

[kenrussell]

I think we should go down the route of building on top of OVR_multiview_multisampled_render_to_texture because it has the best likelihood of leaving the multisamples in chip memory on tiled architectures. Agree with @Artyom17 's assessment that eliminating shimmering may be something that developers will optimize for.

@jdarpinian spent a long time investigating and eliminating mid-frame resolves in Chrome on Android and, aside from poor application behavior, I'm optimistic that the auto-resolve solution can work well for VR/XR content in all web browsers.

[jure]

Can we update the status of this? Is OVR_multiview_multisampled_render_to_texture still the most appropriate option? What work needs to be done to move it forward? I'd be happy to help!

[kenrussell]

This has been discussed in the most recent WebGL conference call in the context of WebXR layers. @cabanier attended on behalf of Oculus and specifically requested EXT_multisampled_render_to_texture. He gave me early offline feedback that they tried using WebGL 2.0's built-in multisampling facilities and carefully called discardFramebuffer at the correct point in the rendering path as suggested by @jdashg, and achieved the same performance as EXT_multisampled_render_to_texture. This is just one data point.

It may still be necessary to expose OVR_multiview_multisampled_render_to_texture. Oculus or another WebXR vendor would be in the best position to try it in their browser, confirm it's needed, and demonstrate usage in either a standalone application or a framework like https://github.com/mrdoob/three.js .

[cabanier]

We already support OVR_multiview_multisampled_render_to_texture behind a flag and a private extension name (OCULUS_multiview) The approach I took with renderbuffer for the non-multiview case wouldn't work here because there is no array equivalent in renderbuffers so it does look like this extension is necessary. We should investigate how much performance we can gain by extending three.js

[kenrussell]

OK, please do try an experiment with Three.js. The performance gain should be substantial based on earlier experimentation by Olli Etuaho (if I remember correctly), so we could then move forward with this extension.