Make use of depth extension in openxr backend

[jdm]

Right now we only pass gl::COLOR_BUFFER as the mask argument for blit_framebuffer to copy the texture data from the webgl texture to the d3d texture. Would including gl::DEPTH_BUFFER as well enable better image stability?

[Manishearth]

Potentially, if we enable the depth extension.

[jdm]

@paulrouget Could you try building https://github.com/jdm/servo/tree/depthkhr and see if there's an noticeable improvement with respect to image stability or jitteriness on any of the more complex demos? This is my attempt to implement https://github.com/microsoft/OpenXR-SDK-VisualStudio/#submit-depth-buffer-for-projection-layers, and it's made of the changes to https://github.com/jdm/webxr/tree/depthkhr and https://github.com/servo/openxrs/tree/depthkhr.

[paulrouget]

error[E0432]: unresolved import `webxr_api::SelectKind`
  --> components\script\dom\xrsession.rs:50:71
   |
50 |     self, EnvironmentBlendMode, Event as XREvent, Frame, SelectEvent, SelectKind, Session,
   |                                                                       ^^^^^^^^^^ no `SelectKind` in the root

error[E0023]: this pattern has 4 fields, but the corresponding tuple variant has 3 fields
   --> components\script\dom\xrsession.rs:222:13
    |
222 |             XREvent::Select(input, kind, ty, frame) => {
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected 3 fields, found 4

error: aborting due to 2 previous errors

Some errors have detailed explanations: E0023, E0432.
For more information about an error, try `rustc --explain E0023`.
error: could not compile `script`.

To learn more, run the command again with --verbose.
Build FAILED in 0:18:44

[paulrouget]

Hm, I needed the proper branch of webxr too.

[paulrouget]

I can't really see any difference. Do you have a demo in mind that has some significant jittering?

By image stability, you are talking about the issue with the content lagging a bit behind?

[jdm]

@paulrouget Oops, I forgot to enable the appropriate extension. See if there's any difference if you add khr_composition_layer_depth: true to this code.

[jdm]

Ok, with the extension enabled the emulator is no longer drawing the background of the babylon colour picker demo in immersive mode. I wonder if some of the depth/z values at https://github.com/jdm/webxr/commit/61096f734d7736a11a871e3c0c2334e545f3ea24#diff-f3cfd7c22eb8b9fb4aeefdabdb68f7e4R621-R625 is causing it to be ignored?

[jdm]

We're going to need to do something a bit different here. We will need to:

• obtain the depth attachment from the webgl framebuffer
• create a separate d3d texture with an appropriate depth format
• blit the depth buffer between them
• attach the d3d texture to the composition project layer chain

[paulrouget]

For the background issue, be aware that that new version of the color picker demo uses AR mode now.

[paulrouget]

See if there's any difference if you add khr_composition_layer_depth: true

It's a lot more choppy. And it feels like there's tearing (top part moving ahead of the bottom part).

[jdm]

https://github.com/servo/webxr/issues/92#issuecomment-564223299 seems like it was made more challenging by the changes to wrap the openxr swapchain in the EGL surface. There is a separate depth swapchain created in the sample code.

[jdm]

So eglCreatePbufferFromClientBuffer does allocate a depth buffer if requested, so the challenge is extracting that data and getting it into the d3d swapchain. Some potential options include:

• glReadPixels
  1. map the d3d texture so its data is writable by the cpu
  2. use readpixels to copy the depth information into the texture
• PixelBufferObject + shader
• ANGLE_get_image extension
• glBlitFramebuffer/BlitFramebufferANGLE

[jdm]

It would be nice if this copying could occur as part of the render_animation_frame method in the openxr backend, but because of the limitation that the EGL surface wrapping the d3d texture (and therefore its associated depth buffer) is only usable from the webgl thread it requires some more hacks instead.