Better integration with gfx-hal

[zmerp]

Hi blaind, I'm really interested in your work since I wanted to make ALVR switch its graphics code and VR runtime to bevy and OpenXR. I saw your PRs and I wanted to give you some suggestions and and also a hand with the implementations. About gfx-hal, I think the OpenXR integration should be made into another crate. Currently there is no way of accessing gfx-hal internals, so I wanted to contribute an API to get backend-specific handles (like VkInstance and VkDevice) and to create gfx-hal objects from these handles. About wgpu, a similar API could be propagated, but I don't know if this is the right call, since the point of wgpu is to have a safe API. The best thing would be for bevy to switch from wgpu to gfx-hal directly, but this easier said than done.

[blaind]

Hi @zarik5 - thank you for the message. ALVR seems like a great project! Regarding this PoC of xr on bevy, all help is welcome :)

I saw your message in gfx-issue, and had a bit of thought already. Some initial pointers:

• taking the openxr-part from gfx-rs could be doable. Its mostly self-contained anyways
  • if possible, communication could be done through raw pointers. This would also encapsulate the openxr implementation, I like the idea
• there's a few parts that'd require tinkering and/or support from gfx, though:
  • multiview (link)
  • supplying xr-required api version (I don't remember if this is really needed, though, have to double check): link)
  • getting a list of xr-enabled devices (link). This is something that's done at wgpu-side
  • on xr, setting only a subset of queue families link

There might also be some openxr-reconfiguration cases which I haven't hit yet. Also, my pull request for gfx support only vulkan currently, but openxr supports opengl too.

wgpu-rs and wgpu crates are ~34k lines of code in total, I guess bevy benefits quite a bit of the graphics handling done here. Moving away from wgpu might not be feasible...

My PoC uses 100% the rendering done by bevy (using wgpu), only change is the render target - whereas in non-xr environment the target is a swapchain supplied by wgpu, in XR environment I'm creating a xr-specific swapchain.

If wgpu is used, then the challenge of moving raw handles between bevy <-> wgpu-rs <-> wgpu <-> gfx-rs will persist. So far, I managed to get away with a solution where the handles need to be moved only in initialization, not during rendering. Not sure if that covers all the cases, though. Current solution is explained here

[zmerp]

Thanks for the reply. I agree that cutting out wgpu is not feasible. I don't have much time (because of school) but I'll try to work on gfx-hal, starting from exposing the raw handles. I'll be working here. If your PR on wgpu gets dismissed I'll propose to expose the handles also there.

supplying xr-required api version (I don't remember if this is really needed, though, have to double check): link)

The example on openxrs uses Vulkan version 1.1.0. I don't think there is any restriction on this.

[zmerp]

Looking more into wgpu I think there is a clean way of implementing this. Since wpgu-core is advertised as not usable by third-party crates, it should allow exposing low level details from gfx-hal. In contrary wpgu-rs API should remain safe and cross platform. wgpu-rs has two backends: wgpu-core and web native. So I think it would make sense to create at this level a new XR abstraction layer that uses OpenXR or WebXR depending on the target. I plan to port ALVR client app to web so I would be glad to contribute the WebXR backend (when I have the time). While WebXR must be implemented at wgpu-rs level, OpenXR could be implemented in wgpu-core, but I don't think it is a good idea. wpgu-core is used also by wgpu-native (for WebGPU in Firefox) which is only a graphics API. Implementing OpenXR in wgpu-core means also that the XR API needs to be wrapped one extra time. Depending on what feedback we get we might consider creating a wgpu-xr crate (that sits between wgpu-core and wgpu-rs), so an extra abstraction layer might be needed. And at that point I could work on a wgpu-codec crate to wrap WebCodecs and native decoding APIs

[blaind]

Few thoughts:

• here's a good explanation of gfx-rs architecture: https://gfx-rs.github.io/2020/11/16/big-picture.html
• currently, there's quite a bit of interplay between bevy and gfx in my poc, abstraction is no easy feat (but maybe doable)
• OpenXR contains rendering, but also a input/output and even external hardware device layer, which might go outside of gfx scope. Some explanation at https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html#input
• one easiest and maybe most beneficial part for abstraction could be the swapchain handling (see https://github.com/blaind/bevy/blob/6226677b9a60623d9b0c00f4dfc4eca80f62ddb8/crates/bevy_openxr_core/src/swapchain.rs#L39) - since that made bevy part a bit more complex - if wgpu handled that, that'd be awesome
  • however, there must be hooks in between (preparation of swapchain/rendering, final rendering, copying to textures) for some layer (app? bevy?) to handle i/o
• (edit: I have no experience in webxr, do not really know how it fits into the picture)

(@ https://github.com/gfx-rs/gfx/pull/3761)

[zmerp]

here's a good explanation of gfx-rs architecture: https://gfx-rs.github.io/2020/11/16/big-picture.html

Thank you for the link!

(edit: I have no experience in webxr, do not really know how it fits into the picture)

If we can break the wgpu-rs abstraction with backend specific details (as proposed here), I think the better way of handling this would be outside wgpu-rs. This means that WebXR can be ignored for now, and OpenXR can be fully implemented inside bevy_openxr_core.

I'm working on gfx-hal, I hope to have something to show soon. My goal is to expose what you need in gfx-hal without implementing anything xr specific in it.

[zmerp]

My development plan is to first finish gfx-hal low level interop with Vulkan and test it modifying the vulkan example in openxrs by replacing vulkan with gfx-hal. After the gfx-hal API is complete I will work on wpgu-core and then on wpgu-rs, modifing the same example for testing. Probably this will take a lot of time. @blaind If you want to work one the same parts we can coordinate on Discord (I'm zarik5#6062).

[blaind]

That sounds a really great approach, will send a message on Discord