Runtime backend selection and out of tree backends

i509VCB commented 1 year ago

Starting with #3051, there has been work to make wgpu able to select the backend at runtime on web and

The webgpu and webgl selection already has an issue here: https://github.com/gfx-rs/wgpu/issues/2804

One proposal for the new API is allowing out of tree implementations of wgpu. This is related to the desire to have wgpu work in environments where the APIs are not open such as consoles.

There are also some issues that need to be ironed out for both web runtime selection and out of tree backends to be done:

Figure out how to deal with feature guarded APIs.

The APIs like Instance::from_hal, Instance::from_core, Instance::as_hal, Surface::create_surface_from_core_animation_layer, Surface::create_surface_from_canvas all are only available with specific backends at compile time. Having the ability to select backends at runtime or introduce out of tree backends means these APIs are a bit out of place as whether these will succeed is not guaranteed but now determined at runtime.

Also for cases like Surface::create_surface_from_offscreen_canvas, the backend itself matters at runtime when calling this. If the backend is the native backend, this call makes no sense. If we wanted to have an Instance represent multiple backends transparently, then we need to figure this issue out.

How should multiple backends coexisting at runtime interact, if at all?

On web, there would be two backends. One for webgpu and one for webgl. The webgpu backend is simply just a thin layer to the API exposed by wasm-bindgen. The webgl API on the other hand involves using wgpu-core on web.

Say the application wants to use either webgpu or webgl at runtime. Should the application need to try and create two instances? Or would one instance suffice? If one instance were to suffice, we would need to have Instance and Surface represent several instances and surfaces, much akin to how the Vulkan loader manages ICDs.

If this was not desired, then a way to test if the Instance was even created in the first place needs to be available. Instance::new cannot fail at the moment. This would need to change in that case.

For out of tree backends, how should wgpu be able to create a `wgpu::Instance` from a `Context`

I'd think it would be easiest to just have a special constructor for that.

Should users of the `wgpu` API be able to access the `Id` and `Data` associated types for each type of object?

This might be useful for some cases where wgpu would be used in contexts beyond what the high level wgpu supports. I am almost certain that console backends could not use raw-window-handle. So this would need to be available for creating a Surface on consoles.

This is also a possible solution to the feature guarded APIs.

Removing the `gfx_select!` macro for the native backend, instead using dynamic dispatch for each backend

@cwfitzgerald mentioned this on matrix, but I don't know where I'd start with this. It's also a longer term issue that doesn't need to be immediately addressed to resolve this issue.

How this would be done would also be directly related to multiple backends existing at the same time.

daxpedda commented 1 year ago

How should multiple backends coexisting at runtime interact, if at all?

On web, there would be two backends. One for webgpu and one for webgl. The webgpu backend is simply just a thin layer to the API exposed by wasm-bindgen. The webgl API on the other hand involves using wgpu-core on web.

Say the application wants to use either webgpu or webgl at runtime. Should the application need to try and create two instances? Or would one instance suffice? If one instance were to suffice, we would need to have Instance and Surface represent several instances and surfaces, much akin to how the Vulkan loader manages ICDs.

If this was not desired, then a way to test if the Instance was even created in the first place needs to be available. Instance::new cannot fail at the moment. This would need to change in that case.

So I've been thinking on how to tackle this.

Personally I'm in favor of one instance tackling both WebGPU and WebGL. Wgpu already has an interface to select a backend, first through Instance::new() and then through Instance::enumerate_adapters(). So I would argue that a WebGPU adapter should be added to the list returned by enumerate_adapters(). Instance::request_adapter() should also work as it does right now, probably prioritizing WebGPU if found.

My suggestion currently looks like this:

Make Instance hold both, a WebContext and DirectContext.
Instance methods need to handle both at the same time, e.g. enumerate_adapters() needs to return a WebGPU adapter.
Surface needs to be able to hold both, a WebContext and DirectContext Surface. Again, methods have to handle both Contexts. This is problematic because of get_current_texture(). We could address this by requiring a Surface to be configureed first, which then settles which Context to use.
Types derived from Instance can figure out which Context to use from the Adapter or Device.

I guess this was kinda the obvious part.

My proposal would be to implement this first, to enable WebGPU + WebGL and tackle the rest at some later point: out-of-tree backends and using dynamic dispatch for each native backend.

My 2 cents on the rest:

Probably the most important question is how do we hold multiple Contexts per Instance. This is probably where DynContext comes in and how we could just make a Vec of Contexts.

Removing the gfx_select! macro for the native backend, instead using dynamic dispatch for each backend

This would also tie in nicely with the Vec of Contexts. But this can be done later.

Figure out how to deal with feature guarded APIs.

The APIs like Instance::from_hal, Instance::from_core, Instance::as_hal, Surface::create_surface_from_core_animation_layer, Surface::create_surface_from_canvas all are only available with specific backends at compile time. Having the ability to select backends at runtime or introduce out of tree backends means these APIs are a bit out of place as whether these will succeed is not guaranteed but now determined at runtime.

Also for cases like Surface::create_surface_from_offscreen_canvas, the backend itself matters at runtime when calling this. If the backend is the native backend, this call makes no sense. If we wanted to have an Instance represent multiple backends transparently, then we need to figure this issue out.

I'm really not sure how this would play well with out-of-tree backends, they might also want to add their own methods. The easy idea that comes to mind would be to add a get_context() function or something that can get and cast the desired Context. Sounds awful.

Which kinda leads me to my alternative idea: make Instance only hold a single Context. Multiple Contexts have to be handled through a wrapper.

I'm not really familiar with the Wgpu codebase, apologies if some of my thoughts here are naive.

junglie85 commented 5 months ago

One approach to handling the selection of backend and allowing arbitrary methods based on the backend could be to use Inline Dyn Extension Traits (IDETs). Winit is considering something similar to this for their next API iteration for a number of reasons, including support for out of tree backends.

gfx-rs / wgpu