Smithay support in OpenGL backend

[skyne98]

Hello there!

At the moment gfx-backend-gl seems to be very closely related to glutin, which makes providing different OpenGL backends for it very hard (if not even impossible). Coming from https://github.com/Smithay/smithay/issues/129 there is an idea to abstract away the need for glutin behind the trait making it possible for other projects to utilize gfx-hal while being in their own specific contexts (as described by https://github.com/Smithay/smithay/issues/129#issuecomment-484464241 and https://github.com/Smithay/smithay/issues/129#issuecomment-459541840).

At the moment this will allow Smithay to implement such a backend creating an entrypoint into the world of gfx-hal rendering (e.g. rendy, amethyst, ggez, gfx-webrender) while the other, more complex backends, are in the works (e.g. vulkan). I personally think that Smithay can become a great foundation for people willing to make not only fast and pretty desktop experiences, but also games and software that doesn't rely on compositors.

Thank you for your time!

[msiglreith]

Hi 👋

that sounds like an interesting project! It's a bit difficult for me to grasp the scope of smithay atm. Therefore, I would quickly give an overview about the window system interface which would be the connection between gfx and smithay I believe. This mostly follows https://github.com/gfx-rs/gfx/blob/master/src/backend/gl/src/window/glutin.rs

Ideally, we would have an headless context acting as Instance which allows us to query basic information about the system (memory, capabilities, ..), additionally a Surface acting as software abstraction of the OS/Hardware/Display/Compositor(?) providing information about these. The surface can be then used to instantiate a Swapchain which will do the actual presentation handling (providing images to draw to, compositing, displaying stuff on the monitor, ..).

From the API side the best case would be that only the surface creation will be platform dependent and all the other parts completely opaque and platform agnostic. Meaning, some kind of create_surface_from_smithay function and the rest is done internally. But we are still far from this point even with glutin under the hood.

Given this as current basis, I would propose to start with a similar implementation as https://github.com/gfx-rs/gfx/blob/master/src/backend/gl/src/window/glutin.rs and using a feature switch for selecting the implementation. This would probably a good point to better understand the interface and requirements/possibilities from both sides!

[elinorbgr]

Hi,

To give some more context, smithay is already integrated with glium as a drawing API. This was pretty easy because glium is generic over its OpenGL provider.

Indeed, one just have to provide implementations of the glium::backend::Backend and glium::backend::Facade traits to use glium over some provided OpenGL context, which is what smithay currently does.

Following this would it make sense for gfx-backend-glx to do something similar? It could expose some unsafe GLProvider trait that abstracts all that the crate needs to use, and then have a glutin cargo feature that provides implementations of this trait and adapter for use with glutin, and external projects like smithay could hook into gfx-rs by providing an implementation of this trait.

There could then be a create_surface_from_custom<T: GLProvider>(...) method that would allow the link to be done.

Would that make sense? I have the feeling that integrating smithay-specific code into gfx would be a weird thing to do tbh.

[kvark]

I'm just reading about this issue and getting totally confused (as often happens...).

Is your concern that gfx-backend-gl is tied to glutin? This is totally not by design, it's just the state of things at the moment. The design is for GL backend to have optional features for window/context providers, e.g. glutin, sdl, glfw, etc. It's just that nobody cared enough yet to add anything but glutin. If smithay provides GL context, than this is the path to go, and @msiglreith explained this in good detail.

I don't understand why there is any mention (here and in https://github.com/Smithay/smithay/issues/129) of having another GL backend. Is smithay not compatible with "gl-rs"? Could we have it supported by glow (cc @grovesNL ) ? Duplicating the backend code is a big concern to me - it has a lot of logic, and fixes are coming all the time.

[msiglreith]

To give a bit more details about how GL contexts are embedded in our GL backend:

glutin can provide to 'flavors' of GL contexts: headless and tied to a window. The headless context backs the Instance and would mostly be provided be the IHV drivers I guess. smithay targets more the 2nd context type if I understand it correctly, but doesn't provide any interface for acquiring the first type (?). The window-tied contexts would be associated with a surface/swapchain and provide the color targets for presentation.

Based on my assumptions, I would argue that glutin might be necessary for providing the first context type. This doesn't exclude smithay support by any means.

There could then be a create_surface_from_custom(...) method that would allow the link to be done. Would that make sense? I have the feeling that integrating smithay-specific code into gfx would be a weird thing to do tbh.

I'm a bit hesitant to provide wrapper around these APIs, at least at the current point. I'm still struggling to see the bigger picture or can imagine how the implementation will look in detail. smithay may provide an API which resembles our needs better than glutin (actually quite likely). Therefore, it would be interesting to see if we gain something from direct access compared to providing a thin GL specific implementation layer for Surface/Swapchain/Instance.

[elinorbgr]

Okay, I think I have not been clear enough, probably because I didn't to use the term "backend" in the same way as it's user in gfx, sorry for that.

So, IIUC gfx-backend-gl is meant to support several "window / context provider", and right now glutin is just the only one that happens to have been implemented. That's pretty great :+1: .

Now, I guess the question is that there is basically 2 ways to achieve that:

• either gfx-backend-gl includes specific code for each possible provider (glutin / sdl / glfw / etc...)
• or gfx-backend-gl exposes a trait ContextProvider that abstracts away its needs from a GL context provider, and maybe also provides implementations of this trait for common providers (glutin / sdl / glfw)

My initial instinct was to go the second route, mostly because this is what glium already does and how we integrated it with smithay. And apparently your roadmap is to go with the first route.

My main reason to personally favor the second one is that, given gfx is a much more general-purpose project than smithay, I fear it would only cause issues to include smithay-specific code into gfx. Smithay needs to initialize an EGL context in a specific way, given it is tied to DRM/GBM and needs to import buffer from wayland client apps. So this is just much more convenient for us to be our own context provider, rather than rely on glutin for example.

Smithay being also currently very WIP, a lot of its internals and public API are likely going to change, meaning having smithay-specific code in gfx will be a pain to maintain for everyone. If gfx-backend-gl exposed a generic API for "a window/context provider", smithay would be able to implement it and the glue code would instead live in smithay's codebase where it'd be easier to maintain. And again, I believe this would make sense given smithay is a pretty niche project compared to gfx-rs.

[skyne98]

I can agree with @vberger. Having smithay-specific code in GFX sounds very hard to maintain. Each and every change in smithay tied to this topic will mean pull requests to GFX. Also, giving an ability to "plug into" GFX's GL backend might appear useful for some of the other niche projects.

[kvark]

I don't think having a trait in the GL backend for context creation would do any harm. Note that this wouldn't reduce any complexity of GL backend (implementations for glutin/sdl/etc would still live there). The benefits are outlined rather clearly, however there are downsides too: GL backend is going to be changing versions rather often, so smithay would have trouble updating to each and every new GL backend crate if it wants to implement a trait from it...

[skyne98]

I think there can be a smithay-gfx-gl crate which handles this and doesn't require smithay to update the gfx version constantly.
Or maybe something like gfx-backend-smithay-gl, but as a part of smithay.

[elinorbgr]

Do I understand correctly that this gfx-backend-smithay-gl would basically be a fork of gfx-backend-gl replacing glutin by smithay?

[skyne98]

I think that there are a couple of routes this cooperation can go:

1. gfx-backend-gl exposes a trait and Smithay's trait implementation lives in the Smithay's codebase. The problem with this approach that @kvark mentioned is that Smithay will have to keep up with rapid GL crate API changes.
2. Smithay GL trait implementation is included in the GFX's codebase. This overall seems like the hardest to maintain solution, considering the project (Smithay itself) is WIP, has changing APIs and is rather niche. However, in my opinion, GFX itself can benefit from a backend that can output directly to the framebuffer.
3. gfx-backend-gl is forked as gfx-backend-smithay-gl and is maintained as a part of Smithay's codebase. This route seems like the easiest route to start with, but can potentially bring issues in the future. Also, this kind of solution won't really benefit GFX's GL backend as it won't receive any changes to make it more abstract.

I think that the second option has the most potential. At later stages, it should be possible to merge this implementation with the GFX's codebase, when everything is much more mature.

Also, what about #2554? It should have made a split between OpenGL and WebGL during its development.

[Drakulix]

I there, I am currently responsible for most of smithay's backend code, let me just leave some thoughts here quickly.

I don't think having a trait in the GL backend for context creation would do any harm. Note that this wouldn't reduce any complexity of GL backend (implementations for glutin/sdl/etc would still live there). The benefits are outlined rather clearly, however there are downsides too: GL backend is going to be changing versions rather often, so smithay would have trouble updating to each and every new GL backend crate if it wants to implement a trait from it...

smithay is a rather fast moving target, I have no problems with frequent updates (right now, you would need to build against smithay master anyway to get it's latest features).

On the other side, maintaining a fork of gfx-backend-gl sounds good for prototyping this, but not for maintenance, because I expect gfx to work on this much more frequently.

In contrast, I do not expect the backend trait to be touched very frequently. It has been stable in glium / smithay for a long time and has served very well for different use-cases. I think the abstraction is solid and does not require any major reworks, once it lands in any shape or form in gfx.

I don't understand why there is any mention (here and in Smithay/smithay#129) of having another GL backend. Is smithay not compatible with "gl-rs"? Could we have it supported by glow (cc @grovesNL ) ? Duplicating the backend code is a big concern to me - it has a lot of logic, and fixes are coming all the time.

Smithay is compatible with gl-rs, but we would like to enable people to use any graphics library they prefer. That is why we do support a glium-Integration right now and we would like to do the same for gfx. The problem is always the same, smithay needs a way to provide the context.

[msiglreith]

@vberger @Drakulix Thanks for the discussion so far! I'm currently looking into reworking the OpenGL WSI parts to match the desired API exposed by Vulkan (Instance <-> Device, Surface <-> Swapchain). It turns out that this seems to be quite difficult to do with 'classic' Wayland + EGL: Instances and Devices (with their underlying contexts) are created out of thin air without any backing display, at the same time each swapchain should have a context (or at least a way to present images) which shares its image resources with the device context. The swapchain is created from a surface with a wl_display and wl_surface. This turns out to not work really well with something like mesa on this layer of abstraction.

That's why I'm wondering if there is a possibility to go a layer below to receive direct access to the image contents similar to the drivers of gpu vendors. But that's the point where my wayland knowledge starts to fade. It appears that smithay access the lower level drm interfaces? Would be something like that achieveable, also in a portable manner (taking any kind of wl_display from external, forward it into smithay, etc.)? I don't expect feature completeness (NVIDIA EGLStream support) or anything like that just trying to see where we could go with this approach.

[elinorbgr]

To go one layer below, the road would be to implement yourself the lower-level layer using the linux-dmabuf and linux-explicit-synchronization wayland protocol extensions. This is basically what Mesa does internally: it uses the wl_display provided to it to setup the protocol object it needs, and then submits the contents as dmabufs to the wl_surface provided to it. To reproduce this, you could create a headless context to do the rendering, then export the framebuffer contents as dmabuf that will then be shared with the display server. I believe there is an OpenGL extension to do that, but my knowledge of OpenGL is rather limited to be quite honest.

Smithay would hardly be a proper reference to achieve this, as it rather sits on the "display server" side of the graphics setup: it does indeed bind directly to the DRM interface (and thus requires to be run as a privileged program), and receives dmabufs from the various wayland clients to display them on the screen.