What are the goals and tradeoffs for the linux backend?

[jneem]

We've discussed the future of our linux backend some in the zulip, and it was also mentioned in #1945. I'm opening this so that we can have a more permanent record of the possibilities, and the various trade-offs involved.

Here are some possible goals:

1. it should be possible to compile a druid project so that it will run on both x11-only systems and wayland-only (no xwayland) systems
2. it should be possible to compile a druid project so that it will run on one kind of system but not the other (in order to minimize dependencies)
3. druid programs on linux should take a short(ish) time to compile
4. druid programs on linux should have no C dependencies, to help keep the build process simple
5. druid-shell should remain small enough to maintain (i.e. not include a re-implementation of all of gtk)

Here are some possible ways to have a linux backend:

1. we maintain a gtk backend, and nothing else
2. we switch to using gdk, but not gtk
3. we have separate x11 and wayland backends,
   1. and we require choosing one at compile time, or
   2. we support runtime detection

Option a satisfies goals 1 and 5. (It satisfies goal 1 because gtk does the backend detection for us). Option b might satisfy goals 1 and 5, and potentially also moves us towards goal 3. Option c is harder to evaluate ahead of time, particularly as far as goals 3 and 5 are concerned.

I'm not 100% sure if goal 4 can be satisfied in a reasonable way. For example, if we want to support opengl on x11 then I think we need to link to libX11. Maybe we can load it at runtime, but anyway there are other hard-to-get-rid-of dependencies, like xkbcommon and libinput.

Are there any options or goals that I missed?

[james-lawrence]

@cmyr, @jneem I have an interest in a wlroots backend and I am willing to put the development work into implementing it, if that makes the long term decision any easier. basically right now only thing stopping me from putting the effort in is SVG problems atm.

[jneem]

I think having a wayland backend (even a partial one) would certainly help in understanding the trade-offs involved. I'm a bit concerned that you wrote "wlroots" instead of wayland -- we don't need explicit support for gnome/kde/wlroots/smithay/..., do we?

By the way, I don't know the current state of #1498, but it might be worth taking a look...

[james-lawrence]

@jneem have to target something, if i'm doing the work its going to be wlroots as that is what I use on my machine. meanwhile gnome doesn't implement parts of the wayland protocol I want to personally use (layershell). regardless, any implementation I do will get druid to like the 90% mark with the other implementations and it'll mostly just be tire kicking to work out kinks.

in short: the core of the work will be generally transferable, there might just be extensions that wouldn't work on gnome. but they'd be opt in. its also important to note that some areas of wayland are still in flux due to variations in implementation details between the various compositors. for instance there was recently an issue with firefox regarding how GTK was reacting to events emitted by non-mutter based implementations.

[james-lawrence]

small update re: wayland support. as it stands today the wayland backend is fairly functional on top of wlroots. most issues are limited to a couple areas:

1. keyboard input - both X11 and Wayland backends have issues around keyboard input mostly mapping from the linux event to druids internal representation. on mutter backends keyboard input doesn't work at all, someone will need to investigate that. I hope it isn't anything problematic. =)
2. druid internal window api's. bunch of window level methods are not fully implemented in practice I don't know how big of a deal these are. but most should be 30 minutes of time to implement if someone hits an issue with them.

[mahkoh]

I'm not 100% sure if goal 4 can be satisfied in a reasonable way. For example, if we want to support opengl on x11 then I think we need to link to libX11.

In the context of druid, it is possible to use opengl without linking to any display-server-specific libraries. For X11:

1. Using the DRI3 X extension, determine the drm device used for rendering.
2. Create a surfaceless EGL context on the drm device.
3. Using libgbm, allocate a front and back-buffer on that device.
4. Import the buffers into EGL.
5. Import the buffers into the X server using DRI3.
6. Render to the EGL buffers as usual.
7. Present the front buffer using the present X extension.
8. Wait for a flip event from the X server.
9. Go to 6.
10. Upon window resize, go to 3.

Similarly for wayland.

Naturally it is impossible to avoid linking to C libraries since the the graphics drivers are written in C.

libinput

There is no reason to use libinput.

[jneem]

Thanks! Just to clarify, my previous comment was based on this page, which claims that "an OpenGL application on X Windows must use GLX, a standardized API, to set up a rendering context." So you're saying that this information is outdated, thanks to EGL?

[mahkoh]

Correct.

[xStrom]

I'll link the zulip discussion referenced earlier, as it contains interesting info.

[james-lawrence]

@xStrom reading through that thread I didn't find much of personal interest; very little technical reasoning beyond not wanting to maintain 3 backend's for a small user base; which while a reasonable conclusion doesn't give much provide a base for a path forward. With that in mind I've been meaning to drop my thoughts on the current situation for awhile and now is as good a time as any I suppose.

The biggest problem with druid from a backend standpoint is the API surface of the various api's is far too large. The large API surface is the result of a bunch of assumptions about required behaviors. @cmyr discusses this issue here

I have a set of patches that could start down a path to eliminate that assumption but I wasn't willing to put in the leg work to get them upstream you can see how I used this API here

Druid WindowConfig API Surface for window configuration is too large.

1. see the backend window api surface being too large.
2. the WindowConfig can be reduced down to a single method and a set of backends you could imagine replacing the current backend builder with a set of platform/agnostic module that implement the window modal druid uses today with its own configuration. such an api would also allow for community developed backends to exist outside of the druid core repos.

API backend paint surface (windowbuilder) assumptions that are not actually universal:

1. assumes titlebar.
2. assumes min/max
3. assumes all windows have menus.
4. assumes all windows have resources.
5. assumes all should know how to open a file / save a file.

all the above can be resolved by inverting the ownership model. titlebar, menu's should own a window and know how to talk to it etc.

druid-shell api surface issues:

• no druid-shell native dialog. druid relies a ton of the native platform abstractions and no one has bothered to implement them for linux. Today that might mean implementing the dbus portal protocols which iirc seem to have support from gnome and kde. some other DEs provide them as well which decouple druid from the underlying desktop environment. though I think the menu bar would be a sticking point here.

most of the outstanding backend issues in X11/Wayland today stem from input handling issues. the rest of work is about the above API surfaces being very window centric and far too large.

The good news is that most of the remaining work for linux X11/Wayland is general and if its fixed can be applied to both x11 and wayland backends.

finally the lackluster SVG support is also a very unfortunate situation I opened a PR on that topic as well. Had I known how poorly it performed I might never have looked at druid originally. One of those 'ah yes excellent they do support it and API looks reasonable. then once using it I ran into a ton of limitations. =)

[xStrom]

Those assumptions are universal to Windows, macOS, and GTK. If it looks like it's just way too much work to get it solved for raw X11/Wayland, then going with GTK instead might be the answer. I'm still holding out hope for the raw backends though.

Also, all of these assumed features are things that need to exist in Druid. Even if we trim down the druid-shell API to the lowest common denominator, we'll still need to implement a solution in druid. It might make sense if our goal is to have druid-shell more decoupled, but it doesn't actually reduce work. So I'm not sure how big the benefit is for the Druid project as a whole.

Regarding the inversion of ownership, I'm having a bit of trouble understanding. So a file save dialog would own the window? It would be multiple-owner situation, where the window is both owned by the save dialog, the menu, and the titlebar?

[james-lawrence]

but they are not universal for UIs. they aren't even universal to window based UIs. my point was the abstractions in druid are causing an impedance mismatch, and the large surface area of those APIs are the source of most of your complaints about the linux. Though most of those are maybe 1-2 hour tasks to fix up; at least for the wayland backend I implemented.

and stop thinking of them as windows. they're all just surfaces onto which you are painting your UI components that together when composed make the dialog/server side decorated window/client side decorated. whether you have a platform specific implementation for windows or use a druid native implemention is fairly orthogonal.

Take a look at my code using the layershell wayland extension I implemented as a source of ideas. replace 'layershell' with 'window' and you could see how it can work without all the window api surface baggage being required for every platform. you can still have your window based abstraction with titlebars and everything; but they should be built on top of the backends. implementing all the missing pieces for window based linux is fairly trivial (particularly in wayland). just no one has had the need or desire to finish polishing it up. my previous comment mentions the things you'd need to pull in to do it well that wouldn't tie you to GTK or KDE given the current API surface for a window.

[jneem]

Just to be sure I've understood you, are you saying that druid-shell shouldn't support platform-native window decorations, menus, and dialogs?

[james-lawrence]

nope. i'm saying the current abstraction is causing an impedance mismatch and leads to problems when you don't use windows/macosx ideas of application windows. right now that pedigree is forced onto everything in druid. it doesn't need to be.

I'm attempting to explain this using druid terminology but that's really just confusing the issue because the abstraction is getting in the way of the conversation. =)

fundamentally right now the complaint is linux is hard to support properly with things like menus and dialogs; and that's true. no matter what path you pick on linux its going to non-native somewhere. because linux isn't a ui toolkit its a kernel.

current druid is purely a floating window UI framework, it doesn't need to make that decision. floating windows traditionally have a titlebar with a textual title and window sizing icons, and a menu bar.

which is fine, but X11* and Wayland are not floating window toolkits. they are compositing protocols.

Gnome (GTK) and KDE are floating window UI toolkits. but no matter which you pick druid applications will look non-native somewhere for someone on linux.

So where does that leave druid & linux? well there are a few options:

1. ignore the problem entirely
2. pick one option and go with it; warts and all.
3. attempt to support all the options on linux (good luck!)
4. reduce the required API scope for the druid UI toolkit and decouple floating windows from the general core API.

personally I prefer 4 and its why I keep pointing to my personal project for inspiration. please take time to look at the layershell initialization and compare it to the its floating window implementation (toplevel in wayland parlance). both are configured entirely differently but druid doesn't actually care... I was able to do all the normal things drop downs, context menus, etc just fine for both.

while I didn't implement File Dialogs or Menus because I didn't need them.... FileDialogs would have been fairly simple. Menus would have been a pain because there is nothing you can do to make them look native for everyone.

in either case druid doesn't need to have file dialogs or menus on the backend implementations. and my cursory glance at the code leads me to believe if you invert the dependency such that FileDialogs understand their backend but the backend knows nothing about the FileDialog things will work much more smoothly and be easier to support. same with menus.

[james-lawrence]

anyways I'm tapping out on this conversation this format isn't a great way to communicate ideas between individuals and takes more time than I want to spend.

[xStrom]

Window titlebars are extremely common on Windows, macOS, and in my experience also on Linux. It should probably be possible to create a Druid window without one, but the common case remains with one. So the idea that the Druid project doesn't provide titlebars is a complete no-go. Thus in my mind the only question is how the titlebars are implemented.

On the one hand we have the GTK option, which provides us with titlebars. Yes they're not going to perfectly fit in with every Linux desktop configuration. That's just the reality of desktop Linux.

However this is actually positive news for Wayland, because the titlebar implementation for Wayland doesn't have to compete with Windows or macOS. It just has to compete with the GTK option. Now I'm not very familiar with Wayland, but I can imagine some solutions here. There is no Wayland default titlebar implementation, right? Well, maybe there is a way to ask the actual compositor to provide its titlebar implementation? That way we could get a familiar looking titlebar in at least some configurations. If this isn't possible either, then it sounds like the titlebar's look just isn't important for Linux and we can draw whatever we want and it'll be as the Wayland designers envisioned.

As for the API scope, I'm afraid it hasn't even reached its final form. There are even more things to consider than what has been directly mentioned in this thread thus far. For example IME is a big one. We should probably have at least a basic understanding of what it will take to get IME working with raw Wayland vs just using GTK.

[jneem]

I agree with much of what James said, in particular the part about linux not really having a native toolkit (and I think some wayland compositors provide titlebars, but I don't think any provide menus). I definitely think that on any x11/wayland backend not using gtk, we'd want to implement certain parts as polyfills in druid, not druid-shell. (I think @sjoshid is working on this aspect of menus.) But on the other hand, I think that druid-shell needs a menu API because otherwise we can't use native menus on platforms that have them. All we need is some way for druid-shell to indicate that the menu API isn't implemented for a particular backend, and then druid should use the polyfill.

[univerz]

As for the API scope, I'm afraid it hasn't even reached its final form. There are even more things to consider than what has been directly mentioned in this thread thus far. For example IME is a big one. We should probably have at least a basic understanding of what it will take to get IME working with raw Wayland vs just using GTK.

looks like there was no problem to do it in winit. (maybe a good time to evaluate winit again)

menus etc

things has changed - druid is now competing with electron-react-like "html" frameworks that can work on any platform & it may not be a good idea to throttle progress with 90s concepts like standardized menu and file dialog. from userbase perspective - desktop is a dead platform for most use cases.

(yes, i work from wayland desktop :))

[james-lawrence]

@jneem you're correct some wayland compositors have server side decorations (titlebars/borders for our parlance atm). wlroots does, gnome doesn't, KDE I'm not sure. toplevel surfaces do have titles. but there are others that don't (like layershell).

@xStrom I'm not terribly familiar with IME; but my cursory glance is that its potentially a slightly different issue from the ones I'm pointing out. I believe I specifically called out that backend's should be dealing with two things:

1. putting pixels on screen
2. inputs.

IME falls into category inputs. titlebars / menus dont. they are visual components of the UI. Personally I believe they belong as standalone widgets

There might be some similarity between IME and FileDialogs w/ respect to showing a window for input selection maybe? at which point I recommend splitting the API into two parts. the backend support functions (if any; i suspect the current text api is actually sufficient...) that are needed to support IME and the IME UX components.

essentially Native Dialogs (File, IME, Print, etc), and Menus should be custom UI widgets built on top of backends; and druid needs to assume they may not exist. if they are application developer initiated (FileDialogs, Printing, etc) then the api should return a Result<FileDialogBuilder> or some such. and for linux stick to desktop portals

now these native UI components may reach into the backend to grab a reference to the native window handle (macosx/windows). that's okay. that's a single method the backend would need to support vs potentially multiple methods per native component.

also I believe IBus is the linux version of IME? someone please correct me if i'm wrong.

edit example toy api:

change WindowDesc to solely deal with surfaces (pick whatever name you want, avoiding Window due to its implicit assumptions) descriptions.

        druid::WindowDesc::new(|app, winhandle| {
            druid-shell::platform::wayland::layershell::Builder::new(app, winhandle, widget)
        }));

        druid::WindowDesc::new(|app, winhandle| {
            druid-shell::platform::agnostic::Window::Builder::new(app, winhandle, widget)
        }));

Move FileDialogs (and other native components) to a native widget component namespace; there are some issues here due to how the code base is currently laid out i think. ideally you'd want to be able to access both the backend native handle and druid::widgets so that we could have a fallback implementation when reasonable (thinking about like menus atm)

   Result<FileDialog> dialog = druid-shell::widgets::native::FileDialog::new(parentSurface); // parent surface would be the winhandle or some such.

[xStrom]

it may not be a good idea to throttle progress with 90s concepts like standardized menu and file dialog.

They certainly don't translate to all platforms when we include web & mobile. However even modern desktop platforms make extensive use of them. Even if we don't care about Wayland supporting these, Druid still needs to support these in a cross-platform way on Windows & macOS - which it already does as it is implemented.

In my eyes the question here is whether we can bring Wayland up to par with the Windows & macOS implementations, or do we need to leave Wayland out and have it be more like web & mobile - a different class of platform. I certainly don't consider the idea of deleting the Windows & macOS implementations as reasonable. (Not accusing any of you specifically suggesting this)

from userbase perspective - desktop is a dead platform for most use cases.

Percentage wise, sure - classic computer market share (desktop and laptop) is shrinking. However in absolute numbers there are more classic computers than ever before. That target audiance is growing!

There are more than 1.4 billion monthly active devices running Windows 10 or Windows 11. That is a huge market!

[james-lawrence]

certainly don't consider the idea of deleting the Windows & macOS implementations as reasonable

no one is suggesting this. we're saying don't make backend rendering, or druid core api's tied to implementation details for them. Its never once been said delete them. move them to a higher level; not remove support. the changes in my repo are able to support multiple implementations for different UI environments seamlessly.

I didn't do the full work to completely cut over each backend because i knew this would be a giant conversation; and wasn't worth the effort for what I was attempting to accomplish.

[xStrom]

When I've been referencing to Druid capitalized, it means the whole project. I can see this being easy to miss as not everyone is familiar with the Druid name style guide. I mention this because I'm thinking of this issue in the context of the whole project. Moving the code/API around might make sense indeed - I don't have any particular takes on moving stuff around at this moment.

What I'm talking about is that when I write my application, can I also compile it for a Linux backend without having to write custom platform specific code? This would be an immensely amazing thing. It does seem possible for targeting Windows & macOS. There are also signs that point that this is possible with GTK.

So when I'm talking about these things in the context of Wayland, it is to aim for a goal where Wayland can be a first class platform that I can also target without having to write custom Wayland specific code in my application.

Hopefully this clears up my motivations here.

Now when you talk about moving around the API @james-lawrence, do you envision a situation where the application writer has to actually care about Wayland enough to write custom code for it? Or perhaps an alternative situation where the Druid project actively discourages using APIs that don't work on Wayland - so the lowest common denominator approach? Something else that I'm missing?

[james-lawrence]

What I'm talking about is that when I write my application, can I also compile it for a Linux backend without having to write custom platform specific code?

yes for the most part with reasonable fall backs (handled automatically by druid). its mostly the window decorations (close/minimize/borders) gnome doesn't support standardization there via wayland sadly. in order to do this well though we'd need to be able to access druid::widgets when implementing the layout for a window.

So when I'm talking about these things in the context of Wayland, it is to aim for a goal where Wayland can be a first class platform that I can also target without having to write custom Wayland specific code in my application.

we all share this motivation. the point here is that linux environments allow a bunch of options that the traditional floating window models don't support. and linux desktops tend to be a little bit more mix and match so you need to actually have a 'whelp fuck it; you get druids custom implementation of menus because the desktop environment doesn't support it`.

this isn't the application writers fault, its a user's choice issue.

Now when you talk about moving around the API @james-lawrence, do you envision a situation where the application writer has to actually care about Wayland enough to write custom code for it?

nope. I'm envisioning that if they want to use wayland (insert other platforms here) specific capabilities that don't fit into the lowest common denominator mold then they can.

what they will have to handle is when a platform doesn't support something. like use printers for an example. if for whatever reason there is no printing service what do we do? we have to tell the developer; hey man printing isn't available so that they can handle it correctly. same with things like accessing webcams, microphones, etc.

Or perhaps an alternative situation where the Druid project actively discourages using APIs that don't work on Wayland - so the lowest common denominator approach?

I'm not against the floating window style layout as the default, or filedialogs, or menus. it just needs to be moved up a little bit in design of druid so we can actually implement reasonable fallbacks when a particular environment is being used.

in this case its less about lowest common denominator, and more about where are the different levels of abstraction in druid and what do they need to do.

low level - rendering/compositing/input <-- this is what the rendering backends should be library level - widgets/tree <-- druid widgets, modals, etc layout/platform level - shell environment functionality and widget layouts <-- this is where menu's, file, printing, IME, url openers, and layouts (floating windows, layershells) should live. there might be multiple types of layouts in a specific platform; wayland has both traditional windows like macosx and windows, plus a couple others.

right now druid is mashing together the rendering/compositing/input level with the layout/platform level. and as a result it actually makes doing some stuff like falling back when server side decorations are not supported difficult to implement when you end up in an application environment that doesn't have some native api.

its hard to implement a fallback menu implementation in druid when the api is implemented by the backend rendering namespace when all the druid UI widgets use the rendering namespace.

[xStrom]

I appreciate you, @james-lawrence, taking the time to push the discussion forward. As it stands now I'm no longer worried that our goals have a significant fundamental mismatch.

Now to talk about potential API shifts, I can think of a few things to consider.

• The original idea of decoupling druid and druid-shell was to allow other projects to use druid-shell. Moving polyfills to druid will reduce the value proposition of druid-shell. It might be the right thing to do anyway, but it's probably worth considering this when redesigning the API. The stuff that can be polyfilled without having to depend on druid might be worth having in a different crate - druid-shell or a new one - so that other projects could benefit from it too. This is what I consider a nice-to-have feature though and I wouldn't do it at a significant cost to the Druid project itself.
• Some of these types of platform features might need very close integration with the rest of the platform code. For example Windows features really love making use of the window message loop. This means that some of these features would have to live quite literally interleaved with the rest of druid-shell code, regardless of how the API is surfaced.

One way to do it would be to use traits. There could perahps be a base trait that is common between every platform, even mobile. Although maybe we shouldn't design that trait until we have real mobile implementations - however the web platform could serve as a proxy. Then a different trait for window management, which all the desktop platforms could implement. Then a trait for menus, a trait for file open/save dialogs etc.

This way we would have official levels of support. A backend could only implement certain traits, but they would be actually implemented and there wouldn't be the situation where the application developers would keep running into unimplemented warnings at runtime.

Also these traits could be implemented by polyfills, in a different crate even. So for example (thinking about alert windows, which I've been working on) we could have an Alerts trait which druid-shell already implements for the Windows backend, but then we could also have a druid implementation of alerts. The developer could then configure their druid instance in a way where on Windows the druid-shell implementation is used, but on Wayland the druid implementation is used. Or they could even configure all platforms to use the druid implementation, because maybe it's just that cool with theme support etc.

When thinking about cases where such trait strategy wouldn't be super straightforward, the Wayland window decorations pop into mind. Specifically because some compositors offer their own implementations. I guess this could be solved by having some sort of WaylandDecorations trait which has some sort of method like fn provides_decorations(&self) -> bool that the polyfill trait implementation can check for to know wheter a polyfill is even needed or if it can redirect the work to the compositor. It's possible there's a better way to do it too, but yeah doesn't seem like a blocker issue.

Okay, now that I gave this a bit of thought, something like the trait system I described seems to alleviate issues that I personally see. What about you @james-lawrence, would this also address your concerns and/or do you see issues with the general approach?

[james-lawrence]

What about you @james-lawrence, would this also address your concerns and/or do you see issues with the general approach?

its basically what wayland backend does right now. (1) (2) (3)

I'm okay with it. its mostly because of the WindowHandle API exposed more than I think it needed to as a result; I needed to polyfill. =)

Specifically because some compositors offer their own implementations. I guess this could be solved by having some sort of WaylandDecorations trait which has some sort of method like fn provides_decorations(&self) -> bool

that'd be one way to do it. I'd just provide an implementation for the backend to use (or not). see the first link with the decor field. its exactly this idea.

[black7375]

If you have support x11 and wayland backends without GTK, EGL sounds like a good choice.

• https://mozillagfx.wordpress.com/2021/10/30/switching-the-linux-graphics-stack-from-glx-to-egl/
• https://blog.gtk.org/2021/05/10/adventures-in-graphics-apis/