Inivite potentially interested collaborators

[probonopd]

Hello friends!

This is a project set up to brainstorm about the "missing Wayland protocols", the stuff that would be needed to bring Wayland closer to X11 functionality. This project is a "grassroots" effort and is in no way affiliated with or goverend by the Wayland project.

As someone who finds Wayland utterly lacking and (intentionally) broken, I would like to reach out to you to invite you to have a look around here, give your opinion, and maybe collaborate. I hope it is ok to mention you here, since you have written about the Wayland deficiencies before:

• @tildearrow (Reference)
• @dudemanguy (Reference)
• @stenzek (Reference)
• @ximion (Reference)
• @kelnos (Reference)
• @lproven (can't find the reference atm but was somewhere on Phoronix)
• @phkaeser (Reference, wlmaker)
• @ivyl (Reference)

Happy to hear about your thoughts.

[kelnos]

I agree in principle, though I think this statement (from the repo README) is a bit much:

Because the windowing system is not a sandbox, and applications should be allowed whatever they want to do.

I don't think applications should be allow to do whatever they want to do. I think the user should be able to do whatever they want to do, but that's not the same thing. The reason that some of the things we want to do on Wayland don't work is for security reasons: one Wayland client should not be able to interfere with or 'steal' data from other clients.

Now, I completely agree that you can't build all the same interesting, useful features into a Wayland desktop that you can with an X11 desktop today, and that's a problem. But I'm concerned that this attitude of "apps should be able to do whatever they want" will lead to a weakening of some of Wayland's actually-useful (security?) features.

Screen capture/sharing is a great example here. At this point the screen capture methods on Wayland work decently well (though I'm not happy with how they're tied to particular implementations of desktop services). But do we really want a generic Wayland protocol that allows any application to get the window contents of any other application? I certainly don't; this is one of X11's biggest weaknesses, IMO.

[kelnos]

Anyway: overall, I'm a bit uncomfortable doing this outside the standard wayland-protocols process. I know that process is slow and bureaucratic (there are useful protocols that have been languishing for years in the process to get into staging/unstable, let alone stable status), but part of the whole problem is fragmentation, and having yet another source of protocols is going to make that worse, not better.

[Dudemanguy]

I do agree that it is very frustrating that upstream is not interested in implementing (and sometimes even rejecting) features present in virtually every other windowing system including X11. Unfortunately, I'm not sure what can really be done. Let's suppose we go all in and make our own custom protocols to fill in the gaps of things missing. That's not hard: it's just xml files after all. The problem then becomes: who is actually going to use it? Compositors would need to support this as well as clients. Patches for compositors are almost certainly going to be rejected so that hurdle is already pretty much insurmountable. I dunno maybe if you make your own compositor that supports this kind of stuff and if it gets really popular perhaps there's a path forward, but that seems like a longshot to me.

[TechScribe-Deaf]

I do agree that it is very frustrating that upstream is not interested in implementing (and sometimes even rejecting) features present in virtually every other windowing system including X11. Unfortunately, I'm not sure what can really be done. Let's suppose we go all in and make our own custom protocols to fill in the gaps of things missing. That's not hard: it's just xml files after all. The problem then becomes: who is actually going to use it? Compositors would need to support this as well as clients. Patches for compositors are almost certainly going to be rejected so that hurdle is already pretty much insurmountable. I dunno maybe if you make your own compositor that supports this kind of stuff and if it gets really popular perhaps there's a path forward, but that seems like a longshot to me.

All of those are valid points as you say. I think it's better to discuss extension on top of Wayland while supporting either Wayland/X11 Protocol. One could define the extension protocol on top of Wayland and then write an extension for existing compositor like Plasma to allow the user to enable the extension if desired which requires a soft fork as you surmised on the possibility of having patches rejected.

On the point of security brought up by kelnos, I have to hard disagree on the basis that users often value usability over security. We already have several security mechanisms in place that are SUFFICIENT for your average linux desktop users. We have flatpak for containerizing untrusted software, we trust the packagers to provide us vetted software, and so forth. I am not saying we should disregard security wholly altogether for all eternity, but I am saying that we should look toward expanding the usability of a Linux desktop experience to alleviate the pain points experienced in transition from X11 to Wayland before we can start exploring solutions for security on this protocol.

And on fragmentation, it is not really a problem, because this project serves as an optional extension on top of Wayland to include the things that Wayland developers refuse to compromise or concede on for compositor to implement, so we naturally have the option to reject the Wayland developers' opinion on design decisions, and that is a healthy thing to have in open source. We are not one united body of developers, we have differing perspectives, experience, and opinions on how to approach challenging problems.

If we were unified body of developers, we would all be stuck on Gnome that thinks it know better than it's users and continue to nuke it's usability, customizability, and features on a whim.

[kelnos]

I have to hard disagree on the basis that users often value usability over security

Oh, I absolutely agree with that. But I think we should set things up so that some high level of security can be maintained by default. A compositor can of course choose to expose "sensitive" protocols to all clients if it wants to, even without some sort of use confirmation step. But I personally would not be interested in implementing anything that erodes security by default.

We have flatpak for containerizing untrusted software

Personally I avoid Flatpak, as I don't really trust its security model, and many popular apps on Flathub seem to have parts of the sandboxing disabled (without it being obvious this is the case at time of install). While I would agree sandboxing/containerization is a good part of a defense-in-depth strategy, I'd rather we make things as secure as possible without needing it. There aren't really any Linux desktops that are built with that kind of isolation in mind, and it shows with the subpar user experience you get with Flatpak and Snap apps.

And on fragmentation, it is not really a problem...

I don't agree. Anything that's optional is by definition inviting fragmentation. And having competing, but similar, protocol ideas adopted between GNOME, KDE, wlroots, weston, etc. -- plus "our" protocol ideas -- means more protocol checks and optional and fallback code paths in toolkits and apps.

One thing that does give me hope is that clearly some people with "power" do recognize that it's too difficult to get things into wayland-protocols. This MR, for example, which is about maybe relaxing the requirements for merging into the ext- namespace. Even though that idea might not go through, it's encouraging to me that there are people who continue to think about governance.

Anyway... I'm personally in an interesting spot in that I'm mostly interested in "desktop environment protocols": that is, stuff that'd primarily be used between a compositor and other DE-level apps, and not so much between a compositor and any random application. So I could easily say "ok, so xfwm4-wayland is going to implement this protocol XYZ so that the system taskbar can get information it needs", and that's fine.

[ximion]

I too would like to solve the issues within Wayland first, for example via the proposed xdg-placement protocol as well as xdg-toplevel-icon. Those (especially the former) are real issues that prevent porting apps to Wayland.

That said, while some of the issues are definitely a problem that we should address, ideally by coming to a cross-desktop agreement so we don't make life harder for toolkit developers than it already is, there are also some proposals here that I do not support. For example, portals for screen capture are a good thing! Same for hotkeys, KDE actually has an ingenious solution by only allowing key capture even for X11 apps when a modifier key is pressed.

We should not make it too easy for malicious apps to keylog everything in the background and to capture the user's screen without consent. Yes, it breaks backwards compatibility, but in this case to me the benefit far outweighs the breakage it causes. Especially since there is a way forward for apps to be ported (unlike with the window-placement stuff which pretty much forces a UI rewrite).

Some issues on the list I don't even get, e.g. without sandbox apps can easily kill each other if they belong to the same user, wayland doesn't prevent that. I also don't get the need to set atoms on windows, for IPC we do have D-Bus or Varlink over any medium which serve that space far better.

All that said, feel free to use the draft protocols I am working on for Wayland, and I definitely appreciate any feedback there :-)

[tildearrow]

Here's my opinion on the matter.

I believe that placing screen/window capture outside is not necessarily a bad thing. It is true that doing so adds to the fragmentation problem and makes the process of creating a new Wayland compositor/display server more difficult, but at the same time it allows off-loading these (potentially?) heavy operations somewhere else, which prevents server message queue stalls and therefore missed frames.

Stable/staging/unstable. It's a good idea, but during preparation stage we should get these into stable as soon as these are completed. Many useful, already-working protocols in wayland-protocols are still unstable despite being so for years.

I would like to suggest the following as well:

• A simple permission system. See the "but security" section.
• Client self-query functions. Clients cannot know the position of their surfaces, for example.
• Data query. Cursor position (complete with xhot/yhot position), keyboard state, window geometries, and all of that.
  • There's AT-SPI2, but it is veeery limited. No xhot/yhot position, no cursor image, no this, no that. Only the bare minimum for "accessibility".
• An equivalent to XTEST. Ability to send keystrokes, move the cursor, move/resize windows and so on.
  • There's ydotool, but it is somewhat limited.
  • The Wayla- erm... GNOM- erm.... Red Hat's approach of libei doesn't convince me. Besides, not in protocol, and proooobably will have one or other limitation that makes it useless for me.
• Frame callback-less rendering (ability to submit new frames at any time) (assuming it doesn't exist yet). I believe that the attempts to fully eliminate tearing have crippled the ability for clients to tear when it actually is favored (e.g. some games).

Kill other applications (like xkill)

This can be done after implementing "Get the process ID and other vital information (like in xprop)".

It is not expected that all Wayland compositors will implement these protocols, but most likely some will

Personally I would hate if this effort ends up being useless, because I really want to see these missing protocols in. We'll have a hard time getting GNOME to support these, but at least let's get in touch with the KDE and wlroots teams once these protocols have been stabilized.

But... security.

Security is important. It's done wrong in both X11 and Wayland. X11 exposes too much functionality (and clients expect this functionality to be available, which renders security solutions such as XACE useless), and Wayland obscures it completely.

We can take a macOS-style approach for security:

• Provide a (tiny) permission system, with functions to query and request access to certain protocols. It is up to the compositor to implement permission management and storage.
• Protocols which extend the display server or access other clients such as global shortcuts, data query, the XTEST equivalent, screen/window capture, and so on, are bound by the aforementioned permission system. If access to a protocol is denied, simply return an error value.
• Since we're doing this early (when compared to XACE which is too late), software developers are more likely to check return values and handle permission denials.

If you want to restrict what applications can do, use a sandbox.

I believe we can do this without adding more layers. As I mentioned before, a simple return-value-based permission system is enough.

[probonopd]

@kelnos, welcome and thanks for chiming in here!

I don't think applications should be allow to do whatever they want to do. I think the user should be able to do whatever they want to do, but that's not the same thing.

Well, if you start from the premise that the user only runs trusted applications and wants those applications to "just work" as the application developer intended, then they are pretty much the same thing. The purpose of a professional workstation-grade operating system (to me, anyway) is to "just work", let applications do their thing, and otherwise get out of the way.

Isn't it unfortunate that today's desktop operating systems tend to make life miserable by making it harder for users to achieve what they want to achieve? Point in case: The default browser in Windows 11 doesn't let me download a zip file because it somehow thinks it is insecure. This has come to the point that the simplest stuff doesn't "just work" anymore. To stay in the Windows 11 example, there are so many nag screens, exclamation marks, popups and notifications that I always end up disabling all of them without even reading them (and being annoyed in the process). Does this accomplish anything besides making life miserable? It is like you opening the door of a car and the car refusing to start, bringing up a dialog box saying "it might be unsafe" to drive.

Some types of users or use cases may require some level of sandboxing (restricting what applications can do). However, I think this concern should be addressed on another level. Some brainstorming on an architecture that could accomplish this:

• https://github.com/probonopd/wayland-x11-compat-protocols/issues/12

[probonopd]

@Dudemanguy thanks for sharing your concerns.

Let's suppose we go all in and make our own custom protocols to fill in the gaps of things missing. That's not hard: it's just xml files after all. The problem then becomes: who is actually going to use it? Compositors would need to support this as well as clients.

After having thought about the whole Wayland situation and having collected its shortcomings for a while, I have come to the conclusion that the fundamental issues that today's Wayland has won't be solved by just continuing to whine about what doesn't work (this has been going on for over a decade and Wayland still doesn't have X11 feature parity). I also realized that there is a significant number of people who would like to see a Wayland with X11 feature parity.

Hence, my (somewhat oversimplified) line of thought goes as follows:

1. Let's collect the missing features (things that today's Wayland can't do without extra software, but X11, Windows, macOS,... can)
2. Let's brainstorm on how to phrase the missing features (ideally conceptually as close to X11 as possible so as to minimize application developer's pain in porting)
3. Write the xml
4. Sit back and wait for at least one compositor (library; yes I think of you wlroots) to implement these features (can be made opt-in by a configuration setting)
5. Whenever application developers get bug reports about missing functionality due to Wayland restrictions, point them to these optional protocols
6. See how people flock to that compositor (library)
7. See how more compositors want to make the applications "just work"
8. World domination ;-)

Too optimistic? Maybe. But then, if no one does anything, then the situation will stay as it is (and people who like a proper open source desktop may need to switch to Haiku for good).

[probonopd]

@TechScribe-Deaf

I am not saying we should disregard security wholly altogether for all eternity, but I am saying that we should look toward expanding the usability of a Linux desktop experience to alleviate the pain points experienced in transition from X11 to Wayland before we can start exploring solutions for security on this protocol.

Yes, today's Wayland seems to be concerned with restricting things even before it has those things work. Personally I think "security" is a bad excuse for not having required functionality (as in: other desktop systems have it) at all.

Would this address your concerns?

• https://github.com/probonopd/wayland-x11-compat-protocols/issues/12

[probonopd]

@kelnos

Anyway: overall, I'm a bit uncomfortable doing this outside the standard wayland-protocols process. I know that process is slow and bureaucratic (there are useful protocols that have been languishing for years in the process to get into staging/unstable, let alone stable status), but part of the whole problem is fragmentation, and having yet another source of protocols is going to make that worse, not better.

Well, there is truth in it, but then -- how many years did the standard wayland-protocols process have by now? And how many obviously missing features were discussed/delayed to death there? Just count how many features that "just work" in X11 were stalled in the official process, and how many people this left back frustrated.

If we make optional protocols outside of the standard process, nothing prevents them from being accepted into the standard once it becomes obvious how much people like them.

This is open source. Love it, leave it, or change it!

[kelnos]

Well, if you start from the premise that the user only runs trusted applications and wants those applications to "just work" as the application developer intended, then they are pretty much the same thing.

Unfortunately I would not start from that premise.

I agree that things should "just work", but there's of course a tension between security and usability. I think we should be willing to sacrifice a little usability for security, but it should be done in a way that doesn't cause frustration, as much as that's possible.

I think it's fine for a compositor to decide to not care about security, if that's the kind of user it's aiming at. But I don't want to assume that's the only kind of compositor; it should at least be possible to add things like permissions prompts.

[probonopd]

Hello @ximion, nice to see you here.

I too would like to solve the issues within Wayland first

I really appreciate the hard work you are putting into this. You are acknowledging that there are missing features, and you are working with the Wayland community to get those missing features into Wayland over the standard wayland-protocols process. That's great! The more that gets fixed in Wayland upstream, the less we have to take care of outside of the standard process. Ideally, everything would get fixed in Wayland upstream and this project would not even be necessary - but unfortunately the last decade or so has shown otherwise.

So, should you ever encounter a situation in which the standard wayland-protocols frustratingly denies your contributions from being merged, know that we'd definitely welcome your contributions here!

And of course we value your thoughts and insights on the ideas being brainstormed here - no matter whether and were you might end up working on them. Welcome!

[probonopd]

Hi @tildearrow, great you made it here.

Stable/staging/unstable

Basically I created these directories to mirror what the standard wayland-protocols process uses, to symbolize that this project aims to complement their protocols, but from the outside. I didn't want to imply that this is necessarily the best way to organize things. There could also be other ways, e.g., using semver for each protocol or even for the whole "complement" suite of protocols (which we still need a good name for).

• A simple permission system. See the "but security" section.

• https://github.com/probonopd/wayland-x11-compat-protocols/issues/12

• Client self-query functions
• Data query
• An equivalent to XTEST
• Frame callback-less rendering

Added to the list.

Personally I would hate if this effort ends up being useless, because I really want to see these missing protocols in. We'll have a hard time getting GNOME to support these, but at least let's get in touch with the KDE and wlroots teams once these protocols have been stabilized.

At this point I think innovation happens in small projects first and once stuff works there, large projects may be more inclined to make stuff work there, too. And if Gnome decides to not want this, then fine. The world doesn't revolve around Gnome.

[a-weasel]

In general I prefer functionality over security, so I completely agree with your point. That said, you could conceivably add some sort of permissions system (Arcan does it with a virtual filesystem using normal unix permissions), such as disallowing other users from doing some things like inspecting your window contents.

But apps from the same user definitely need this functionality for very legit reasons, especially for power users. Which is why Wayland is beyond unusable. And if I have to choose I'd rather be exposed than suffer through manual work I could automate.

[probonopd]

Exactly. Let's make the stuff work first, and then figure out what to take away under which circumstances from which apps and users. But let's make stuff work by default. (Remove permissions, don't add them.)

[kelnos]

I'm really not interested in implementing things where security is an afterthought at best.

A compositor will of course be free to grant all permissions to all clients by default if that's what the author wants to do, but I do want protocol design to consider security up front.

[ilya-fedin]

• Frame callback-less rendering (ability to submit new frames at any time) (assuming it doesn't exist yet). I believe that the attempts to fully eliminate tearing have crippled the ability for clients to tear when it actually is favored (e.g. some games).

You can submit buffers at any time, frame callback only tells you when it's better to do. You can call update() in Qt in a loop, draw something different in paintEvent every time and see that Qt uploads the buffer (via WAYLAND_DEBUG=1 or by looking on the window content) and drains CPU just fine.

[MartinRykfors]

Hello @probonopd

This repo compiles some thoughts about missing features needed for accessibility tools to work under Wayland, in particular Talon Voice which is an application that allows controlling one's computer using voice commands. Talon is not usable under Wayland and the developer behind it has stated that the upcoming version will be the last public release of a Talon build for Linux until the state of accessibility on Wayland has become mature.

I use Talon on Linux but I am not involved in the work of compiling these features in that repo, and I have only a shallow understanding of the workings of wayland and the issues faced by the developers of a11y tools on Linux. But it might be worth to reach out to the author of the repo above.

(Also, I would like to request that you do not reach out to the developer behind Talon Voice as he has repeatedly stated in the Talon Slack community that he is uninterested in repeating the same discussion about the state of Wayland support)

[probonopd]

Thanks @MartinRykfors for reaching out. Welcome! I, too, "have only a shallow understanding of the workings of wayland". But I firmly believe that only if we (the people) make our needs known loud and clear, anything can improve.

So let's be very vocal on what we need, and let's propose protocols that might address the shortcomings.

[phkaeser]

Thanks for adding me to the thread. I'm interested to see some of Wayland's functional gaps closed (the icon protocol standing out to me -- now following @ximion MR to wayland-protocols :) ).

My gut's preference is to add Wayland protocols suitable to solve the problems -- possibly in novel or different ways -- rather than re-implementing X11 that had their share of legacy and trouble. That said, my understanding is shallow for sure, and I haven't yet suffered through upstreaming steps. I've just been told it'd be ... slow.