search

rust-windowing / winit

Window handling library in pure Rust
https://docs.rs/winit/
Apache License 2.0
4.75k stars 893 forks source link

Implement kms backend? #1006

Closed goddessfreya closed 7 months ago

goddessfreya commented 5 years ago

Right, so there was some chatter about implementing a kmsmod backend like SDL's. Mouse input could be provided by a library outside of winit that uses evdev.

Such a backend would allow us to run applications in the tty, without the presence of X11/Wayland/DirectFB, which I'm sure would open up a lot of silly possibilities. To use alacritty as an example, such a backend would allow it to replace similar projects like kmscon, which I think would kind of be nifty.

@chrisduerr What's your opinion on running alacritty in the tty?

Osspial commented 5 years ago

Could you provide a link to the SDL discussion and what exactly kmsmod is? I can't seem to find either from some quick google searches.

chrisduerr commented 5 years ago

@chrisduerr What's your opinion on running alacritty in the tty?

Running Alacritty itself in the TTY seems a bit pointless tbh, since it builds on GUI and GPU Acceleration at its foundation, but doesn't provide much functionality when it comes to window management (it instead outsources that to the WM).

However, it would be possible to rewrite the renderer of Alacritty for a TTY backend, but at that point it would probably be easier to just skip glutin/winit completely.

However I've never had any interest in replacing my console, so maybe other people have different opinions here. I don't think it wouldn't be fun to play around with, but I would never upstream support for a PR that implements this since it's just a load of mostly unnecessary (to me) complexity.

And I feel like that might be similar for winit/glutin in general. I'd be curious why this was requested. Which kind of application should this support and who's the target audience? Since many features of winit/glutin probably don't exist there anyways, would it maybe be better to just write a completely separate toolkit? What can winit/glutin provide here that wouldn't have to be written from scratch?

Also additional platforms of course always mean additional work and maintenance overhead. Testing in CI might be difficult and it is probably even more work than just testing linux with a display server. So who would be willing to maintain it? Initial implementation is one thing, but often maintainership is even harder than the initial support.

Of course these are just my two cents, making guesses about a chat conversation I haven't been part of, but in general I'm not immediately in love with the idea.

goddessfreya commented 5 years ago

Could you provide a link to the SDL discussion and what exactly kmsmod is? I can't seem to find either from some quick google searches.

Discussion took place on discord. Go to the gamedev channel on the rust community discord and search for there's one thing winit doesn't do as far as i know that i would like to: run straight out of a terminal.

https://wiki.archlinux.org/index.php/Kernel_mode_setting

goddessfreya commented 5 years ago

but doesn't provide much functionality when it comes to window management (it instead outsources that to the WM).

You're in luck, nor does kmscon.

And I feel like that might be similar for winit/glutin in general. I'd be curious why this was requested. Which kind of application should this support and who's the target audience?

This was suggested for the shits and giggles.

Since many features of winit/glutin probably don't exist there anyways, would it maybe be better to just write a completely separate toolkit? What can winit/glutin provide here that wouldn't have to be written from scratch?

Well the idea was to get existing programs using glutin working in the tty with minimal changes to them. It could be a separate library, of course, but it's going to end up exposing the same api as winit/glutin. Also, most of the features should be present, if we use kmsmod with evdev. If the computer supports hardware cursors, mouse support would even be present. If not, a software impl could probably be schemed up in a separate library.

Testing in CI might be difficult and it is probably even more work than just testing linux with a display server. So who would be willing to maintain it?

Currently we don't test programs in the CI, just make sure they compile. I'd argue testing this backend would be easier than others. Ctrl-alt-F2, cd into glutin, then cargo run --example window, or something like that.

Also additional platforms of course always mean additional work and maintenance overhead. [...] Initial implementation is one thing, but often maintainership is even harder than the initial support.

Agreed. The person who submits it should proably also maintain it. (Altho I suspect this backend will require minimal maintenance.

elinorbgr commented 5 years ago

For what it's worth, this is very similar in scope to what smithay does, if you only keep the backend part and disable the wayland-clients-handling features.

NickeZ commented 5 years ago

If I understand correctly, this would be very cool for security sensitive "embedded" and/or "kiosk" applications where you don't want to run the X11 stack or other c-software full of possible leaks.

Lokathor commented 5 years ago

Yeah the point is that low end devices (raspberry pi, odroid, etc) can have things to display but not want to put an x11 session in RAM.

aleksander commented 5 years ago

So something like kmscube would be possible? Cool!

cheako commented 4 years ago

kmscon isn't maintained anymore, I'll just call this virtual console? To write an abstraction layer for an application to choose from winit and an internal handler for virtual console might be just the same as writing a eventloop for winit. I'd like to get a feeling for how well something like winit/src/platform_impl/linux/vc would be received?

Lokathor commented 4 years ago

SDL2 supports DirectFB in their docs and I was able to get it to open a fullscreen window on my rpi without turning on X. Was it probably going through DirectFB when it did that? Should we support DirectFB as a driver?

goddessfreya commented 4 years ago

kmscon isn't maintained anymore,

kmscon != kms.

inodentry commented 4 years ago

Everyone is going on about using this for alacritty for a kms-based console. While that is cool, it is just one use case. I'd like to point out other awesome use cases for this functionality:

  1. GUIs for embedded systems. If you only have one single fullscreen application that should have total control of everything and the user should directly interact with. It would be nice for that application to run without requiring X11 or a wayland compositor or anything of that sort. Just launch it and it takes care of its display and rendering and input on its own however it likes (using this proposed winit functionality).

  2. Games. This would allow to develop a game that effectively has direct access to the display, GPU, input, etc. This would remove any overhead that would otherwise come from the windowing system protocol / compositor and result in the best possible performance and latency.

nezticle commented 4 years ago

I see this as very analogous to what we have in Qt with the EGLFS platform: https://code.qt.io/cgit/qt/qtbase.git/tree/src/plugins/platforms/eglfs This itself has a lot of complication because there are a couple of different approaches to actually get the EGLDisplay and EGLSurface (fbdev, drm/kms), and then further there are sometimes vendor specific tricks you have to do as well. In the case of the EGLFS plugin there are additional plugins to handle this, which is much more complicated than the current platforms support. So maybe check out EGLFS for some inspiration for how this could look/work here.

NathanRoyer commented 3 years ago

Hi, I'm here following my duplicate PR: #1865.

there are a couple of different approaches to actually get the EGLDisplay and EGLSurface (fbdev, drm/kms)

fbdev and its doc are outdated now; DRM/KMS is the currently advised way to go, because while fbdev merely allowed to manipulate a framebuffer, DRM intends to provide in-kernel support for more GPU control.

If I understand correctly, this would be very cool for security sensitive "embedded" and/or "kiosk" applications where you don't want to run the X11 stack or other c-software full of possible leaks.

That's exactly my case, I want to use programs like Servo as a kiosk app in an embedded environment which lacks a compositor. Servo and it's major component webrender rely on winit.

As stated in #1865, I have working rust code demonstrating the use of DRM/KMS, so I'm going to try to implement this backend.

cheako commented 3 years ago

There is also the use case of writing compositors.

NathanRoyer commented 3 years ago

I have some sort of draft and it compiles, but in the end I think I will have no use for it with servo. My implementation replaces a lot of x11/wayland code instead of sitting beside it (platform_impl/linux is such a mess) so I consider it bad and I doubt I will make it clean in the future. However my test raised some questions about how this should be done; Should there be:

Also raw_window_handle has to pass some DrmHandle I added in my implementation, representing an EGL surface. DrmHandle has two fields (card + resource handle) and I find it quite dirty. winit clients draw on the surface using egl. This is one way to do but there must be others.

cheako commented 3 years ago

I don't think there should be a 1:1 mapping to the concept of Windows, likely another abstraction Displays or Cards that implements a common trait with Windows would be the way to go. If an application is targeting virtual consoles it's likely they won't be interested %90 of any Window abstraction. In the interests of having small purpose dependent crates I believe that higher level abstraction(*s) would be used for applications that want to target both.

StratusFearMe21 commented 2 years ago

I don't think there should be a 1:1 mapping to the concept of Windows, likely another abstraction Displays or Cards that implements a common trait with Windows would be the way to go. If an application is targeting virtual consoles it's likely they won't be interested %90 of any Window abstraction. In the interests of having small purpose dependent crates I believe that higher level abstraction(*s) would be used for applications that want to target both.

  • There would be cases were the decision is made at compile time and other applications would want to make that decision at runtime, so there is room for two crates that provide a single abstraction.

Yes, it's true that if you have the intent to make a kmsmod application, you don't have the desire to also make it compatible with Wayland/X, however, implementing it like this would nerf the fun factor of having kmsmod be part of the Window abstraction. Imagine being able to run a bevy game directly out of the console with minimal/no changes. That would be so cool

kchibisov commented 2 years ago

@StratusFearMe21 if you can port glutin, so we can have opengl with your winit backend, I can port alacritty, so you'd have some kind of GPU accelerated terminal on tty.

But in general, getting OpenGL to work seems like a good start to test drive kms backend, since it's likely easier to use hardware rendering than anything else here.

That's in the context of your recent patch to add kms backend, I'm fine with having it around, since it sounds kind of cool.

StratusFearMe21 commented 2 years ago

@kchibisov Sure, that sounds good.

StratusFearMe21 commented 2 years ago

@kchibisov, It is nearly complete. I just have to do the finishing touches to my glutin fork to get it to work, here it is. I do have one issue with it though, the pixel format. You see, gbm has a set of defined formats that you must set when creating the EGL surface, however I don't know enough about EGL or OpenGL to translate all the glutin pixel formats to gbm formats. If someone or you would like to help me with this, the relevant code is located here

StratusFearMe21 commented 2 years ago

Other than that, the entire fork is finished

StratusFearMe21 commented 2 years ago

P.S, according to my testing, all the glium examples work with the fork above. They can be found here

StratusFearMe21 commented 2 years ago

Nevermind the previous comment about pixel formats, I've solved that problem and made a PR to here

StratusFearMe21 commented 2 years ago

@kchibisov @goddessfreya It is complete, the KMS/DRM alacritty fork is done.

Pros

Cons

StratusFearMe21 commented 2 years ago

btw, here is a RAM usage breakdown for my install of Arch Linux using this fork of Alacritty

chrisduerr commented 2 years ago

The mouse cursor is invisible

Considering Wayland already draws its own mouse cursor iirc, it should be relatively trivial to do the same, right?

StratusFearMe21 commented 2 years ago

True, however, Wayland can change how the cursor is drawn depending on the graphics library in use. Winit cannot since it's graphics library agnostic.

kchibisov commented 2 years ago

True, however, Wayland can change how the cursor is drawn depending on the graphics library in use. Winit cannot since it's graphics library agnostic.

Cursor rendering on Wayland is client side. And most clients use some libs to load cursor and draw it.

You can use https://github.com/esposm03/xcursor-rs to achieve that, since that's what wayland-rs is using.

cheako commented 2 years ago

Wayland/X/kms was a compile time choice, so I don't see the downside of being forced to deal with the *inconvenience of not having a Window abstraction. The code changes would either be minimal or substantial, in the latter case I doubt a stub abstraction layer would be sufficient.

vanfanel commented 2 years ago

@StratusFearMe21 Having Alacrity on KMS/DRM is a dream come true for me. Thanks, really, for the work you put into this!

Two questions come to my mind: -Is it really impossible to have other KMS/DRM apps to be launched from that terminal? Apart from the obvious fact that alacrytty/kmsdrm is owning the KMS/DRM system: can't it "let it go" while the KMS/DRM program runs? -Is it possible to use shaders with this? Like in, just imagine a TTY console with proper scanlines + shadow mask + curvature...

StratusFearMe21 commented 2 years ago
  1. Im not entirely sure on that. Ill have to do some more research into that.
  2. If you're willing to mess about with the alacritty renderer, yes, shaders are absolutely possible
vanfanel commented 2 years ago

@StratusFearMe21 Please, PLEASE keep me/us all informed about your findings with regards to point 1... just imagine having an advanced, accelerated TTY console on GNU/Linux at long last. It's long overdue so I think that what you are doing is very important.

By messing with alacritty renderer, do you mean adding loadable shaders support?

StratusFearMe21 commented 2 years ago

Yes. The best part about adding that is that it would apply to all platforms.

mraerino commented 1 year ago

i updated @StratusFearMe21's PR to match current master: https://github.com/rust-windowing/winit/pull/2795

notgull commented 7 months ago

This issue came up in our weekly winit maintainer's meeting. Our question is: how badly do people want this?

If one of our stakeholders (egui, bevy) really wants this, I'm happy to write and maintain a DRM/KMS backend. If someone is willing to step up and become a maintainer, we can have them write and maintain the backend. But only if they are committed to maintaining this backend.

Otherwise, it's a lot of additional work for a platform that I doubt will be popularly used. Thus far most of the rationale is "it'd be cool to use Alacritty as a TTY". But it would add a lot of additional maintenance just for that.

cheako commented 7 months ago

You won't get users until you have an implementation... This chicken-egg style thinking is self-destructive. Unless you are advocating that KMS is deprecated and shouldn't have any users, then the use cases for winit are the use cases for KMS.

kchibisov commented 7 months ago

Unless you are advocating that KMS is deprecated and shouldn't have any users, then the use cases for winit are the use cases for KMS.

KMS is basically writing wayland compositor, my opinion that if you're into writing wayland compositor, better use smithay or wlroots and run winit as wayland app inside. If we're talking about traditional kms stuff, it's also not really portable to anything other than linux and freebsd, the rest is far more challengeable to get right.

Updating it also requires special knowledge for little to no benefit, since the primary consumer for simple kms stuff will be kiosk application, but you can use cage for that just fine.

I'd welcome such backend out of tree to exist after winit 0.31.0.

cheako commented 7 months ago

video players, FPS/games where latency is king, login managers, boss screens and screen savers... I'm sure I could think of more.

kchibisov commented 7 months ago

Modern wayland compositors have low latency and soon can do async page flips. They also direct scanout arbitrary surfaces and not just fullscreen. To write login manager you can use smithay, I did that myself, took less than 1k LoC and I hand rolled drm swapchain.

Screen savers for kms?

People who interested can write such backend out of tree after 0.31.0 as I said. Having special purpose backends in tree is not what I really want to see, since I'm always the one dealing with all of that in the end.

daxpedda commented 7 months ago

Having special purpose backends in tree is not what I really want to see, since I'm always the one dealing with all of that in the end.

I'm not sure what exactly this means, but if we have a reliable maintainer, in this case @notgull, I don't see a reason why not to have this in-tree. If you somehow "end up maintaining it anyway", you can kick it out then.

chrisduerr commented 7 months ago

If you somehow "end up maintaining it anyway", you can kick it out then.

If only making software was that simple.

notgull commented 7 months ago

To clarify, I only offered to maintain this theoretical DRM/KMS backend if one of winit's major users wanted it. As far as I know, this isn't the case.

The reason being is that it's actually quite a bit of work to maintain a winit backend. Especially for a backend that would involve a substantial amount of new code, you also have to keep up with new designs and bug fixes.

Not to mention, it ends up being a lot of work for other maintainers trying to implement new design patterns. I'm not a maintainer for any backend except X11, but I have to know all of the other backends moderately well in order to implement new features or do any refactoring.

Anecdotally, we've already had two people step up to be maintainers and then bail out for what I assume to be the realization that maintaining this backend is more than they bargained for. So it's definitely work that can't be done enthusiastically.

If someone else wants to step up be my guest, but we'd expect you to maintain it for at least a while.

cheako commented 7 months ago

Sorry I didn't mean for this to turn into an argument, I just wanted to clarify that the one argument about not having users sounds fallacious and might not be a good argument. It's obvious and unimpressive that nobody is using code that hasn't been written and using the lack of ppl trying as a justification.

kchibisov commented 7 months ago

I'm only against having in tree because it's the hardest backend to get right and maintain as well as bring to some feature parity. But being it out of tree won't make any difference to it, other than not being automatically updated by core maintainers.

ids1024 commented 7 months ago

There are a couple issues here.

I wonder if it might be good to create a Rust library for handling framebuffers, which could provided a somewhat higher level API over drm/kms, and other backends if anyone cares much (Linux's older fbdev API, wscons for some hardware on NetBSD/OpenBSD, Redox framebuffer). That library could provide raw-window-handle support, and perhaps a winit-compatability layer could be built on top of that, along with something for handling input.

cheako commented 7 months ago

Winit would be useful to run KMS apps nested. It's not really material if winit's API is structured to allow virtual console as a backed or if winit is a backend for a higher level API, but that is the heart of this issue. It's confusing to think that winit would become a low level backend alongside KMS, would one assume that a KMS backend would be raised to unorthodox heights as a result of this arrangement?

kchibisov commented 7 months ago

winit is moving towards winit-core and backends in external crates, thus you'll be able to write your backend for whatever you want (probably even tui).

daxpedda commented 7 months ago

We have agreed that there is currently no sufficient external (and internal) motivation to maintain this in-tree.

So until this changes, the goal is to support it out-of-tree, which is covered by https://github.com/rust-windowing/winit/issues/3433.

Thanks everybody for your input so far!

madsmtm commented 7 months ago

I'd like to add: While I recognize that this could be useful in other contexts, it really does sound like most people want this in Alacritty. So in the meantime, until we can do out-of-tree backends, you could also consider supporting this directly in Alacritty, i.e. without doing a Winit fork.