Add support for direct fullscreen rendering on Linux

[tronical]

It would be great if it was possible to run an application on Linux in full screen without a windowing system (wayland or x11). In some systems the multi-window or multi-seat capability the windowing systems offer is not needed.

In practice on Linux this can be achieved on many hardware platforms using the Mesa's GBM interface, the EGL GBM extension and then the EGL API to get a GL context. There may also be some platforms where GBM is replaced with a vendor specific interface, but in practice those do eventually lead to an EGL window surface. For example when using the proprietary video core driver on the Raspberry PI, the Dispmanx interface provides the EGLNativeWindowType. The libinput provides support for keyboard/mouse/touch input.

These are all traditionally C interfaces, but fortunately the Smithay project provides great APIs to access these components with idiomatic Rust APIs, along with an event loop interface to tie them together. In Smithay this is used to implement the anvil wayland compositor.

This could either be a separate rendering backend, where we'd have to re-organize the GL renderer for code sharing. Or alternatively the existing GL backend could be changed to add support for a module that utilises smithay's backend modules and event loop, besides winit.

[inferiorhumanorgans]

If you're okay depending on Qt you can accomplish this with the EGLFS platform plugin on Linux. If your version of Qt was built with the EGLFS plugin you should be able to set QT_QPA_PLATFORM to eglfs and run your app without X or Wayland. This works on the Raspberry Pi fork of Debian, but I can't speak to other Linux variants. For example:

$ QT_QPA_PLATFORM=eglfs cargo run

[tronical]

Kind contributor @StratusFearMe21 is working on supporting direct rendering in winit and glutin: https://github.com/rust-windowing/winit/pull/2272 🚀 This would enable Slint as well.

[tronical]

I made a proof of concept to do this directly in Slint, using Vulkan's extension to access the display directly, calloop, and libinput. I've uploaded the code into https://github.com/slint-ui/slint/tree/simon/vulkanfs

[flukejones]

For anyone trying to do this in an embedded environment:

#!/bin/sh
export SLINT_BACKEND=Qt

export QT_QPA_PLATFORM=eglfs
export QT_QPA_EGLFS_INTEGRATION=eglfs_kms
export QT_QPA_EGLFS_ALWAYS_SET_MODE=1
export QT_QPA_EGLFS_KMS_ATOMIC=1
export QT_AUTO_SCREEN_SCALE_FACTOR=0
export QT_QPA_EGLFS_KMS_CONFIG=/opt/evolve-hmi/eglfs-tj028.conf

/usr/bin/evolve-hmi 2>&1

eglfs-tj028.conf

{
 "device": "/dev/dri/card1",
 "outputs": [
      { "name": "DSI-1", "mode": "800x480", "size": "800x480" }
 ],
 "hwcursor": false
}

A good reason to use QT for this is that startup time is much faster. The difference between using weston and eglfs is approx 2-5s depending on if clean boot.

[flukejones]

The winit code mentioned above was forked due to inaction by the original author to https://github.com/rust-windowing/winit/pull/2795

[flukejones]

I made a proof of concept to do this directly in Slint, using Vulkan's extension to access the display directly, calloop, and libinput. I've uploaded the code into https://github.com/slint-ui/slint/tree/simon/vulkanfs

@tronical are you planning to have a go at doing this with egl at all? Reason I ask is that vulkan on embedded GPU (vivante series) isn't supported until GC8000 model. I've found that QT eglfs is adequate, but would love to drop that dependency from the build - and weston isn't really a good alternative due to it adding additional seconds to the startup time.

[dongjian]

My embedded device runs linux but it doesn't have a GPU. So QT eglfs doesn't seem to fit. Is there any other way for slint ui to achieve direct framebuffer rendering on linux without a GPU ?

[tronical]

I made a proof of concept to do this directly in Slint, using Vulkan's extension to access the display directly, calloop, and libinput. I've uploaded the code into https://github.com/slint-ui/slint/tree/simon/vulkanfs

@tronical are you planning to have a go at doing this with egl at all? Reason I ask is that vulkan on embedded GPU (vivante series) isn't supported until GC8000 model. I've found that QT eglfs is adequate, but would love to drop that dependency from the build - and weston isn't really a good alternative due to it adding additional seconds to the startup time.

Yes, I read up a little about the gbm surface handling, creating EGL window surfaces from it, and how to do page flipping - I'd like to implement this, too. I have a local WIP for this that renders a few frames with egl instead of Vulkan, but it's not cleaned up yet. On the upside, it's just ~200 lines of code. I'd like to finish that yes and once it works prepare a PR that adds Vulkan or egl rendering on top of kms as an experimental backend for Slint.

[flukejones]

@tronical that's excellent news mate. As soon as you need testing ping me and I'll put it through its paces.

[flukejones]

My embedded device runs linux but it doesn't have a GPU. So QT eglfs doesn't seem to fit. Is there any other way for slint ui to achieve direct framebuffer rendering on linux without a GPU ?

You might look at linuxfb backend of QT maybe. Mileage will vary depending on your environment and what is available.

[tronical]

@tronical that's excellent news mate. As soon as you need testing ping me and I'll put it through its paces.

Okay, I have something that might be worth testing. It's highly experimental and I've merely tested this on my laptop.

There's a simon/linuxkms branch in the repo. In your Slint crate feature selection, disable default features, select the compat-1-0 feature as well as renderer-linuxkms-femtovg.

The application then needs to run as root unfortunately - no libseat support yet. There should be touch support, but I'm not entirely sure it works. Mouse support works, but there's no mouse cursor rendered.

Otherwise this uses a gbm surface to render and non-atomic mode setting for page flips.

[tronical]

Additional notes for testing:

Set the environment variable SLINT_DRM_OUTPUT to select a specific output. Set it to the value “list” to see the available outputs.

The preferred mode of the output is selected. If there is no preferred mode, the highest resolution will be chose.

Both of these need additional fixing to make them more usable, like for specific mode selection and persistence via a file instead of environment variables.

[flukejones]

Hi @tronical I did a quick test build:

error: cannot find macro `eprintln` in this scope
  --> /home/luke/.cargo/git/checkouts/slint-8153123e5dffa129/20fb8724d0aa/internal/backends/selector/lib.rs:68:17
   |
68 |                 eprintln!("Could not load rendering backend {}, fallback to default", backend_config)
   |                 ^^^^^^^^

error[E0433]: failed to resolve: use of undeclared crate or module `std`
  --> /home/luke/.cargo/git/checkouts/slint-8153123e5dffa129/20fb8724d0aa/internal/backends/selector/lib.rs:44:34
   |
44 |             let backend_config = std::env::var("SLINT_BACKEND").unwrap_or_default();
   |                                  ^^^ use of undeclared crate or module `std`

error[E0599]: no function or associated item named `new` found for struct `Backend` in the current scope
  --> /home/luke/.cargo/git/checkouts/slint-8153123e5dffa129/20fb8724d0aa/internal/backends/selector/lib.rs:29:57
   |
29 |             Box::new(i_slint_backend_linuxkms::Backend::new())
   |                                                         ^^^ function or associated item not found in `Backend`

With features = ["std", "compat-1-0", "renderer-linuxkms-femtovg"]

[tronical]

My bad, when I test the feature combination I still had another backend in there so I didn't notice this :(. I pushed a fix.

[flukejones]

Alright, quick test on the STM32MP157:

# ./evolve-hmi
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Other("Error reading DRM resource handles: unknown system error: EOPNOTSUPP: Operation not supported on transport endpoint")', gui/src/main.rs:38:32
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This is due to:

pub fn create_egl_display() -> Result<EglDisplay, PlatformError> {
    let drm_device = SharedFd(Arc::new(
        std::fs::OpenOptions::new()
            .read(true)
            .write(true)
            .open("/dev/dri/card0")
            .map_err(|err| format!("Error opening /dev/dri/card0: {}", err))?,
    ));

It would be best to try each available card in that directory - for my device card1 is the correct device and with this set the KMS branch runs perfectly, including touch.

Awesome work!

[flukejones]

The KMS branch running well. This also includes use of #2823 and #2810 to create the darkened overlay with popup in the right place.

[tronical]

This photo just made my day!

Yep, hardcoding /dev/dri/card0 is incorrect. I'll fix that :)

[flukejones]

Hmm, libinput error: event0 - Goodix Capacitive TouchScreen: client bug: event processing lagging behind by 81ms, your system is too slow. Unsure where to begin with debugging this. The app is very basic so far and uses very very little CPU, with a continuous touch bumping that to 4%.

A persistent issue seems to be that things aren't as smooth as I would have liked - this was true on qt-eglfs as well as weston. Even though the app does feel a bit more responsive with KMS. It manifests as looking perhaps a little like missing frames? What EGL calls does the stack make? I did some experimenting a little while ago and one GL call in particular was dead slow on this GC400 (vivante).

[tronical]

I see the libinput error on my laptop, too - I think it may be a bug in the way I implemented the event loop.

Performance wise there are two things:

1) After eglSwapBuffers we post the buffer to the screen and blockingly wait for the page flip. This could and should be done async. 2) The FemtoVG renderer is notoriously heavy on the fragment shader. I've spent a fair amount of time trying to get it to perform well on a GC7000UL, but eventually resorted to Skia. From my experience: Avoid drop-shadow-*, avoid clip: true on a Rectangle with a border-radius.

[flukejones]

Skia has been the biggest PIA to build at all. Do you have any tips? Specifically, cross-platform builds for it are the worst experience and last time I tried it fell over on ring (crypto).

I had two places with drop-shadow, removing those and the radius on a clipped rectangle worked wonders! Thank you for the pointers. Right now this appears to be faster and more responsive than the eglfs build. And it is so very much faster than our QT version with all its QML and FFI overhead.

I don't have any animations yet, so I'm entirely unsure about if it seems to drop frames, or if it's input lag that I'm seeing. I will animate something for a test tomorrow and see how it performs.

[tronical]

There is one more optimization to femtovg that I did a while ago (fast image blit if I clipped), but unfortunately it regressed. A few weeks ago I fixed it in https://github.com/femtovg/femtovg/commit/2b05b9390c03bd5d5e45f109054fb01e192e62a9 but this isn't released yet. I'll do that in the next days and then cargo update should get you that one. The screenshot looks like it's blitting images. (You'd be shocked about the amount of instructions femtovg used to apply in the fragment shader for that :)

On the Skia side: Yeah, the ring build issue I had as well. It's a little iffy to cross-compile :(, but it is possible. I documented the environment variables needed a while ago here. It gets a little more complicated with Yocto. How are you building Slint/your app?

[flukejones]

Blimey... Okay that is a noticeable improvement @tronical. Well done! (for femtovg update).

[tronical]

I suggest the following plan going forward:

1. We plan on releasing 1.1 next week - without this feature.
2. When 1.1 is out, I plan on finishing this to an MVP level and then propose it for inclusion into master.
3. Meanwhile I'll start probably moving this to a feature/linuxkms branch that'll be subject to rebasing, cleanups, but also CI build testing.

[jakerr]

Just a quick test report:

I just tried syncing to the feature/linuxkms branch.

I was able to build this for reTerminal (an Rpi CM4 development board) by building a cross image based on docker/Dockerfile.aarch64-unknown-linux-gnu but that required a few additional libraries be added in the docker file:

libudev-dev:arm64 libinput-dev:arm64 libgbm-dev:arm64

Also adding

ARG DEBIAN_FRONTEND=nointeractive

After the FROM directive was helpful when updating the docker image as sometimes it would try to stop me to ask about the timezone but hang since there's no shell attached.

I then built the printer demo with:

cross build --target aarch64-unknown-linux-gnu --workspace --exclude slint-node --release --bin printerdemo

scp'd the resulting binary to the RPI CM4 and started it via ssh with

SLINT_BACKEND=linuxkms ./printerdemo

Unfortunately the reTerminal has a quirk where noone has seemed to be able to rotate the screen to horizontal in kms mode (I guess they assume you'll use a window manager that can paper over this.) Anyhow I'm assuming if I were building an app in slint I could have slint do the 90 degree rotation for me?

One other issue I experienced is that tap targets were very hard to hit. At first I thought the screen wasn't responding to touch at all but after about 10 taps I can usally finally get it to recognize what I'm trying to hit. This may be a calibration issue, but on some screens like the "USB" screen in the printer demo where the filename targets are long enough it is quite easy to tap.

As long as the tap is recognized it actually seems very snappy. (Much snappier than the reTerminal demo apps which are made in QT python.)

[tronical]

Thanks @jakerr for testing - much appreciated.

The install steps you mentioned make sense. Now that the changes are merged into master, there's also libseat support, which means you also need libseat-dev for building. On the upside, this removes the need to run as root when logged in via a tty. (remove via ssh still requires root, at least on my setup)

Performance wise I noticed a difference between debug and release builds. With debug builds I get the warning from libinput that events are sometimes processed too slowly, but with release it feels snappy.

The touch input is something that's a little concerning. @ogoffart how well does it work on your laptop with touch screen?

[ghost]

Is keyboard support planned?

[tronical]

Yes. At the moment there's only support for ctrl-alt-delete/ctrl-alt-backspace as shortcuts to terminate the process (so that's just an escape hatch), but I'd like to support proper keyboard input. The mechanics for that are in place in calloop_backend/input.rs.

[tronical]

Olivier was kind enough to implement keyboard support today ❤️

https://github.com/slint-ui/slint/pull/3197