Wayland doesn't work (with xwayland)

[dphaldes]

This is a continuation of #11. I have received the updated version of XWayland and focus pass-through well. Too well to be honest, as the egui windows get no input at all. No Mouse movements or clicks. The basic example's mouse position display don't change, so I am assuming it is not getting any mouse input. XWayland Version 24.1-1 Arch Linux Plasma 6.0.4 on Wayland

[coderedart]

If we don't get any mouse event during this frame, we get the global mouse position and push a manual mouse move event into egui at https://github.com/coderedart/egui_overlay/blob/master/crates/egui_window_glfw_passthrough/src/lib.rs#L593

You can try adding print statements at that location, to see the values of window size (bounds), global cursor position and previous cursor position.

Wayland doesn't support getting global cursor position (by design), so glfw just returns 0,0 as the global cursor position. This could be the problem.

If you edited any kwin rules in the past, you should check and remove them as one of those rules can prevent focusing a window. Again, its a maybe.

Also, post a tiny video of the issue. Try show the Vec from the glfw struct in egui, so we can see the events live while cursor is moving.

ui.label(&format!("{:?}", &glfw_backend.frame_events));

If possible, We should probably move to discord (egui or rust official or rust community) where we can chat live. This asynchronous github conversation will take forever :)

[phddoom]

Has there been any progress on this issue? I think I am having the same or similar issue and would be glad to help provide any output or help need to get this working on Plasma Wayland or figure out if this is an XWayland bug. Thank you

[coderedart]

You can try posting a video showing the problem. make sure to print events from https://docs.rs/egui_window_glfw_passthrough/0.8.1/egui_window_glfw_passthrough/struct.GlfwBackend.html#structfield.frame_events every frame to the terminal.

It will help if we could see the events that are triggered (focus and mouse position) in X11 vs Wayland sessions.

[phddoom]

Sorry for the late reply, I was away from internet for awhile, I should be more reachable going forward.

You can try posting a video showing the problem. make sure to print events from https://docs.rs/egui_window_glfw_passthrough/0.8.1/egui_window_glfw_passthrough/struct.GlfwBackend.html#structfield.frame_events every frame to the terminal.

It will help if we could see the events that are triggered (focus and mouse position) in X11 vs Wayland sessions.

Is this the best way to print events: ui.label and include that in the video?

ui.label(&format!("{:?}", &glfw_backend.frame_events));

I tried that approach before commenting without recording and it seemed to print either nothing or an empty array and then nothing. I thought I might have not correctly modified the example so I want to confirm the correct way to print the events so you get the best debugging output.

Thank you

[coderedart]

Sorry for the late reply :) I just got back from a trip too.

you can just print to the terminal directly and make sure the video shows the terminal output.

[phddoom]

Here are two videos one in KDE Plasma X11 and one in KDE Plasma Wayland

x11.webm wayland.webm

Let me know if you need anything else

[coderedart]

As I guessed in https://github.com/coderedart/egui_overlay/issues/37#issuecomment-2128586865

Wayland doesn't support getting global cursor position (by design), so glfw just returns 0,0 as the global cursor position. This could be the problem.

I already debugged this issue in the past, but made an issue (and a README warning) on my personal project which used this crate, but forgot to update that info on this repo. I transferred this issue to this project at #38 for better tracking. I will also update the README later (or someone else can do it <3).

For a brief summary, the way overlay works is setting a passthrough flag inside glfw. For Windows and Mac, there's an explicit feature for that.

For X11/Wayland, we have a "set region" function which can take an arbitrary number/combo of rectangles and only allow input to work on that area. We simply set an empty rectangle as region, and input passes through.

On Mac/Windows/X11, you can get global cursor position, check if its within window / "egui areas" using ctx.wants_mouse/wants_keyboard and toggle the flag based on egui's reply.

With wayland, you only get cursor position if its focused on your window (for "security" reasons as stated by wayland devs). So, we just get [0,0] (or empty) as long as we are not focused and we can't really check if cursor is over our egui area which means we will never set passthrough to false. (unless user explicitly calls ctx.want_mouse/keyboard to manually make it true without cursor position).

We got three ways forward for wayland:

1. First, this is technically an Xwayland bug because they aren't simulating the input region (xshape extension of X11) properly. But idk if there's already a bug report raised.
2. You can try registering a global shortcut like ctrl + shift + Y with dbus (zbus crate will pull in 170-ish dependencies including async-std with long compile times. be careful with that) or inputbot (introduces keylogger concerns, but crossplatform win/linux). Whenever user presses that shortcut, toggle your passthrough flag yourself (ctx.wants_mouse something).
3. just tell user to use alt-tab to switch to overlay.
4. request egui to expose its "areas" publicly. If egui exposed the areas where it needed input(we might probably already get it from ctx.data some anymap thing, I gotta look at it later), we could just set that as input region using wayland crate directly and skip this global cursor/passthrough toggle ceremony specifically on wayland. This also skips xwayland, as we will use wayland directly.

plasma is still buggy on my pc, so it will take a while before I add wayland support. But if someone's willing to contribute, that would be great.

The easiest method is number 4. get egui's areas and get window id from glfw. call the wayland set input region fn, and do that whenever egui areas change.

[coderedart]

I just checked, and with latest plasma wayland + xwayland, it works fine. It provides mouse position (and window position) as X11 is supposed to do. re-open the issue if there's any problems.