search

H-M-H / Weylus

Use your tablet as graphic tablet/touch screen on your computer.
Other
7.07k stars 282 forks source link

Wayland support? #3

Open zachlatta opened 4 years ago

zachlatta commented 4 years ago

Does this project support Wayland? I'm running the latest on https://aur.archlinux.org/packages/weylus-bin/ and while I have input working, I'm getting a black screen on the iPad browser I'm using it from.

Input also seems to be smooth in Xwayland apps, but jumpy in Wayland native apps.

Update: See recording of this at https://www.youtube.com/watch?v=GrJXqqLKas4. All input is made via Apple Pencil on an iPad. Notice that the input is smooth when over xeyes, but jumpy over Alacritty (which is a Wayland app).

H-M-H commented 4 years ago

Right now screen casting is not supported for Wayland. Currently the screen is captured either via autopilot, which only supports X11, or using XShmCreateImage here: https://github.com/H-M-H/Weylus/blob/435373be5141a2b03aeab6549225a5a5790ed1b7/lib/linux/capture.c .

To get Wayland to work there are some things I still need to find out:

If someone wants to help me here a minimal working example of taking a screenshot of a selected window in Wayland would be great, provided it is written in Rust, C or C++. Alternatively point me to a tool that already does that. I will probably write some Trait to abstract different window systems, so I would rather not want a pull request now.

H-M-H commented 4 years ago

About the input issues: I am not sure what's causing that, I do not do anything specific to X11. Can you tell me if libinput debug-events or evtest outputs anything strange?

Also this should probably be a separate issue.

H-M-H commented 4 years ago

Looks like the tool you used (https://github.com/ammen99/wf-recorder) for recording and https://github.com/emersion/grim can be used as a reference.

zachlatta commented 4 years ago

Hey @H-M-H, thank you for your willingness to work on Wayland support and I'm sorry I can't do more to help right now. Realistically, I don't think I'm going to be able to help you debug libinput or evtest right now. I love this project and think it's absolutely amazing. Wish I had more time to help out.

H-M-H commented 4 years ago

Alright, I totally understand that, thanks for your report anyways! Maybe someone else can help out.

H-M-H commented 4 years ago

I did some research and the situation regarding capturing the screen or even windows on wayland is pretty dire. There is no equivalent to X11's XShmGetImage or XGetImage, Wayland itself just does not support capturing the screen. Instead it is the responsibility of the compositor to expose an API to take screenshots. But unfortunately this is not standardized, there is:

pointhi commented 4 years ago

What about PipeWire?

Capture and playback of audio and video with minimal latency.

H-M-H commented 4 years ago

Interesting, I did not know about PipeWire, the docs are a bit lacking but PipeWire is used by webrtc, so a look at how they do it might help. @pointhi do you know what latency one can expect using PipeWire? For reference: capturing a 1920x1080 raw bgr0 frame on my system takes about 0.8 milliseconds if the screen contents do not change too much.

pointhi commented 4 years ago

I never used PipeWire, but it was designed for remote desktop applications on Wayland to my knowledge.

ilya-epifanov commented 3 years ago

strangely enough, unfocused windows do get captured, until the very first event sent to them which focuses them and then the output becomes black

H-M-H commented 3 years ago

I think I have a good idea of what's going on: If a window is not focused, Weylus captures it directly by its window handle but if it is active Weylus captures the part of the root window containing the window to be captured. The reason for this is to have things like context menus that are their own windows in themselves still show up. But I guess XWayland doesn't like capturing the root window and just decides to have it black.

H-M-H commented 3 years ago

Please check if things work better with the latest build: https://github.com/H-M-H/Weylus/actions/runs/585393503#artifacts

H-M-H commented 3 years ago

I started experimenting with pipewire as this seems to be the most promising solution here: https://github.com/H-M-H/Weylus/tree/wayland

For now only requesting a screen cast via org.freedesktop.portal.ScreenCast is implemented. That means there now is a file descriptor that can be used to connect to pipewire. Unfortunately the docs on how to use pipewire directly are still very lacking. For example pw_remote_connect_fd mentioned in the docs for org.freedesktop.portal.ScreenCast has been removed in pipewire 0.3 and seems to be pw_context_connect_fd now. All I could find on video processing is: https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/master/doc/tutorial5.md

It looks like pipewire implements a gstreamer plugin, I suspect gstreamer's API is not as badly documented, so this is perhaps worth a look.

A problem that remains is that it looks like there is no way to get the position of a captured window, which means Weylus won't be able to properly map the input to the window. As it stands only capturing the full desktop will be usable under Wayland.

H-M-H commented 3 years ago

For anyone interested, https://github.com/H-M-H/Weylus/commit/d632a7dfdfacd27aaa0576076f9e2d3b0b32d93c implements screen casting via gstreamer. Binaries can be found here. There still are quite some issues, refer to the commit message for details.

H-M-H commented 3 years ago

Looks like once this is resolved, working with cursor metadata via gstreamer will make dynamically showing and hiding the cursor possible.

Regarding window titles I opened a feature request here: https://github.com/flatpak/xdg-desktop-portal/issues/569

dmoltisanti commented 3 years ago

Thanks a lot for working on this :)

H-M-H commented 3 years ago

Thanks! So far walking this path has been rather rocky and I have hit quite some bugs on the way:

Bugs

Missing features

Fortunately I could workaround all PipeWire deadlocks and at least on GNOME Weylus can now capture the desktop and windows and so far it seems not too unstable.

Feel free to try the latest builds.

Considering the current situation I will definitely mark Wayland screen capture experimental and make it opt in.

l-o-l commented 3 years ago

Hi,

I've tried installing weylus on a Pi 4 running manjaro sway. Everything works other than desktop capture. I can capture chromium tabs and weylus itself, but the desktop capture returns the following warning:

WARN weylus::capturable: Failed to get list of capturables via dbus/pipewire: Failed to obtain screen capture.

Chromium itself is able to capture the desktop, as I am able to share the screen from web apps such as Microsoft Teams. Before running I set:

export XDG_SESSION_TYPE=wayland export XDG_CURRENT_DESKTOP=sway /usr/lib/xdg-desktop-portal -r & /usr/lib/xdg-desktop-portal-wlr

This works for chromium, and idea what could be done to get it to work in weylus?

H-M-H commented 3 years ago

This means that Weylus only captures windows available via XWayland. Please check if https://gitlab.gnome.org/snippets/19 also fails.

Some more questions: How exactly does the process for window/desktop selection work with chromium, do you get some kind of popup window that lets you select what to capture and does that popup window also appear for Weylus?

dron1885 commented 3 years ago

Hi, also getting WARN Failed to get list of capturables via dbus/pipewire: Failed to obtain screen capture.. No native Wayland windows and only a desktop in the Capture list. The desktop option sets the correct aspect ratio but shows only black.

This means that Weylus only captures windows available via XWayland. Please check if https://gitlab.gnome.org/snippets/19 also fails.

Yes, this snippet works.

Some more questions: How exactly does the process for window/desktop selection work with chromium, do you get some kind of popup window that lets you select what to capture and does that popup window also appear for Weylus?

Don't know how chromium does this, but firefox (for example https://mozilla.github.io/webrtc-landing/gum_test.html) and the python snippet both change the mouse cursor to a cross and I need to click on a desktop I want to share. Actually, Weylus also does this.

Also, input forwarding to desktop does work (but there is no image).

EDIT: Arch Linux, Sway WM, AMD gpu

mattpepin commented 3 years ago

In Gnome/Mutter, the whole desktop works fine (Entire screen option)! It's pretty cool! Now I wish you could use Mutter's headless stuff and create a virtual screen! :)

H-M-H commented 3 years ago

@dron1885 @l-o-l I got it working under sway now, please check if the latest build works for you as well: https://github.com/H-M-H/Weylus/actions/runs/1012629238#artifacts

@mattpepin do you know how Mutter's headless stuff works? Perhaps I could add some examples or even an accompanying script to Weylus.

dron1885 commented 3 years ago

@H-M-H Can confirm that it's working. Thanks! (But it seems it requires additional testing in multimonitor setup. From my quick testing, while Waylus was showing a second monitor all inputs were translated to the first one.)

H-M-H commented 3 years ago

Can confirm that it's working. Thanks!

Excellent, thanks for testing!

From my quick testing, while Waylus was showing a second monitor all inputs were translated to the first one.

This is a known problem and sadly not easily fixed. For monitors xdg-desktop-portal provides positions but unfortunately those are compositor dependent/the coordinate system is unspecified and there is no straightforward way to translate them into something useful. For windows the situation is even worse as there is no position given.

Unfortunately I do not see this being solved in the foreseeable future. In my opinion the whole screen recording/remote desktop situation around Wayland is most unfortunate. There have been discussions to add screen recording to Wayland for years with the result that the team behind Flatpak is now somehow responsible for this and there are no plans to extend Wayland itself or provide an official API/Protocol. This means the Flatpak devs effectively have full responsibility/control over how screen casting works. The problem now is that Flatpaks focus is not exactly to provide a nice screen cast/remote desktop interface for Wayland [1]. I can not and will not blame them for that, it is totally understandable if they deem some features required to get a nice remote desktop experience working unnecessary for Flatpak. This is why I don't quite believe in this working anytime soon and the whole discussion doesn't even touch pipewire, which is still under heavy development.

If despite this anyone knows how to obtain the positions of pipewire streams in a way useful to Weylus I very much would like to hear that though.

[1]: There actually is a remote desktop interface. But among other things it doesn't handle resizing windows or pressure sensitivity of touch input.

raffaem commented 3 years ago

Hello,

just a clarification. What does it mean that for Wayland there is no support for "capturing the cursor"?

If I move the stylus on my tablet, will the cursor move accordingly?

H-M-H commented 3 years ago

There are two problems related to this question:

raffaem commented 3 years ago

Hello,

I am able to move the cursor using the pen on my Fedora 34 Worstation, but the screen on the tablet is completely white.

How can I mirror the PC screen on the tablet?

H-M-H commented 3 years ago

You will have to make sure PipeWire with xdg-desktop-portal is correctly setup, I am not sure how this is done on Fedora. If screen sharing with your browser works, it should also work with Weylus. Perhaps this entry from the Arch Linux Wiki can help you: https://wiki.archlinux.org/title/PipeWire#WebRTC_screen_sharing

tinywrkb commented 3 years ago

Wayland capture doesn't work for me. The only time it sort of worked is when I created a virtual display, and selected it, but the video was frozen, so that wasn't usable.
In Sway, you can create a virtual display like this: $ swaymsg create_output && swaymsg output "HEADLESS-1" resolution 2048x1536, or alternatively via the config file.

H-M-H commented 3 years ago

Hmm, does this snippet: https://gitlab.gnome.org/snippets/19 work for you? Or alternatively does WebRTC screen sharing work? If one of these does work, this is most likely a problem with Weylus otherwise it's a problem with your setup.

tinywrkb commented 3 years ago

The snippet works as far as I can tell.
OBS screen capture via PipeWire works for me perfectly, and also WebRTC screen sharing in Chromium with the Wayland Ozone backend.

H-M-H commented 3 years ago

I just logged into a Sway session myself and after running

$ export XDG_CURRENT_DESKTOP=sway
$ /usr/lib/xdg-desktop-portal -r
$ /usr/lib/xdg-desktop-portal-wlr -r

Weylus displays the whole desktop without problems. Please show me the output of WEYLUS_LOG_LEVEL="DEBUG" weylus, perhaps the log contains some clues.

tinywrkb commented 3 years ago

I just logged into a Sway session myself and after running

$ export XDG_CURRENT_DESKTOP=sway
$ /usr/lib/xdg-desktop-portal -r
$ /usr/lib/xdg-desktop-portal-wlr -r

This is not needed if you override the systemd user service xdg-desktop-portal.service and set the value of XDG_CURRENT_DESKTOP as sway.

# ~.config/systemd/user/xdg-desktop-portal.service.d/override.conf
[Service]
Environment="XDG_CURRENT_DESKTOP=sway"

Weylus displays the whole desktop without problems. Please show me the output of WEYLUS_LOG_LEVEL="DEBUG" weylus, perhaps the log contains some clues.

I'm not sure how much this is helpful, but here's the log.
I also added the session bus capture, which can be viewed with Bustle and probably also Wireshark. weylus_2021-11-02T02:08:08.pcap.tar.gz

fx-chun commented 3 years ago

Weylus doesn't seem to work at all for me on Wayland, failing with this error message when I select the PipeWire output:

  WARN weylus::websocket: Failed to init screen cast: Failed to create element from factory name!

.. which looks like it's from gstreamer-rs, so I'm not entirely sure how to debug this. I verified that screen capture using xdg-desktop-portal-wlr works on my system using OBS. The name of the PipeWire output is titled Pipewire Unknow, path: XXX in the dropdown menu on the webapp, if that's of any help.

fx-chun commented 3 years ago

Seems like it's happening here in pipewire.rs:

let src = gst::ElementFactory::make("pipewiresrc", None)?;
fx-chun commented 3 years ago

Ah, looks like my distribution did not include the PipeWire GStreamer plugins correctly :) My bad!

mgonzalezm commented 2 years ago

Hello,

I'm using sway/wayland (Xwayland disabled) on Archlinux, I followed the installation instructions stated in the Readme, but i can't make it work

when starting weylus from a alacritty terminal I get WARN weylus::config: Failed to read configuration file: No such file or directory (os error 2) Can't open display: wayland-1

When run with --no-gui parameter I can connect, but it seems that weylus is still looking for X thread '<unnamed>' panicked at 'Can't open X display. Is it currently running?', /usr/local/cargo/git/checkouts/autopilot-rs-fed2c023a9e5d94a/63eed09/src/internal.rs:63:9 note: run withRUST_BACKTRACE=1environment variable to display a backtrace thread '<unnamed>' panicked at 'calledResult::unwrap()on anErrvalue: SendError { .. }', src/websocket.rs:415:60

using --wayland-support makes no difference

is Xwayland compatibility necessary?

H-M-H commented 2 years ago

@mgonzalezm Hmm, you are right. Currently Xwayland is required. The culprit for the crash is the autopilot library which I currently use as fallback if uinput or my platform dependent screen capturing code does not work. autopilot assumes an X-server is present and just hard crashes if it isn't. Please try this build here https://github.com/H-M-H/Weylus/actions/runs/1831407176#artifacts and check if it crashes as well.

mgonzalezm commented 2 years ago

Please try this build here

@H-M-H The crash is gone, but now I get: ERROR weylus::websocket: Got invalid id for capturable: 0.

I think something is missing in my setup, but i don't know what. Why i keep getting Can't open display: wayland-1 when running whithout --no-gui? doesn't wayland-1 is the default socket under wayland?

H-M-H commented 2 years ago

@mgonzalezm alright, so autopilot was indeed the cause for the crash. Have you made sure xdg-desktop-portal-wlr is up an running? You can verify whether it is running by checking if screen capturing works with OBS or Firefox.

The error message you got means that Weylus could not find any screens or windows to capture. Have you made sure to start Weylus with --wayland-support?

Regarding Can't open display: wayland-1, it looks like fltk only supports X/XWayland. I guess I'd have to rewrite the gui with another library (this wouldn't be that bad as the gui is rather minimal anyways, the actual problem is finding a nice gui library).

mgonzalezm commented 2 years ago

@H-M-H sorry I forgot --wayland-support, my bad.

and yes xdg-desktop-portal-wlr is running:

$ systemctl --user status xdg-desktop-portal-wlr ● xdg-desktop-portal-wlr.service - Portal service (wlroots implementation) Loaded: loaded (/usr/lib/systemd/user/xdg-desktop-portal-wlr.service; static) Active: active (running) since Thu 2022-02-10 19:00:34 CST; 23h ago ....

Now I can connect, and select Pipewire Unknown, path:34 on the drop-down list (this is the only option), but after few seconds:

WARN weylus::websocket: Failed to init screen cast: Element failed to change its state!

and only a black screen is shown on the browser

I think that not having a wayland compatible gui is not a big deal, as long as the same options be available on the command line.

Thank you for your help and your excellent work

H-M-H commented 2 years ago

Hmm, this error stems from within gstreamer somewhere in this call to set_state: https://github.com/H-M-H/Weylus/blob/33cf5f138b5a7eefa8cf07c71eb0403a4fce849f/src/capturable/pipewire.rs#L148 Please check if this snippet works: https://gitlab.gnome.org/-/snippets/19 or alternatively check if OBS can capture your screen. This might be a problem unrelated to Weylus.

mgonzalezm commented 2 years ago

@H-M-H Yes OBS works ok, I can see my screen.

I wouldn't now if the snippet works, after the pop-up to select the monitor to share, this is all I can see in the terminal

$ python xdp-screen-cast.py session /org/freedesktop/portal/desktop/session/1_780/u1 created sources selected streams: stream 36

H-M-H commented 2 years ago

@mgonzalezm I am sorry, I was a little busy with other things the last days. The script I linked should start recording your screen using gstreamer and then open a window that shows the recording. This typically results in some funny recursion to look at. If this is not the case this pretty much verifies that this is a gstreamer issue and not something related to pipewire only (OBS directly interfaces with pipewire, skipping gstreamer). What's the output of gst-inspect-1.0 pipewiresrc | grep Version. At least on my system using kwin_wayland (KDE) and pipewire version 0.3.47 streaming works fine using the script as well Weylus itself.

rgrunbla commented 2 years ago

Hello,

I'm facing the same problem as @noneucat, i.e. I get a 2022-05-04T16:29:01.243927Z WARN weylus::websocket: Failed to init screen cast: Failed to create element from factory name! and screen sharing does not seem to work on my OS.

I'm using NixOS, and I tried running weylus with for example GST_PLUGIN_PATH=/nix/store/v8yddpay5ilm3m59yj64pdnc13gjkav1-pipewire-0.3.49-lib/lib/gstreamer-1.0/libgstpipewire.so weylus without much success (the module seems loaded according to strace, but I still get this message).

Any idea what could cause this / how to debug ?

Thanks.

Edit: Running weylus with GST_DEBUG=5, I get more information:

0:00:15.874726515 177259 0x56307e6b5490 DEBUG     GST_PLUGIN_LOADING gstpluginfeature.c:106:gst_plugin_feature_load: loading plugin for feature 0x56307e6b8920; 'pipeline'
0:00:15.874760196 177259 0x56307e6b5490 INFO     GST_ELEMENT_FACTORY gstelementfactory.c:489:gst_element_factory_create_with_properties: creating element "pipeline"
0:00:15.874787417 177259 0x56307e6b5490 DEBUG                GST_BUS gstbus.c:256:gst_bus_init:<GstBus@0x56307e69de70> created
0:00:15.874805956 177259 0x56307e6b5490 DEBUG                    bin gstbin.c:498:gst_bin_init:<GstBin@0x7fc1bc004160> using bus <bus0> to listen to children
0:00:15.874822389 177259 0x56307e6b5490 DEBUG                GST_BUS gstbus.c:256:gst_bus_init:<GstBus@0x56307e69df30> created
0:00:15.874842136 177259 0x56307e6b5490 DEBUG               GST_POLL gstpoll.c:681:gst_poll_new: 0x56307e6a0e80: new controllable : 1
0:00:15.874863422 177259 0x56307e6b5490 DEBUG               GST_POLL gstpoll.c:848:gst_poll_add_fd_unlocked: 0x56307e6a0e80: fd (fd:29, idx:0)
0:00:15.874872373 177259 0x56307e6b5490 DEBUG               GST_POLL gstpoll.c:1014:gst_poll_fd_ctl_read_unlocked: 0x56307e6a0e80: fd (fd:29, idx:0), active : 1
0:00:15.874879547 177259 0x56307e6b5490 DEBUG                GST_BUS gstbus.c:310:gst_bus_new:<bus1> created new bus
0:00:15.874887630 177259 0x56307e6b5490 DEBUG          GST_PARENTAGE gstelement.c:3478:gst_element_set_bus_func:<GstPipeline@0x7fc1bc004160> setting bus to 0x56307e69df30
0:00:15.874895376 177259 0x56307e6b5490 DEBUG               pipeline gstpipeline.c:248:gst_pipeline_init:<GstPipeline@0x7fc1bc004160> set bus <bus1> on pipeline
0:00:15.874904199 177259 0x56307e6b5490 DEBUG    GST_ELEMENT_FACTORY gstelementfactory.c:522:gst_element_factory_create_with_properties: created element "pipeline"
0:00:15.874921808 177259 0x56307e6b5490 WARN     GST_ELEMENT_FACTORY gstelementfactory.c:701:gst_element_factory_make_with_properties: no such element factory "pipewiresrc"!
0:00:15.874930773 177259 0x56307e6b5490 DEBUG        GST_REFCOUNTING gstpipeline.c:258:gst_pipeline_dispose:<pipeline0> 0x7fc1bc004160 dispose
0:00:15.874945344 177259 0x56307e6b5490 DEBUG        GST_REFCOUNTING gstbin.c:517:gst_bin_dispose:<pipeline0> 0x7fc1bc004160 dispose
0:00:15.874968024 177259 0x56307e6b5490 INFO         GST_REFCOUNTING gstelement.c:3382:gst_element_dispose:<pipeline0> 0x7fc1bc004160 dispose
0:00:15.874976511 177259 0x56307e6b5490 DEBUG               GST_POLL gstpoll.c:772:gst_poll_free: 0x56307e6a0e80: freeing
0:00:15.874992833 177259 0x56307e6b5490 INFO         GST_REFCOUNTING gstelement.c:3428:gst_element_dispose:<pipeline0> 0x7fc1bc004160 parent class dispose
0:00:15.875000638 177259 0x56307e6b5490 INFO         GST_REFCOUNTING gstelement.c:3460:gst_element_finalize:<pipeline0> 0x7fc1bc004160 finalize
0:00:15.875007630 177259 0x56307e6b5490 INFO         GST_REFCOUNTING gstelement.c:3465:gst_element_finalize:<pipeline0> 0x7fc1bc004160 finalize parent
2022-05-06T09:50:27.815216Z  WARN weylus::websocket: Failed to init screen cast: Failed to create element from factory name!

Full log: log.txt

H-M-H commented 2 years ago

@rgrunbla: Thanks for looking into this! GST_PLUGIN_PATH should point to the directory with the plugin, not the shared library itself. What is the output of gst-inspect-1.0 pipewiresrc?

Also, if this problem persists, please open a separate issue.

Dmarc commented 2 years ago

Maybe this solution using vdl and deskreen could be inspiring for Wayland support ?

tutacat commented 1 year ago

@H-M-H @zachlatta This issue should be closed as pipewire support has been added

Iey4iej3 commented 1 year ago

It seems to me that, under Wayland (I am using sway), one cannot select a specific window, but "Pipewire unknown", which shows the current desktop. I am not sure whether it is possible to capture a specific window under Wayland, however.

tutacat commented 1 year ago

It seems to me that under Wayland one cannot select a specific window, but "Pipewire unknown", which shows the current desktop. I am not sure whether it is possible to capture a specific window under Wayland, however.

No, it can't. For security, you select the window only in the pipewire popup. The program can't access every window instantly, and without confirmation. Weylus isn't forwarding the window title, but that's not a problem. To change the window, you can use the 'refresh sources' button, or reload the page, then choose the window you want to mirror on your screen.

For most people, it may be easier to just mirror the whole display. With X11 there are normally no controls to what programs can access on the same server. Giving access to the monitor through Pipewire doesn't give access to all windows.

You may still be able to access X11 windows directly through Weylus, but you won't be able to access Wayland windows.