search

OpenBoard-org / OpenBoard

OpenBoard is a cross-platform interactive whiteboard application intended for use in a classroom setting.
https://openboard.ch/
GNU General Public License v3.0
2.4k stars 429 forks source link

Request: Support screenshot in Desktop mode under Wayland #679

Open letsfindaway opened 2 years ago

letsfindaway commented 2 years ago

Under Linux/Wayland, the screenshot feature in Desktop mode just creates black rectangles. See #665. The reason for this behavior is, that Wayland does not allow to grab arbitrary region of the screen without explicit user consent. This also affects mirroring of the desktop in multi-monitor setups and podcast recording.

Instead of using QScreen::grabWindow(), we have to use a DBus service to take a snapshot.

Pointers to documentation

Flatpak

Flatpak is a containerized application format. To take screenshots from the sandbox, the org.freedesktop.portal.Screenshot interface is specified. Some pointers:

Others

The portal service described for Flatpak seems to be the basic mechanism, which should also work on Gnome, KDE and others.

Discussions

Here some pointers to discussions, even if they don't target C++ or Qt:

letsfindaway commented 2 years ago

First experiences

I made some initial tests with the desktop portal with the following findings:

This might be ok for the use case of a screenshot in desktop mode, as requested in the issue #665. It is however not suited for other use cases like screen mirroring or podcast recording. For such use cases there is also a ScreenCast function available in the desktop portal. This function creates a PipeWire stream containing the screen cast. Using this function will therefore add a dependency to the PipeWire client library and needs knowledge on how to process the stream data.

An interesting approach for a similar problem is taken by the screencast program "vokoscreenNG". Here the GStreamer API is used to process the stream created by the portal. See the code in https://github.com/vkohaupt/vokoscreenNG/tree/3.0/src/wayland. Instead of processing the frames, this code just plugs the existing building blocks together to create an encoding pipeline.

letsfindaway commented 2 years ago

After reading some more discussions about this including the experiences of the Flameshot maintainer I'm not so sure whether this feature will arrive soon in OpenBoard. Here are some links:

Summary from my point of view: Creating screenshots through the desktop portal requires cooperation of several parts: The team around the desktop portal, but also the teams around KDE, Gnome and other desktop environments, which in the end have to implement the functions behind. Each of them has its own point of view. And the discussion has not ended. There are now implementations in the field where each screenshot has to be acknowledged by the user. Remember that OpenBoard uses screenshots also to mirror the desktop view to the second monitor, and for that makes screenshots several times per second.

Ok, for that we should use the also existing screencast portal. Agreed. This pulls in a dependency on gstreamer to process the video stream. Remember, OpenBoard is multiplatform: Windows, MacOS, Linux. Currently the platform specific parts are quite small, thanks to Qt's platform abstraction. With that, the platform specific part needed for Wayland on Linux would get much, much bigger.

One problem with the desktop portal is also that freedesktop.org does not prescribe or define the behavior when invoking the API. This is left to the implementation. So when using the portal you can never be sure what actually happens to the user. There have e.g. been proposals to ask the user only once, and then to create an acoustic feedback or a screen flicker each time a screenshot is taken.

In the end the desktop portal needs a complete different implementation for screen capturing or screencast, especially for desktop mode:

And in the end it is not clear that an implementation can work on a wide range of distributions and versions, as things around the desktop portal are still moving.

So I assume, the recommendation will be to use X11 for quite some time. Wayland might probably be reconsidered if it is absolutely necessary for the use case of the Geneva schools and/or if the desktop portal is useful enough to support OpenBoard's use cases on a broad range of distributions and desktop environments and/or Qt provides an abstraction of this interface, hiding the platform specific details. This is currently not the case.

But again: this is just my point of view. For the moment I stop investigating this further.

letsfindaway commented 4 months ago

Now, nearly two years later, I continue to work on this topic, as it will hit us at some point in time anyway. I currently make more experiments, which give me more insights.

There are now implementations in the field where each screenshot has to be acknowledged by the user. Remember that OpenBoard uses screenshots also to mirror the desktop view to the second monitor, and for that makes screenshots several times per second.

At least on KDE Plasma you do not have to acknowledge a screenshot at all. Let's test it on Gnome!

Ok, for that we should use the also existing screencast portal. Agreed. This pulls in a dependency on gstreamer to process the video stream. Remember, OpenBoard is multiplatform: Windows, MacOS, Linux. Currently the platform specific parts are quite small, thanks to Qt's platform abstraction. With that, the platform specific part needed for Wayland on Linux would get much, much bigger.

gstreamer can replace ffmpeg for podcast encoding. Then in the end it is just replacing the dependency to ffmpeg by another to gstreamer. And the code for a video encoder with gstreamer is much smaller than it was for ffmpeg, because the gstreamer API is on a little bit higher level.

One problem with the desktop portal is also that freedesktop.org does not prescribe or define the behavior when invoking the API. This is left to the implementation. So when using the portal you can never be sure what actually happens to the user. There have e.g. been proposals to ask the user only once, and then to create an acoustic feedback or a screen flicker each time a screenshot is taken.

Agreed and still valid. But let's just check Gnome and KDE and then see how big this issue is,

In the end the desktop portal needs a complete different implementation for screen capturing or screencast, especially for desktop mode:

  • Capturing part of the screen
  • Screen mirroring
  • Podcast

And in the end it is not clear that an implementation can work on a wide range of distributions and versions, as things around the desktop portal are still moving.

I'm not so pessimistic any longer. Screen mirroring with the ScreenCast Desktop Portal works nicely, and also Podcast recording can easily adapt to gstreamer.

So I assume, the recommendation will be to use X11 for quite some time. Wayland might probably be reconsidered if it is absolutely necessary for the use case of the Geneva schools and/or if the desktop portal is useful enough to support OpenBoard's use cases on a broad range of distributions and desktop environments and/or Qt provides an abstraction of this interface, hiding the platform specific details. This is currently not the case.

I do not see that Qt is going to abstract that. But the Desktop Portal and gstreamer provide enough to pave a way for the development of OpenBoard to work with Wayland.

But again: this is just my point of view. For the moment I stop investigating this further.

I just started it again and will report back when I have more results on that.