search

flatpak / xdg-desktop-portal

Desktop integration portal
https://flatpak.github.io/xdg-desktop-portal/
GNU Lesser General Public License v2.1
563 stars 188 forks source link

Add a portal to draw overlays ("always on top" surfaces, e.g. picture-in-picture) #612

Closed teohhanhui closed 1 year ago

teohhanhui commented 3 years ago

(Continued from https://bugzilla.mozilla.org/show_bug.cgi?id=1621261#c29)

@unrelentingtech:

layer-shell is designed for desktop elements, it is intended as a privileged protocol and so should not be used for PiP.

We need to get everyone to agree on a PiP protocol specifically, with the appropriate restrictions and simple interface.

It feels to me like a PiP protocol would be too specific and not generalizable to other use cases.

As a user I'd definitely be expecting this to be in xdg-desktop-portal instead, much like org.freedesktop.portal.ScreenCast would prompt the user.

The Android equivalent: https://developer.android.com/reference/android/view/WindowManager.LayoutParams#TYPE_APPLICATION_OVERLAY

NeatSketch commented 3 years ago

The Android equivalent: https://developer.android.com/reference/android/view/WindowManager.LayoutParams#TYPE_APPLICATION_OVERLAY

Besides overlays, Android also explicitly supports PiP (which is more restrictive). Similar separation may make sense here too. https://developer.android.com/guide/topics/ui/picture-in-picture

valpackett commented 3 years ago

And explicit PiP on Android (and iOS?) does not prompt for permissions → we don't need portals for it, it should be an unprivileged Wayland protocol.

NeatSketch commented 3 years ago

And explicit PiP on Android (and iOS?) does not prompt for permissions → we don't need portals for it, it should be an unprivileged Wayland protocol.

The user can disallow PiP per app on Android, although the PiP permission is granted by default.

I agree with the Android platform that PiP usage should be priviledged, but it can be an opt-out feature.

teohhanhui commented 3 years ago

The user can disallow PiP per app on Android, although the PiP permission is granted by default.

I agree with the Android platform that PiP usage should be priviledged, but it can be an opt-out feature.

I guess that's something that can be in e.g. GNOME Settings (under Applications or Privacy).

matthiasclasen commented 3 years ago

Always-on-top and picture-in-picture sounds somewhat contradictory

teohhanhui commented 3 years ago

Always-on-top and picture-in-picture sounds somewhat contradictory

Picture-in-picture means a smaller video playing as a sub-frame within the whole display's output. Like a screen within a screen. So it should be always-on-top by definition.

matthiasclasen commented 2 years ago

Either it is in-picture (then it should be covered when I raise a window on top of the main picture it is in), or it is always-on-top, then it will stay put when I move the main window around. I just don't see how the two go together.

But regardless of that terminology nitpicking, I am very uncertain that this is really portal material - there's no underlying desktop functionality (as far as I know) that backends could use to implement this.

swick commented 2 years ago

then it should be covered when I raise a window on top of the main picture it is in

Ah, the main picture is the whole output so you're not able to raise anything on top of it. Weird terminology indeed but it's used widely so better stick to it.

But regardless of that terminology nitpicking, I am very uncertain that this is really portal material - there's no underlying desktop functionality (as far as I know) that backends could use to implement this.

The problem is that on the wayland side implementing a PIP surface role would make it available to every client since there is no authentication mechanism in place. We would either have to live with that or get authentication working with wayland or get some token (probably a fd) from a portal as proof.

valpackett commented 2 years ago

I think authentication shouldn't be required for this. Instead, compositors should implement restrictions like size limits, no-transparency and no-click-through (mandatory full-surface input region) for PiP surfaces. Heck, this would be incredibly difficult to use for shady things without absolute window positioning and quiet screenshotting anyway.

rmader commented 2 years ago

Either it is in-picture (then it should be covered when I raise a window on top of the main picture it is in), or it is always-on-top, then it will stay put when I move the main window around. I just don't see how the two go together.

Window managers would refer to it as "always on top" floating windows while browsers, both Firefox and Chrome, call it PiP for some reason :/

I think authentication shouldn't be required for this. Instead, compositors should implement restrictions like size limits, no-transparency and no-click-through (mandatory full-surface input region) for PiP surfaces. Heck, this would be incredibly difficult to use for shady things without absolute window positioning and quiet screenshotting anyway.

Mind giving it a try (writing a Wayland protocol)?

valpackett commented 2 years ago

Yeah, I guess I could prototype this and submit a protocol. I'd like to hear if there are any objections to the idea though.

swick commented 2 years ago

I think authentication shouldn't be required for this. Instead, compositors should implement restrictions like size limits, no-transparency and no-click-through (mandatory full-surface input region) for PiP surfaces. Heck, this would be incredibly difficult to use for shady things without absolute window positioning and quiet screenshotting anyway.

Maybe that will be sufficient. Either way, not a big deal. There already is plenty of ways to wreak havoc with xdg shell surface roles.

Yeah, I guess I could prototype this and submit a protocol. I'd like to hear if there are any objections to the idea though.

Such a protocol will probably be very uncontroversial. Submit a MR early so people get a rough idea and can give feedback early on. You could also ask people on the list here https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/MEMBERS.md but it's likely faster and easier to get a rough prototype out of the door.

rohmishra commented 2 years ago

Either it is in-picture (then it should be covered when I raise a window on top of the main picture it is in), or it is always-on-top, then it will stay put when I move the main window around. I just don't see how the two go together.

@matthiasclasen Unless we are breaking user expectations, They need to be both. A "PiP" window should be displayed Always on top and on all workspaces regardless of which application is actually active. This is the convention followed by Android, iOS/iPadOS, macOS. On windows you just create a new window with those properties enabled. This is also how it is done on X. However this solution no longer works on wayland.

And explicit PiP on Android (and iOS?) does not prompt for permissions → we don't need portals for it, it should be an unprivileged Wayland protocol.

@unrelentingtech On iOS and android, the PiP window can only display content and declare which buttons it wants (play/pause, next, previous, repeat).The window will have a close/back-to-main-window button always. The app should not be able to hide or remove this. The app cannot intercept any input directly. Also, the system is in charge of the size and position of the window. This is something we might want to replicate with a specialized PIP API. Note PIP can be used for displaying more than just media content or for calls. On android apps like maps and some delivery apps will show a map for navigation or to track delivery.

I think authentication shouldn't be required for this. Instead, compositors should implement restrictions like size limits, no-transparency and no-click-through (mandatory full-surface input region) for PiP surfaces. Heck, this would be incredibly difficult to use for shady things without absolute window positioning and quiet screenshotting anyway.

IMHO the system should be responsible for default positioning like on Android/iOS. We would also want to avoid providing clicks to app windows using this. The buttons should probably be handled by your shell. App should only be able show content in any window created with "PiP" flags and rely on rely on other APIs to ask your shell to add buttons/progress-bar.

Here is a list of all options or controls an app might want to request/provide that i could come up with:

GeorgesStavracas commented 1 year ago

I tend to agree with @matthiasclasen - this doesn't seem like portals material. There's been discussions on Wayland about a protocol extension to cover this use case too. I'll close this issue for now, under the assumption that there won't be anything to be done on portals side, but if later on we figure a portal addition is needed, we can reopen this issue.