search

w3c / picture-in-picture

Picture-in-Picture (PiP)
https://w3c.github.io/picture-in-picture
Other
311 stars 42 forks source link

disablePictureInPicture interoperability #184

Open mounirlamouri opened 4 years ago

mounirlamouri commented 4 years ago

We should start looking at what are the issues we need to resolve in order to publish our CR. @beaufortfrancois could you triage the issues that are vNext (thinking of element-based Picture-in-Picture), and start gathering folks around the ones that need to be solved for CR?

mounirlamouri commented 4 years ago

FWIW, I think we should resolve the naming for the pseudo-class but on top of my head, this may be the only critical change we would have to do.

mounirlamouri commented 4 years ago

Is there any issue/feature that the group would like to see added/fixed in order to go to CR?

kened-ferreira commented 4 years ago

Are you planning to allow multiple picture-in-picture?

mounirlamouri commented 4 years ago

I don't expect the specification to allow that given that every system I'm aware of with native picture-in-picture do not allow multiple picture-in-picture.

kened-ferreira commented 4 years ago

It would be really useful. I have to use multiple browsers simultaneously to have multiple pips.

kened-ferreira commented 4 years ago

I don't expect the specification to allow that given that every system I'm aware of with native picture-in-picture do not allow multiple picture-in-picture.

The limitation you mentioned refers to each instance of the app. I can run, for example, two instances of vlc (or bsplayer), so I can have two pip. What if each browser tab is treated as an instance? I think the limitation imposed by the OS would be overcome.

tidoust commented 4 years ago

Are you planning to allow multiple picture-in-picture?

For reference, I note that @kened-ferreira raised this in #183.

beaufortfrancois commented 4 years ago

@mounirlamouri I've triaged all issues and PRs for Picture-in-Picture.

Have a look at what's left at https://github.com/w3c/picture-in-picture/issues?q=-label%3Av2+is%3Aopen+

beaufortfrancois commented 4 years ago

@jernoble I've noticed some web platform tests for Picture-in-Picture are failing: https://wpt.fyi/results/picture-in-picture?label=master&label=experimental&product=safari&q=status%3A%21pass%20and%20status%3A%21ok

Here are my notes:

jernoble commented 4 years ago
  • disablePictureInPicture is present in the IDL but not actually implemented. It should be pretty easy to add.

This is intentional. As I noted when this property was proposed, WebKit‘s position is that sites should not be allowed to block the user from entering picture-in-picture if the user so chooses. We don’t plan on implementing this.

  • A video whose src is the result of canvas.captureStream() does not update its ready_state, causing some mediastream related wpt tests to fail.

Ok, we’ll look into this.

  • Requesting Picture-in-Picture should consumes user gesture, it does not. Intended behaviour?

WebKit hasn’t implemented the User Activation API, where “consuming a user gesture” is defined. If there’s a failing test here, it should probably be a part of the activation test suite, and not PiP specifically.

mounirlamouri commented 4 years ago
  • disablePictureInPicture is present in the IDL but not actually implemented. It should be pretty easy to add.

This is intentional. As I noted when this property was proposed, WebKit‘s position is that sites should not be allowed to block the user from entering picture-in-picture if the user so chooses. We don’t plan on implementing this.

@tidoust what are the options in case like this where we have two browsers incompatibilities? AFAIK, we could mark this feature as OPTIONAL, but the IDL will still be there so not having the property would be incorrect. The alternative would be to drop the feature from the spec entirely. Is that correct?

  • Requesting Picture-in-Picture should consumes user gesture, it does not. Intended behaviour?

WebKit hasn’t implemented the User Activation API, where “consuming a user gesture” is defined. If there’s a failing test here, it should probably be a part of the activation test suite, and not PiP specifically.

https://html.spec.whatwg.org/multipage/interaction.html#consume-user-activation

tidoust commented 4 years ago

@tidoust what are the options in case like this where we have two browsers incompatibilities? AFAIK, we could mark this feature as OPTIONAL, but the IDL will still be there so not having the property would be incorrect. The alternative would be to drop the feature from the spec entirely. Is that correct?

That is correct. Just to clarify, with OPTIONAL, you mean making the effect of the IDL attribute optional, not making the IDL attribute itself optional, right?

(Making the IDL attribute optional seems a bad idea in any case, as that would require web applications to check on the attribute's presence before they set it.)

Setting the disablePictureInPicture attribute may already have no effect given that the spec uses a SHOULD. If the specification gets completed to note that a user agent may choose not to honor the attribute's value at all, would Webkit be amenable to expose the (essentially inert) IDL attribute, @jernoble ?

In any case, from a process perspective, it may be wise to mark the feature "at risk" (see CR section in Process document) if it remains in the spec. If WebKit exposes the attribute without supporting its optional behavior, I believe that we would still need two independent implementations that actually disable PiP when the attribute is set.

marcoscaceres commented 4 years ago

That is correct. Just to clarify, with OPTIONAL, you mean making the effect of the IDL attribute optional, not making the IDL attribute itself optional, right?

Making the IDL attribute is not ideal. We should leave it (expecting that it's a hint), and expect it to reflect the HTML attribute.

Setting the disablePictureInPicture attribute may already have no effect given that the spec uses a SHOULD. If the specification gets completed to note that a user agent may choose not to honor the attribute's value at all, would Webkit be amenable to expose the (essentially inert) IDL attribute, @jernoble ?

Not @jernoble, but from the Gecko side, I think we would be happy for the attribute to go away entirely. If it remains, I imagine Firefox would still allow users to bring up a context menu, and select "PiP it!" on any video. I feel bad that we will be lying to developers (or that developers may start doing annoying things, like putting divs on-top of videos to prevent the right context menu from appearing), but users should be able to access this feature irrespective of what the site wants.

In any case, from a process perspective, it may be wise to mark the feature "at risk" (see CR section in Process document) if it remains in the spec. If WebKit exposes the attribute without supporting its optional behavior, I believe that we would still need two independent implementations that actually disable PiP when the attribute is set.

Agree. I think it might be good to just mark this "at risk" for now.

jernoble commented 4 years ago

I feel bad that we will be lying to developers (or that developers may start doing annoying things, like putting divs on-top of videos to prevent the right context menu from appearing), but users should be able to access this feature irrespective of what the site wants.

I strongly agree. Sites who implement custom playback controls can (and do) already make it difficult to enter PiP. Enforcing this user-hostile behavior at the User-Agent level should be a non-goal, IMO.

mounirlamouri commented 4 years ago

Dropping the agenda label as this was discussed at the previous meeting. If you all think it should be discussed again, feel free to add the label back.

mounirlamouri commented 4 years ago

I strongly agree. Sites who implement custom playback controls can (and do) already make it difficult to enter PiP. Enforcing this user-hostile behavior at the User-Agent level should be a non-goal, IMO.

FWIW, we've added that because we received feedback from developers that they wanted to prevent some videos to be played in Picture-in-Picture. Because the API would allow a website to detect that a video has entered PIP and the website can then call exitPictureInPicture(), it seemed that we would either have to play a cat and mouse game that would hurt the platform in some ways like not exposing that the video is in PIP or hurt the user by offering PIP when the website would leave PIP immediately. Chrome offers Picture-in-Picture in 3 different places in the UI, we want to make sure that if the user triggers PIP, it works.

marcoscaceres commented 4 years ago

Because the API would allow a website to detect that a video has entered PIP and the website can then call exitPictureInPicture()

I wonder then if "enterpictureinpicture" should only fire as a result of .requestPictureInPicture()?

Then .exitPictureInPicture() should only work on site initiated .requestPictureInPicture(), instead of user initiated ones. The sites shouldn't know when the user puts a video in PiP mode.

mounirlamouri commented 4 years ago

I do not think we should keep the websites in the dark about what's happening. This would be harmful to the platform because it would have a lot of side effects. One on top of my head: we are exposing the size of the PIP window so websites can do ABR and now I website may send a full HD video to a tiny PIP video, wasting resources on the client and server side. I'm sure we could find others.

jernoble commented 4 years ago

Then .exitPictureInPicture() should only work on site initiated .requestPictureInPicture(), instead of user initiated ones. The sites shouldn't know when the user puts a video in PiP mode.

An alternative would be to require exitPictureInPicture() to be called from a user gesture, which ensures that "exit PiP" is an action performed by the user.

mounirlamouri commented 4 years ago

That would have a lower cost but would have still a cost and possible backward compatible issues. To be clear, these are the type of things I wanted to avoid by simply letting websites have control. If Firefox and Safari see problems with websites actively blocking PIP sessions triggered by the UA, maybe we should start looking at solutions at that point. There will be no good solutions but only bandaid that will make the problem less painful and here taking a move to block abuse is important and preempting moves may lead to a worse result faster. For example, exitPictureInPicture() requiring a user gesture has a cost in backward compat, probably a cost for some reasonable usage but most importantly, I'm sure Chrome isn't the only UA to auto-exit Picture-in-Picture in some cases. A website could find one of these cases to exit Picture-in-Picture to workaround the limitation (in the case, of Chrome scr='' exits PIP, I believe Safari exits PIP when the element is removed from the DOM).

eric-carlson commented 4 years ago

There will be no good solutions but only bandaid that will make the problem less painful and here taking a move to block abuse is important and preempting moves may lead to a worse result faster. For example, exitPictureInPicture() requiring a user gesture has a cost in backward compat, probably a cost for some reasonable usage but most importantly, I'm sure Chrome isn't the only UA to auto-exit Picture-in-Picture in some cases. A website could find one of these cases to exit Picture-in-Picture to workaround the limitation (in the case, of Chrome scr='' exits PIP, I believe Safari exits PIP when the element is removed from the DOM).

While we can't prevent sites from being user hostile, but we should not make it easier.

marcoscaceres commented 4 years ago

Requiring a gesture to exit seems like a reasonable compromise. If sites do things like setting src="" or removing the video from the DOM, they may end up compromising their own user experience (but that's kinda on them). So I agree with @eric-carlson, we shouldn't make it easier and should encourage sites to the right thing by designing the API in a way that enforces best practice.

marcoscaceres commented 4 years ago

Just to follow up, we discussed this internally in Mozilla a little bit... if we end up implementing, this is what we will likely do: if the user initiated PiP, the site won't be notified at all that it's in PiP mode. However, if the user enables PiP through the site's UI, then events will fire etc. Requiring a gesture to also exit still seems like a good idea.

beaufortfrancois commented 3 years ago

Any updates on this @jernoble?

  • A video whose src is the result of canvas.captureStream() does not update its ready_state, causing some mediastream related wpt tests to fail.

Ok, we’ll look into this.

pliu6 commented 3 years ago

We are working on it and will fix it in a future release.

By the way, I think the issue is not directly related to the picture-in-picture implementation, and we can fix (work around) the test failure by changing the test.

Any updates on this @jernoble?

  • A video whose src is the result of canvas.captureStream() does not update its ready_state, causing some mediastream related wpt tests to fail.

Ok, we’ll look into this.

beaufortfrancois commented 3 years ago

By the way, I think the issue is not directly related to the picture-in-picture implementation, and we can fix (work around) the test failure by changing the test.

Feel free to update tests at https://github.com/web-platform-tests/wpt/tree/master/picture-in-picture. I'll be happy to review them.

beaufortfrancois commented 3 years ago

This is intentional. As I noted when this property was proposed, WebKit‘s position is that sites should not be allowed to block the user from entering picture-in-picture if the user so chooses. We don’t plan on implementing this.

For info, this causes some real UX issues as websites stop Picture-in-Picture after entering it to disallow it. See https://twitter.com/firt/status/1307083085666750468 and https://www.theverge.com/2020/9/18/21445912/youtube-videos-website-picture-in-picture-ios-14-google-apple-premium

cc @jernoble

jernoble commented 3 years ago

And that UX issue is entirely the fault of the web site, and not the user or the user agent. Blocking Picture-in-Picture is no less user hostile than breaking Picture-in-Picture; it just makes the user agent culpable in that hostility.

eric-carlson commented 3 years ago

For info, this causes some real UX issues as websites stop Picture-in-Picture after entering it to disallow it. See https://twitter.com/firt/status/1307083085666750468 and https://www.theverge.com/2020/9/18/21445912/youtube-videos-website-picture-in-picture-ios-14-google-apple-premium

"Websites do terrible things, so lets add API to make it easier"?

beaufortfrancois commented 3 years ago
  1. The disablepictureinpicture HTMLVideoElement attribute is a hint for the user agent that the website communicates to basically say this video is not meant to go into Picture-in-Picture. 360 videos for instance are one example where it's not suitable for the user to see the "raw" video.

image

  1. The disableremoteplayback HTMLMediaElement attribute in the Remote Playback API is implemented in WebKit and is quite similar.
edwardchanjw commented 3 years ago

Do we wanna to have alternative API, Is the PIP is enable/disable on the page etc. For example: isPictureInPictureActive

Potentially for YouTube platform, I am notice YouTube might wanna to ask User after less than hour of listening, Would hope if any customized good video playing app (PIP type) potentially created for multi-"tasking entertainment", would allow the user to have more inactive hours.

Video paused. Continue watching Could be an API for check above and One PIP Windows only?

chrisn commented 1 year ago

Revisiting this issue, as I'd like us to move the spec forward to CR if we can. Reading through the discussion here, there isn't consensus on disablePictureInPicture, but we haven't come to a resolution. Options discussed:

Has anything changed since this was last discussed?

I recommend doing another triage for newer issues raised also contacting Mozilla for update on their position.

cc @beaufortfrancois @jernoble

chrisn commented 1 year ago

As the discussion has focused on disablePictureInPicture, I'll rename this issue and raise a new one for the steps to publish CR.

chrisn commented 3 months ago

There's a spec inconsistency: https://w3c.github.io/picture-in-picture/#request-pip says OPTIONALLY (which we should rephrase as MAY), whereas https://w3c.github.io/picture-in-picture/#disable-pip says if the attribute is added to the video element, the user agent SHOULD ..., which could be a MAY.

With these changes, we don't think we need to mark this feature as at-risk.