Proposal: pause iframe media when not rendered

gabrielsanbrito commented 3 months ago

What problem are you trying to solve?

Web applications that host embedded media content via iframes may wish to respond to application input by temporarily hiding the media content. These applications may not want to unload the entire iframe when it's not rendered since it could generate user-perceptible performance and experience issues when showing the media content again. At the same time, the user could have a negative experience if the media continues to play and emit audio when not rendered. This proposal aims to provide web applications with the ability to control embedded media content in such a way that guarantees their users have a good experience when the iframe's render status is changed.

What solutions exist today?

There is a proposed "execution-while-not-rendered" permission policy that halts all JavaScript execution of a not-rendered iframe. However, there are use cases where an application might want to just not render audio, instead of pausing JavaScript execution.

How would you solve it?

We propose a new permission policy "media-playback-while-not-rendered", which should pause any media being played by iframes which are not currently rendered. This would apply whenever the iframe’s "display" CSS property is set to "none".

Please find more detailed info in the feature explainer: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/IframeMediaPause/iframe_media_pausing.md

Anything else?

We are looking forward to gathering feedback on this proposal and also checking if the community and implementers are interested in this feature.

Thanks!

zcorpan commented 3 months ago

cc @whatwg/media

jernoble commented 3 months ago

Why can't the embedder just remove the <iframe> from the DOM? That would currently have the effect of pausing <video> elements in the <iframe>'s DOM, and could be extended to apply to all audio-generating APIs which exist as JS objects in the frame.

The explainer says:

[T]he only option is for the website to remove the iframe completely from the DOM and recreate it from scratch when it should be visible again.

Surely the website can store the removed frame in a JS object for re-insertion later, rather than re-creating the <iframe> from scratch.

zcorpan commented 3 months ago

@jernoble removing an iframe from the document removes the entire browsing context.

Surely the website can store the removed frame in a JS object for re-insertion later, rather than re-creating the <iframe> from scratch.

Reinserting the same iframe element will create a new browsing context and do a new navigation, resulting in a new video element. If the framed page is not same-origin with the parent, the parent also can't store a reference to the old video element.

zcorpan commented 3 months ago

Also see https://github.com/whatwg/html/issues/9793

jernoble commented 3 months ago

Ah, that explains it then.

dalecurtis commented 3 months ago

Would this also suspend AudioContext and handle WebRTC sessions? Or should we just expect those to get muted?

foolip commented 3 months ago

From the explainer, part of the proposed solution is to change the definition of allowed to play. Going one level deeper, what are the checks that run, from the media-in-an-iframe point of view?

The explainer points to the iframe not being rendered and also "does not intersect the viewport". IIUC, the idea is that the "let's pause stuff" signal comes from the iframe's visibility, not the media elements within the iframe. That does make this easier I think, and means the underlying rules can be the same for all audio-producing APIs on the platform.

Would the main building block here be intersection observer for all iframes, and propagating "iframe is not visible" state down through nested iframes?

gabrielsanbrito commented 3 months ago

Would this also suspend AudioContext and handle WebRTC sessions? Or should we just expect those to get muted?

@dalecurtis Our initial proposal is that this would also suspend any AudioContexts.

From the explainer, part of the proposed solution is to change the definition of allowed to play. Going one level deeper, what are the checks that run, from the media-in-an-iframe point of view?

@foolip in the current version of the explainer, we tried to propose a solution that would fit the current web standards. We considered 2 scenarios: "

Scenario 1: When the iframe is not rendered and it attempts to play audio; and
- Callers should treat this scenario as if they weren't allowed to start media playback. Like when the autoplay permission policy is set to 'none' for an iframe.
Scenario 2: When the iframe is already playing audio and stops being rendered during media playback.
- Callers should treat this scenario as if the user had paused media playback. the media "pause/suspension" caused by the iframe not being rendered

" Given these scenarios, I don't think that the media-in-an-iframe should do any extra steps besides what it already does currently. BTW, the explainer does not propose auto-resume when the iframe gets rendered back.

The explainer points to the iframe not being rendered and also "does not intersect the viewport". IIUC, the idea is that the "let's pause stuff" signal comes from the iframe's visibility, not the media elements within the iframe. That does make this easier I think, and means the underlying rules can be the same for all audio-producing APIs on the platform.

Would the main building block here be intersection observer for all iframes, and propagating "iframe is not visible" state down through nested iframes?

@foolip Yes. We propose that the signal should come from the iframe's visibility. I guess intersection observer could be a good building block. However, we also need to take into consideration the case where the iframe is outside the top-level document viewport but is still rendered. In this case, it should still be allowed to play. AFAIK, the intersection observer is not capable to capture this case.

For this proposal, I think that the IntersectionObserverEntry interface could maybe propagate "iframe is not rendered" down through the nested iframes. Maybe we could even add a new boolean isRendered to IntersectionObserverEntry interface, so that iframes can query their own render-state. WDYT?

haywirez commented 3 months ago

As there are other CSS techniques to hide iframes (visibility: hidden, position: absolute; width: 0; height: 0; border: 0; border: none;), should the proposal consider a "media-playback-while-out-of-viewport" policy as well?

marcoscaceres commented 3 months ago

The proposed name ("media-playback-while-not-rendered") isn't super clear to me... and apologies for bike shedding... but maybe it should be "pause-media-if-not-visible" or something, as it's literally what it does.

Looking at it more in context of allow=""

<iframe allow="media-playback-while-not-rendered">

It does make sense... but maybe it should be -while-not-visible (and the execution should maybe change to that too?)

liberato-at-chromium commented 2 months ago

Is it necessary to tie the "let's pause stuff" signal directly to some notion of visibility? From the above discussion, it seems like there are different ways to pick what that means.

An alternative is to make the signal more explicit, so that the parent could set it to "allowed" or "not allowed" based on whatever visibility, or other, criteria it wants to use.

gabrielsanbrito commented 2 months ago

As there are other CSS techniques to hide iframes (visibility: hidden, position: absolute; width: 0; height: 0; border: 0; border: none;), should the proposal consider a "media-playback-while-out-of-viewport" policy as well?

@haywirez I think we could also do that if there is interest from the community too. Just starting out with a small scope to see how it goes.

@marcoscaceres @liberato-at-chromium, yeah it looks like that "rendered" might not be the most clear termination. I think that using -while-not-visible instead of while-not-rendered might make conveying meaning easier. We would then need to expand the explainer scope to cover other visibility scenarios too - e.g. visibility: hidden, etc.

marcoscaceres commented 2 months ago

@gabrielsanbrito, can we clarify a use case for me: would it be reasonable to pause the media if the iframe is scrolled off the page? (I'm thinking like an automatic intersection observer... or how lazy loading works... then when the iframe comes back into view, media would resume)

I get the case of applying display: none and having the media pause, but it also feels a bit heavy-handed because it would potentially mean having to re-layout?

Also, I'm still not sure this falls under a Permissions Policy (there is no "permission" to be asked... I know Permissions Policy doesn't always apply to asking for permissions, but that's mostly a historical quirk)... maybe it could just be an attribute on the iframe that declares the behavior of media when not rendered and/or scrolled off the page (if the scrolling use case applies here)?

cc'ing @clelland in hope that he can chime in on the use of Permissions Policy for this.

marcoscaceres commented 2 months ago

Ah, lol, I missed @zcorpan link to https://github.com/whatwg/html/issues/9793 ( autopause ... that feels like maybe a more natural fit), but instead of a boolean, it could be a enum value to give more control (e.g., autopause="when-not-rendered" or whatever)

gabrielsanbrito commented 2 months ago

[...] would it be reasonable to pause the media if the iframe is scrolled off the page? (I'm thinking like an automatic intersection observer... or how lazy loading works... then when the iframe comes back into view, media would resume)

@marcoscaceres, yes it is reasonable. We decided to start with the "not-rendered" scenario to begin with a smaller scope. But we can also increase scope to include the viewport scenario too. In my opinion, the enum idea can be beneficial and we can spec different behavior types - e.g. "when-not-rendered" and "when-not-in-viewport" using the same autopause parameter. However, the enum approach would be restricted to HTMLMediaElements only (see next paragraph).

Also, I'm still not sure this falls under a Permissions Policy (there is no "permission" to be asked... I know Permissions Policy doesn't always apply to asking for permissions, but that's mostly a historical quirk)... maybe it could just be an attribute on the iframe that declares the behavior of media when not rendered and/or scrolled off the page (if the scrolling use case applies here)?

The permission policy provides a centralized control point for the frame: the same configuration would be automatically applied to all playback elements (not only HTMLMediaElements, but also AudioContexts, Web Speech API (possibly), etc). This way we could control other API too through this proposal. We took inspiration from the "execution-while-not-rendered" proposal.

marcoscaceres commented 2 months ago

The permission policy provides a centralized control point for the frame: the same configuration would be automatically applied to all playback elements (not only HTMLMediaElements, but also AudioContexts, Web Speech API (possibly), etc). This way we could control other API too through this proposal.

Right, the intention is clear... just using Permissions Policy as the mechanism to achieve that doesn't seem ideal to me.

We took inspiration from the "execution-while-not-rendered" proposal.

Yes, but at the same time that's just another unofficial proposal (AFAIK). I wouldn't use that as the basis on which to base this.

I'm not discounting it as wrong: I'm saying we should consider this carefully because it sets precedence for future things. Why I'm hoping @clelland will take a look too.

whatwg / html