PIP and page visibility

[juberti]

In the demo I am working on, we want to stop displaying video (and display some message) when the window is hidden. However, if we're pipped, we can't tell when we're minimized, because PIP always causes the visibility API to return 'visible'.

Would it be possible to add another property to the page visibility API to indicate PIP mode? This would allow a page using PIP to understand if its main document is still visible, which seems like something it would want to know.

See also #80.

[beaufortfrancois]

Thank you for filing this issue.

It looks like we may not want to force the Page Visibility API eventually to always return "visible" if there's a PiP element. Web developers could check the visibility state of both the page and the PiP video.

Update: requestAnimationFrame() won't work anymore as page will be considered as hidden when minimized while it is working in Chrome with the current implementation of the spec.

document.addEventListener('visibilitychange', function() {
  if (document.visibilityState == 'hidden') {
    // page is hidden (even if there's a PiP video)
  }
  if (document.pictureInPictureElement) {
    // there's a video playing in PiP
  }
});

What do you think @jernoble @mounirlamouri?

[mounirlamouri]

I think that is the simplest way to solve the problem stated above. I'm in support.

[beaufortfrancois]

Shall we specify in the spec that we should still get requestAnimationFrame events to fire even if page visibility is hidden or is it an implementation detail?

[jernoble]

I think that (whether rAF should fire) should be left as an implementation detail. I don’t want to handcuff future UAs by limiting performance optimizations.

Could this use case be better solved with intersection observers? It seems like @jubertii isn’t necessarily interested in the page’s overall visibility, but whether a certain portion of the page (not including the video in pip) is visible.

[jernoble]

Also, the same problem exists with the Remote Playback API. I’d rather not have to add different enums for visibilityState for each spec that has a “visible but...” behavior.

[juberti]

I'm interested in understanding whether the page (not including PIP) is overall visible.

My understanding is that intersection observers only solve whether an element is visible in the viewport, which isn't an issue for my scenario, as the video will always be in the viewport. (The question is whether the viewport is in fact visible to the user.)

[jernoble]

Right, that was my question: whether intersectionObserver.observe(document.documentElement) will trigger when the document.visibilityState would otherwise turn to 'hidden'.

[jernoble]

Our local IntersectionObserver expert says this isn't part of the spec, so it looks like my workaround isn't going to be possible.

[jernoble]

Okay, so, three options:

1) Do nothing. The spec says "visible" unless the entire page is not visible at all, so we treat this use case as if only a tiny portion of the page was visible on the screen. 2) Add some "quasi-visible" state to the visibilityState. And hopefully do it in a spec-agnostic way so that other specs with the same issue get the benefit of the new enum state, and that means doing it in the Page Visibility API. 3) Decide that a minimized web view with a visible picture-in-picture presentation doesn't meet the language requirement that the "Document is at least partially visible on at least one screen", and allow the visibilityState to move to "hidden".

[juberti]

Option 1 doesn't solve my problem, since I can't tell if the page is un-minimized when PIP is up.

Option 3 would be easiest, I think, but I would suggest that we spec that rAF should still run (perhaps not mandatory), since otherwise I think that rAF definitely won't run, which would be bad.

Option 2 might be a future option once we understand the overall problem space better - I'm a bit reluctant to add in a lot of special cases to the PV API.

[jernoble]

I might be confused. Isn’t option 2 the one that adds special cases (new enums) to the PV API?

[juberti]

Sorry - markdown changed the option numbers on me. I updated the comment to avoid auto-numbering, it probably makes much more sense now.

[jernoble]

Gotcha. Ok with that clarification in mind, why must rAF still run when visibilityState is “hidden” and the video element is in PiP?

[juberti]

In cases where the PIP is being driven from a canvas, need some way to drive the redraw of the canvas. (Right now, I'm using setInterval, but that seems somewhat crufty.)

[jernoble]

Okay, then this sounds like a case of "having your cake and eating it too". :)

You want visibilityState === 'hidden' so you can detect when you're minimized (and presumably do less work), but you want rAF() to fire so you can do more work. And that's ignoring that rAF is a terrible way to paint video in the first place, since for anything other than 60 fps video, many of those rAF canvas paints will be painting the same frame as last time. At least with setInterval, you can fire at approximately the frame rate of the underlying stream rather than the frame rate of the display.

[juberti]

I don't have a strong opinion here, but the page can already know to do less work if it detects it's hidden and pipped, and we can skip video draws if we detect that video.currentTime hasn't advanced.

Generally I just want to make sure that PIP video is just as smooth as regular document video, and my understanding is that setInterval can't deliver this level of consistency.

[jernoble]

Cavas->MediaStream->Video is already a hacky, crufty workaround that's not guaranteed to be as smooth as native video in the first place. If you're looking for smoothness, I'd start by addressing the shortcomings in the video stack that force you to use canvas in the first place.

[juberti]

it's the limitation of PIP that only a single video can be pipped :)

[jernoble]

Yeah, and what you're asking for is the ability to PiP a canvas, not a video element. That you're having to go from canvas->MediaStream->video rather than taking a canvas directly into PiP is evidence that this is a hacky solution, and maybe not something we should optimize for.

[mounirlamouri]

Isn't the issue that the canvas wouldn't redraw without rAF? If we can PIP a canvas directly, would this actually resolve the problem?

[juberti]

exactly, if we didn't get rAF, we wouldn't have a stable heartbeat to redraw the canvas.

[jernoble]

The difference, however, is that for the general case of a "video element in PiP", you don't need to rAF(). For the case of a "canvas element in PiP", it's self evident that you do. If we mandate that rAF must fire when a "video element is in PiP", that will necessarily block off UA performance optimizations. If we mandate that rAF must fire when a "canvas element is in PiP", that doesn't affect the much more common case of a non-canvas backed video element in PiP.

[juberti]

I see where you're going with this; I'm fine with that approach.

[beaufortfrancois]

@juberti Can we close this issue? https://github.com/WICG/picture-in-picture/pull/112 has been merged and we agreed that Page Visibility events will fire independently of the visibility of the Picture-in-Picture window. See https://wicg.github.io/picture-in-picture/#page-visibility

[juberti]

Closing per discussion.