Picture in Picture PoC

[defagos]

As a Pillarbox developer I want to have a better understanding of how picture in picture could be implemented in Pillarbox so that any SwiftUI- or UIKit-based application is able to easily implement this feature.

Acceptance criteria

• A basic SwiftUI integration example is available.
• A basic UIKit integration example is available.
• An API proposal is made.
• A basic overview of the required components is provided.

Hints

• Adopting Picture in Picture in a Custom Player
• Adopting Picture in Picture in a Custom Player

Expected behaviors to implement / work correctly

• Enter PiP when throwing the app to the background.
• Manual PiP with dedicated button.
• Move to PiP via button, send app to the background, then resume from PiP overlay (must not glitch).
• Automatic PiP exit when switching to an app and returning while the player is still visible.
• Should be able to exit from PiP if navigating to the currently playing item (instead of interrupting playback to play the same content).

Tasks

• [x] Integrate with the SwiftUI demo.
• [x] Create an example with UIKit.
• [x] Make it work with the system UI.

[defagos]

Possible integration scenarios for PiP to consider for the final implementation:

• VideoView in SwiftUI iOS app.
• VideoView in UKit iOS app.
• SystemVideoView in SwiftUI tvOS app.
• SystemVideoView in UKit tvOS app.

[defagos]

A first PoC has been pushed onto the archive/605/picture-in-picture-poc-1 branch. Here are a few learnings extracted from this PoC which could be useful for our future implementation.

Player layer management

SRG Media Player implemented lazy view instantiation (and storage) at the controller level. This now appears to be fundamentally flawed:

• We could not share the same controller between several video layers (only one view would display moving images).
• PiP, essentially an AVPlayerLayer temporarily borrowed by an AVPictureInPictureController, was implemented as a player behavior, while fundamentally all PiP requires is a layer.

The PoC implementation shows that a much simpler way of managing PiP is possible:

• A VideoView must have a state to identify whether its player layer supports PiP.
• Supporting PiP means that a VideoView must be able to transfer ownership of its player layer to a PiP controller when appropriate. Conversely a VideoView must also be instantiable from an existing player layer given back by a PiP controller when PiP ends. This last fact is essential to ensure that the system can correctly animate the layer back into place after restoration has completed (completion handler provided by the PiP controller restoration method called by the implementation).

What is still not clear at the end of this PoC is where the PiP controller should be stored. The PoC stores the PiP controller at a singleton level (since there is only one active PiP session at most) but this means that we cannot have different PiP states associated with separate VideoViews. There is only one truth.

We could imagine having several PiP-enabled views on screen, though, each being able to start PiP independently. When entering PiP the controller (and thus the layer) would be shared to a central service (similar to the PictureInPicture of the PoC) managing the single possible active PiP session and restoration from it. This last approach might not be feasible, though, since we cannot be sure which instance would win when the application is sent to the background. So this requires a bit more investigation first (if not we already have a viable strategy).

Possible implementation strategies

The video view should provide opt-in support for PiP, e.g.

• VideoView is added a Boolean parameter for PiP support (defaults to false). As seen in the PoC this usually requires view representable UIView updates to manage Boolean changes. In the PoC implementation this costs a run loop to avoid body updates triggering other body updates, which leads to a SwiftUI runtime warning.
• DetachableVideoView could be a second view type which is always enabled for PiP. This would avoid the above trick (PiP controller creation can be made at view creation time). Still there might be some tricky cases to consider (e.g. switching between DetachableVideoView and VideoView in response to a toggle button while PiP is active) but those would need to be considered with the Boolean option anyway.

Video view implementation

We need to be able to build a video view from a layer, thus our implementation needs to be slightly revisited so that the player layer is a sublayer of the video view, either freshly instantiated or created with a layer received from a PiP controller.

Possible PiP integrations

Having PiP support entirely separated from Player has another benefit. It namely allows users to implement PiP easily with reduced capabilities, or in a deeper way.

Simple client-side PiP integration

The Netflix PiP integration does not provide any PiP button. PiP can only be enabled when sending the app to the background. In this case the player view from which the layer is detached is never dismissed, there is therefore no way to persist any particular state so that PiP can continue.

In this case the only action required to support PiP is to use a view supporting it. All the rest is handled internally and there is no restoration mechanism to implement. No PiP button must be provided to the user.

The only downside of this integration is that the user cannot browse the app while playing content. They are always stuck at the player level.

In comparison to Letterbox the fact that PiP should be handled at the view level only means we can have simple PiP integrated in no time in all apps, which was previously not possible.

Deeper PiP integration

Apple tv+ PiP integration ensures that the app can be navigated while PiP is active. This requires persistence when the player view is dismissed.

Apps should have a view model for their player UI (in the simplest case this VM is the Player itself, but if more logic is required a proper VM owning a Player would be more appropriate). Restoration is simply achieved using a view able to reflect the VM and present it again.

Learnings

We learned more things along the way:

• The AVPictureInPictureController isActive property is a bit slow since PiP is considered active only after the transition animation has been made. If implementing our own isActive concept (set to true as soon PiP will start) we are not forced to wait until PiP has started to decide whether playback must be stopped or not during dismissal. This provides for a snappier experience, compare Pillarbox and Letterbox demos.
• Restoration requires the completion handler to be called ideally at the end of the restoration animation. There is currently no way to know when a SwiftUI modal transition completes, except waiting for a time corresponding roughly to the transition animation. This should be enough until SwiftUI provides official completion handlers for modal transitions (animations were added completions in iOS 17 so there is still hope).
• When PiP starts the UI should be dismissed (even if backgrounding the app, as in the tv+ experience) and restored afterwards This is not also more consistent from a user perspective (the result is the same, no matter how PiP was started).
• When implementing PiP for the standard AVPlayerViewController we must dismiss the view controller in the did start method. Attempting to do so in the will start method leads to playback being incorrectly paused when moving the app to the background.

UIKit implementation

The UIKit implementation does not require more work:

• The object managing PiP SwiftUI lifecycle, PictureInPicture in the PoC, can also provide a delegate protocol for the standard PiP lifecycle.
• UIKit apps can implement these methods, as they would have done with AVPictureInPictureControllerDelegate anyway.

A question remains, namely whether we should provide a state publisher for the PiP management service, PictureInPicture in the PoC, so that UIKit apps can easily register to changes. But maybe simple KVObservability suffices.

Challenges

• Cleanup: If possible we should try to get rid of any AVPictureInPictureController instance stored globally when PiP is stopped. But maybe moving PiP controller storage / creation near the VideoView would help.
• Several players supporting PiP in an app: If an app has several player layouts, all supporting PiP, with different VMs, maybe we should be able to provide a source at dismissal time so that restoration can reliably know which view it must restore.

SwiftUI demo

We could:

• Implement PiP support in the main player.
• Maybe also in a 2nd more complex view with a larger view model, e.g. the playlist view.
• Fix routing so that restorable destinations have ids which do not depend on URNs.

UIKit demo

We could write a simple separate UIKit demo, like in the PoC. But we should ensure it always compiles correctly, even if it is a small sample app.

Documentation

We should likely provide document both for basic PiP integration (basically free) as well as deeper integration, which requires more involved view model design work but can still be nicely integrated with reusable view components having a Player as parameter.

Open questions

• Could / should we retain the Player during the PiP session so that the session is guaranteed to stay alive? This behavior is likely intersting to investigate when implementing support for AVPlayerViewController.
• How can we have a single consistent formalism for VideoView and SystemVideoView so that changing one view with the other is seamless? (e.g. via the player layout demo setting)