Closed defagos closed 1 year ago
I made a small PoC with a slider and several gesture recognizers:
Gesture recognizers are easy to implement and can be easily prioritized (e.g. .gesture(doubleTap.exclusively(before: tap))
). The slider behaves well.
I am confident we can implement a player UI in SwiftUI.
We had a pretty interesting discussion within the team, mostly about playlist and continuous playback support. Since AVQueuePlayer
need to be at the center, we discussed general design ideas about how we could better implement playlists if we were to rethink Letterbox:
AVQueuePlayer
allows to do (insertion, deletion, relocations). Items in the playlist would probably be defined by a URN, start time, settings (e.g. standalone parameter), etc. AVQueuePlayer
the orchestrator itself. We would be able to adopt a player API resembling the one of AVQueuePlayer
, both at the Media Player and Letterbox level, so that playlist support is guaranteed to work well. Of course, with all the other behaviors we currently have on top AVPlayer
playback (metadata, segments, etc.), we likely have to retrieve the media / media composition by the controller directly, but maybe not for playback purposes (if this can be achieved by a custom resource loader). Of course this concept, inspired by @StaehliJ who investigated a same idea with ExoPlayer, should be verified with a PoC first. An important question when using custom resource loaders is whether we can have good error reporting, as we know there are potential issues. This should be verified as well.
We should also get inspiration by reading through ExoPlayer documentation, as some ideas might prove to be helpful.
If I were to recommend a perimeter for a PoC based on the remarks above, a good scope would probably be:
AVQueuePlayer
for playback purposes.An interesting idea to PoC for playlist support: Playing urn:<urn>
in the player, and letting the resource loader manage metadata retrieval. This way we could prepare content without preparing metadata ahead. Does not mean that other metadata updates would occur through this channel, we could have a separate data retrieval process for other purposes. Also we need to check whether error handling can be handled properly and if this works correctly over AirPlay.
Other important question: When does resource pre-loading happen? Can we guarantee that an Akamai token is retrieved at the very last time? We should study when resource pre-loading might happen in an AVQueuePlayer
and possibly use resource renewal to load the tokenized URL at the latest moment (for token-protected content only, other content can be pre-loaded at any time).
I think we should also discuss going the monorepo way with Letterbox. We could then iterate faster on our player code base while still maintaining a clear separation between components by having each of them (core player, data provider, analytics, diagnostics, LB player) be a separate product within a single SPM package.
Moreover we should drop all the dependencies that are superfluous, like FXReachability, libextobjc, Mantle or SRG Logger. for example. If we later decide to separate the packages (e.g. the core player) we can easily do it. We would also only require a single demo project and we could share some test helpers which are now duplicate between projects.
I think we would likely need to support the legacy Letterbox for a while, which is maybe why we could use a codename for a new Letterbox, e.g. Pillarbox, until the final release. This would make it easier to identifiy about both products while they coexist .
Google Cast integration would also be necessary in a PoC. Not at the core player level probably (since the player is on the receiver, while AVPlayer is local), but at the UI level.
For controller configuration we should introduce as much immutable configuration as possible (e.g. update intervals, skip intervals if customizable, etc.).
For that we should have a configuration builder, something like:
let controller = SRGLetterboxController()
.configure { builder in
builder
.setUpdateInterval(30)
.setSkipInterval(10)
}
}
The builder pattern could also be used for construction of other immutable instances, e.g. analytics labels.
Some use case mentioned by SwissTXT which a better playlist could help implementing: Catch up with live, which means play all missed highlights in sequence and then switch to the livestream automatically.
This would likely be something easy to implement with a better playlist management and player items responsible of getting the media composition, though the feature would likely be implemented in an app rather than in Letterbox itself (more business logic involved).
Front-end teams to involve in discussions for a new Letterbox:
Given all the issues we recently had with stream packaging and segments / standalone playback, and given the cost-efficiency desired by some teams, we could discuss dropping segments entirely if this makes sense from a user experience point of view. Segments namely introduce a lot of complexity which might not be needed anymore.
For apps needing standalone content each individual clip would then be published independently, thus limiting the impact to only a few clips instead of all segments.
Snippet support would likely be interesting in the future.
We must definitely use the new async API for querying asset properties, see https://developer.apple.com/wwdc21/10146 and https://developer.apple.com/wwdc22/110379. iOS 15 for most of them, otherwise iOS 16+.
✅ I made a first PoC for a queue player with a list of items, each of them responsible of retrieving the media composition entirely via a resource loader. The result is very promising and matches all my current expectations:
processRequest
immediately returns false
or if finishLoading
reports an error to the resource loader after asynchronous resource retrieval.We can therefore imagine a core player implementing playlist with AVQueuePlayer
and, working with AVPlayerItem
and self-contained resource loading, would make it possible to play any kind of stream at the Letterbox level, provided we define a common media playback context format which can be bridged with IL or Play CH metadata, making it possible to play and mix any kind of content.
Other PoCs to be made afterwards:
AVPlayer
?ObservableObject
with @Published
state information. States will be enums with associated values, e.g. a seek will provide start and destination in its associated values directly. We can also likely implement a periodic time observer published property backed by a traditional periodic time observer (no custom publisher is likely needed to achieve this result).AVPlayer
async APIs.AVAssetResourceLoadingContentInformationRequest
renewalDate
property).AVQueuePlayer
.AVPlayerItemPlaybackStalledNotification
for stall detection.~ Seems that isPlaybackLikelyToKeepUp sufficesisPlaybackLikelyToKeepUp
works as expected.AVPlayerItemMediaDataCollector
for metadata retrieval. Seems that data collection can also be associated with time ranges.✅ I made a second PoC of a video feed. Though the implementation is fairly basic this shows that performance is quite good when scrolling. This is likely the best we can achieve (except if things can be further optimized with async asset loading somehow), but at least it is a lot better than what we have with the current Letterbox version.
✅ I made a third PoC for error propagation from a resource loader. Loading the media composition requires us to better propagate errors, which was historically an issue. This was preventing us from performing DRM error reporting at the Letterbox level, but things have improved, and I discovered by chance that the type of the error and its value (non-zero) are especially important so that propagation succeeds. This also was clearly improved in iOS 15, which is probably a hint this is the version we should target.
I just discovered that AVPlayer
supports various policies for background video playback since iOS 12, see audiovisualBackgroundPlaybackPolicy
. We will likely be able to implement the same behavior as before in a much simpler way.
Before starting any serious work on the player we should really look at all AVFoundation
and AVKit
headers to discover new APIs we might have missed.
The AVFoundation-Combine
library introduces a few publishers for AVPlayer
, in particular one for periodic time observation. The publisher is directly implemented and might be worth a look.
I also stumbled on an article which applies what I think should be sufficient, namely a current value subject or a passthrough subject.
Anyway, a good solution should likely register the time observer upon subscription and release it automatically when the subscription is cancelled. Maybe this is possible with a simple publisher and handleEvents
, otherwise a custom publisher might be better.
✅ I made a fourth PoC showing how a reactive layer can be built on top of AVPlayer
. Works nicely and should be a pretty solid inspiration for our implementation. I used AVFoundation-Combine
implementation above to implement periodic time publishers and checked that removal and associated player / subscription deallocations do not leak.
✅ I made a fifth PoC showing how a reactive player with business logic can be built on top of a lower-level reactive player. This PoC also shows that we can easily attach a player to an existing view, but also to several views at the same time, while keeping the UI in sync.
✅ I made a sixth PoC for documentation with DocC.
Useful links:
I also found that with SPM 5.7 we can no longer provide a package name to refer it in target dependencies. This most likely means we should match the repository name with the package name (though aliases can now be defined).
I think that using DocC is very valuable. We should use Xcode as much as possible to adopt common documentation style (e.g. ///
in general) and use separate markdown files only sparingly, e.g. for a repository welcome page with basic integration instructions.
☑️ I made a seventh PoC for project structure and SPM packaging.
I found two annoying issues:
.when(platforms:)
), dependencies without tvOS binaries like Google Cast SDK will make compilation fails (also see associated linked thread). The issue likely affects source packages, except if the sources can be compiled for all platforms (which is what I think happens with Mapp SDK). To be done:
The workaround is not necessary for the Mapp iOS SDK because it is built from source. But the WebKit dependency makes compilation fail for tvOS SwiftUI previews. If we need to integrate this SDK we can either work on a fix (because its packaging is currently a bit messy) or simply integrate the common SDK .product(name: "MappIntelligenceSDK", package: "MappIntelligence-iOS-v5")
since the iOS and tvOS SDKs have very few additions we will likely never need anyway.
I made a PoC for better workflow using GitHub and it is promising.
An idea about states (enums with with associated labels where appropriate):
Letterbox states are a bit richer than core player ones so they should be a superset if the core player states. With precisely defined states writing a (possibly custom) UI should be a breeze.
I tried integration Akamai QoS SDK but without success (missing core dependency), and it seems to be packaged with its own player AmpPlayer
. Likely a bad idea.
✅ I updated the reactive player PoC to support background video playback with the audiovisualBackgroundPlaybackPolicy
API. It works really as expected.
One limitation, though: We still have to detach the layer if we want to support background playback with the screen locked. This might be easy to implement but, if not, I would likely recommend dropping this feature, as it relies on non-official APIs we should rather avoid in general.
Some Letterbox issues are deeply related to how it was supposed to work at the time it was written. We discovered a few issues since then, some of them more annoying than others.
If we wanted to fix them all we could rethink SRG Media Player and SRG Letterbox, that means listing everything that is supported first, then thinking about an architecture that could help us solve the following issues (which ones are relevant should be agreed on) while preserving our feature set, performance and battery life:
AVPlayerQueue
).AVPlayer
andAVPlayerLayer
s first).This is of course no small tasks with no guarantee it will ever happen, but this issue can help us track important ideas which could help us think about Letterbox future.
If the idea of rewriting Letterbox is validated so that all these new features and issues can be fixed, we should:
AVQueuePlayer
and a singleAVPlayerLayer
, we can namely investigate if scrolling hiccups can be eliminated and what cannot be eliminated (work on the main thread required by media services, e.g.).