search

w3c / media-source

Media Source Extensions
https://w3c.github.io/media-source/
Other
267 stars 57 forks source link

Define a preemptive eviction MSE API for low-memory platforms #232

Open wolenetz opened 5 years ago

wolenetz commented 5 years ago

This issue tracks the incubation work necessary to include a new feature in MSE vNext that seeks to:

  1. Allow web applications to configure MSE SourceBuffers to buffer "infinite" GOPs, including preemptively (not just at appendBuffer() or remove()) performing garbage collection ("coded frame eviction") of media within the currently playing GOP that has not yet been played, while letting the remainder of the GOP continue to play. This configuration comes with consequent reduction in capability for reliable suspend/resume/seeking if the coded frames necessary for resuming after suspension or for finalizing the seek are no longer buffered.

  2. Allow web applications to configure MSE SourceBuffers to more aggressively evict buffered media, preemptively, at the implementation's discretion (e.g., due to memory pressure), again with consequent reduction in capability for reliable suspend/resume/seeking.

  3. Give implementations greater control over the default garbage collection mode of SourceBuffers, and constrain the set of all possible modes available to the web application. For instance, some implementations may not support preemptive garbage collection; others may require it.

Like other MSE vNext incubations, I expect to publish an explainer/proposal soon in a branch in the WICG MSE repo, and use this issue in this upstream w3c repo to track discussion and feedback throughout. I will update in a comment on this issue once that explainer is ready.

A little history on this feature request: Initial discussion in the context of low-latency, live streaming of infinite GOPs (e.g. just one keyframe, followed by a bunch of non-keyframes) not supported by MSE's GOP-based range removal algorithm, and subsequent discussion in a Chrome/Firefox face-to-face in May 2018, FOMS 2018 (day 1, day 2, and FOMS 2019 - especially in the multi-browser face-to-face that occured in March 2019) have all led to this feature request.

@jernoble, @jyavenard, @greentorus, @Codeusa, @mmmmichael

wolenetz commented 5 years ago

Note that this issue is part of a larger set of live/low-latency feature requests being considered:

Giving app explicit control over jitter buffer/latency is likely to be worked on more in the context of HTMLMediaElement and not just the MSE extension of it.

Giving app more control over MSE buffered range gaps and playback behavior around them is expected to help improve interop that otherwise could impair MSE playback UX. Though not restricted to just live/low-latency streaming, stalled playback at unexpected buffered range gaps is especially painful in such streaming.

wolenetz commented 5 years ago

Note, implementation work in Chromium associated with incubating this feature is tracked by https://crbug.com/963717

wolenetz commented 5 years ago

I plan to tackle non-preemptive coded frame eviction modes/policies first (a subset of the objective #1 tracked by this issue). I'll have an explainer published and linked here within about 24 hours from now.

wolenetz commented 5 years ago

I have a proposal for new (non-preemptive) coded frame eviction policies now posted at https://github.com/wicg/media-source/blob/mse-eviction-policies/mse-eviction-policies-explainer.md

Please take a look and give feedback.

Most especially - I don't really like the policy names "before-current-gop" and "before-next-demuxed": suggestions would be very welcome and helpful!

wolenetz commented 5 years ago

@jernoble - please take a look at the explainer I posted today at https://github.com/wicg/media-source/blob/mse-eviction-policies/mse-eviction-policies-explainer.md In particular, I would really appreciate your input into what kind of eviction policy may enable the preemptive/purgable buffers that I recall we discussed at FOMS 2019.

greentorus commented 5 years ago

Looks good to me, including the descriptive eviction policy names. I didn't participate the FOMS 2019 discussions: By "preemptive/purgable buffers" you are referring to "before-current-gop" and "before-next-demuxed", not to "pre-emptively evict[ing] (not just during the Prepare Append Algorithm) any coded frames at the discretion of the user agent"?

wolenetz commented 5 years ago

@greentorus (https://github.com/w3c/media-source/issues/232#issuecomment-493850780) - yes: in addition to the existing "evict automatically during the Prepare Append Algorithm", there were requests to allow for eviction to occur at any time, "pre-emptively". The proposal at https://github.com/wicg/media-source/blob/mse-eviction-policies/mse-eviction-policies-explainer.md adds other requested eviction policies (and a way for apps to specify which one(s) to use), but does not solve the pre-emptive evictions request.

jpiesing commented 5 years ago

@jernoble - please take a look at the explainer I posted today at https://github.com/wicg/media-source/blob/mse-eviction-policies/mse-eviction-policies-explainer.md In particular, I would really appreciate your input into what kind of eviction policy may enable the preemptive/purgable buffers that I recall we discussed at FOMS 2019.

One thing that the explainer doesn't address is what would happen when pre-loading content (e.g. an advert). Are there any circumstances under which data that has not been played might be evicted or is the 'guidance' that QuotaExceededError should be thrown when it's not possible to preload any more?

jernoble commented 5 years ago

I hate to bikeshed too early, but there exist audio codecs with sample dependencies, and as such "GOP" may be the incorrect term for describing the policy: "discard all samples up until the most recent sample with un-decoded dependent samples". (Unless we redefine audio samples to be "ear pictures".) Also:

...what kind of eviction policy may enable the preemptive/purgable buffers that I recall we discussed at FOMS 2019.

The devil, as always, is in the details. Is it fine to report a buffered range that includes media data that may have been purged? Or should we check the purged state whenever buffered is called? For the before-current-gop and before-next-demuxed modes, maybe we only ever report buffered ranges from the previous sync sample or currentTime onwards? Off the top of my head, the latter seems the most interoperable and most deterministic.

wolenetz commented 4 years ago

@jernoble (https://github.com/w3c/media-source/issues/232#issuecomment-494463928) good point. I'll be updating the proposal/explainer soon. I'm also working on an experimental prototype in Chrome of a possible API shape for this.

JohnRiv commented 3 years ago

Noting that this is of interest to CTA WAVE, particularly @jpiesing's question.

wolenetz commented 3 years ago

This looks to me to be in scope for V2. I'm currently working on a Chromium prototype, which should hopefully assist spec developement and solution of this issue.

wolenetz commented 1 year ago

Prototype didn't land - it was focused mostly on low-latency (infinite GOP) streaming without seek support, though it did contain some notion of better standardization of an eviction range selection algorithm.

WebCodecs / Media Stream playback handles low-latency scenarios probably better than the prorotype did (and importantly, WC/MS are in use on the web platform now).

I think the remaining actionable MSE portion of this issue would be to specify a normative preemptively-collectable version of MSE. Perhaps simplest to be done by having distinctly named API, e.g. "ManagedMediaSource" (credit: @dalecurtis for this name suggestion), "ManagedSourceBuffer", etc. Preemptive eviction results possibly could be custom emitted to a callback provided during ManagedMediaSource or ManagedSourceBuffer creation.

As such, I'm narrowing the scope of this issue.

rmmh commented 1 month ago

For reference, Apple has shipped ManagedMediaSource in iOS 17 & Safari 17. #320