Consider adding API for app to know how much room is left in the SourceBuffer

[wolenetz]

User agent implementations have their own heuristics for MSE coded frame eviction, resulting in difficulty for apps that try to predict whether or not there's enough room for an appendBuffer() to succeed without yielding QuotaExceededError. This leads to complex web app code to moderate how much they fetch to buffer-ahead, when the signal of memory pressure from the user agent (QuotaExceededError) occurs only upon attempted appendBuffer of already-fetched media data.

Can we do better? This issue assumes yes and requests that we do so :)

Note #40 is an inverse of this (notify the app when there's enough room now available in the SourceBuffer).

Some ideas I have around this (certainly incomplete, meant as a discussion starting point):

• Simple API to request of SourceBuffer how many bytes beyond those currently buffered can be appended without triggering coded frame eviction. This approach has a couple complications of its own:
  • In muxed SourceBuffers, thresholds for eviction may already be approximated, since the actual proportion of bytes in each track is unknown until coded frame processing has completed after the appendBuffer is done, and the user agent implementation may impose per-track byte limits rather than an overall per-SourceBuffer or some other limit. Chromium does such approximation, and could give a precise answer to this proposed API. I'm unclear if other implementations would be able to do similar.
  • If the user agent dynamically adapts the memory limits (e.g., based on available system memory) at runtime, an answer from this proposed API might not be stable when the actual appendBuffer() occurs later. If this is the case, the spec for this would need to make it very clear that this "headroom bytes" may change, and I'm less certain of the value of this proposed API route.
  • Spec documentation would be needed to clarify that the response of this proposed API is invalidated when any appendBuffer() or remove() is done.
  • Note that compliant user agent implementations are not necessarily required to emit QuotaExceededError if an appendBuffer() is done of an amount greater than this proposed API returns, since coded frame eviction could remove enough to allow the larger append.
• And/or have this "headroom bytes" be included as an event parameter of the 'updateend' event (or similar MSE SourceBuffer events like 'update' or 'abort', etc.) I believe this approach may lead to app confusion since these events are delivered asynchronously at some time after SourceBuffer.updating has become false. I've seen MSE users encounter difficulties with such asynchrony, and delivering this "headroom bytes" in a similarly delayed fashion could lead to the web app mistakenly assuming that value is still valid.

I'd really welcome other proposals for how to alleviate this MSE web application difficulty. @jdsmith3000 @mwatson2 FYI - please chime in.

-edited to fix MD formatting

[mwatson2]

I agree this is a real problem and it is most acute in devices with a strictly limited memory buffer for audio / video data. Knowing the amount of available memory is not only useful for determining if appends will succeed / fail but also in making adaptive streaming choices (if there is limited memory, adaptive streaming algorithms may choose lower bitrates in order to maintain a larger buffer (in time) against future network fluctuations, even if a higher bitrate is available and could be sustained with current throughput.

It would also be good to give the application some visibility / control of the eviction of already played media. Many players have a "skip back" function that skips a few seconds back in the content and so the player wishes to keep played media around for this time window. So, it would be good to be able to know not only whether a new append would succeed but also whether it would result in played media being discarded.

[jyavenard]

Personally I believe that using bytes is too restrictive, and can only result in a particular storage implementation.

For example, one could consider using durations instead and allow the sourcebuffer to contain a maximum of say 30s total data (regardless of the resolution)

More useful I believe would be to have a way to evict data asynchronously. Right now, eviction can only occur synchronously during the prepare append algorithm

I don't see however how provided that information back to the JS help more than producing QuotaExceededError once the source buffer is full. What could the user do with this information that can't be done after the fact. I believe the end result is the same.

So you attempt an appendBuffer , get QuotaExceededError, so you manually use remove for what you don't need. If auto eviction couldn't naturally occur, there's typically not much you can manually remove anyway.

So now instead you know there's no more space available prior calling appendBuffer, and there's still not much you can manually remove.

[beaufortfrancois]

/me hijacks this thread as issue below is kinda related.

As far as I can tell, Chrome MSE video buffer size limit is 150MB while Firefox one is about 100 MB. Test by yourself at https://beaufortfrancois.github.io/sandbox/media/source-buffer-limit.html

It would be nice for web developers if we could, at least for now while there is no API yet, have some consistency across web browsers. I'm not saying 150 is better than 100. Let's simply use the same numbers for audio and video to make web developers life easier.

From what I can see, 150 was picked because it is approximately 5 minutes of 4Mbps content.

For info, I've tried my sample in Edge 15 with the "Always On" MSE VP9 flag and I get 5GB of size limit... Not sure what is going on...

WDYT @jyavenard @wolenetz? 150, 100, or something else?

[beaufortfrancois]

And I've just found out Safari uses different numbers:

Allow SourceBuffers to store up to 304MB each, enough for approximately five minutes of 1080p video and stereo audio.

const unsigned fiveMinutesOf1080PVideo = 290 * 1024 * 1024; // 290 MB is approximately 5 minutes of 8Mbps (1080p) content.
const unsigned fiveMinutesStereoAudio = 14 * 1024 * 1024; // 14 MB is approximately 5 minutes of 384kbps content

[beaufortfrancois]

Here's a summary of my findings so far I'll update:

	Chrome	Firefox	Safari	Edge
Video	150MB	100MB	290MB	?
Audio	12MB	15MB	14MB	?

_Sources: https://chromium.googlesource.com/chromium/src/+/6d9223a67b5c7058e8c0f503901d4658e9eb4141/media/filters/source_buffer_platform.cc https://github.com/mozilla/gecko-dev/commit/c94e9927bc593827f9d3501a76c59659700e658f https://github.com/WebKit/webkit/commit/fcb9656a38329ef7ea88dd535e0bbb86917dc253_

[jyavenard]

I responded earlier on what I thought about memory measures in bytes based threshold. A promise based appendBuffer would make the handling of rejection much easier if that's the end result we want to achieve.

I'd also like to implement a threshold based on duration rather than bytes. Like a maximum of 30s of buffered content.

[jyavenard]

Also what about mobile platforms. 150MB may not be feasible on some low end phones

[beaufortfrancois]

Reaching a consensus on the API shape (promises and duration) will take some time (VNext). In the mean time, having some consistent browsers MSE limits seems simple and will benefit web developers shortly.

[wolenetz]

Implementations can also have different system limits; e.g. Chromecast limits are lower than desktop Chrome. Also, things like dynamic memory pressure modify these limits. Since different MSE API users certainly provide vastly different bitrate and framerate streams; I wouldn't want to make a mistake that requires a one-size-fits all (size or duration) across implementations because that could regress important metrics like underflow frequencies (or OOMs, alternatively). This is why there is a QuotaExceededException on failed attempted appends in MSE REC: it allows implementations to choose the best coded frame eviction algorithm's removal ranges selection heuristic for their platform, while giving the MSE web app notice that it can respond to (by doing things like changing bitrates, explicitly removing media, retrying after some time or with some smaller piece of data, etc.)

tl;dr: There's certainly room for improvement, but I don't think requiring consistent (size or duration) limits across all implementations would be good.

[wolenetz]

Backlog until/unless API approach determined more clearly or priority is increased. Note that #232 may allow implementations to provide an asynchronous ability to evict preemptively.

[wolenetz]

This issue is part of a group of "introspection API" or similar API feature requests: #259, #209, #172, #40, #35.