Consider adding API for app to get the PTS of the GOP containing currentTime

[wolenetz]

MSE apps that wish to use remove() can encounter problems if they attempt to remove the currently-decoding-and-playing GOP: at a minimum, disrupted continuous playback can result.

Can we do better? I believe so :)

Some ideas I have around this (certainly incomplete, meant as a discussion starting point):

• Add a SourceBuffer method to retrieve the earliest PTS of coded frames in the GOP containing currentTime. The app could then use the result to ensure that a later remove() operation doesn't include the GOP that was currently being played. This approach has a few complications:
  • An app that wished to use it effectively would probably need to ensure that playbackRate >= 0, e.g., the playback is not going backwards. In many user agents, reverse playback of HTMLMediaElement is not currently enabled.
  • currentTime, while playing and playbackRate > 0, is always increasing, so the "current GOP" may change to a later one between calling this proposed API and the eventual remove(). This just means that less is removed, and the GOP underlying the play-head remains undisturbed (if the app doesn't remove it, of course).
  • Even if paused or playbackRate <= 0, this API should still give the earliest PTS of coded frames in the GOP containing currentTIme. (Perhaps we might change this such that if playbackRate is strictly < 0, the API instead returns the highest ending presentation time of the GOP containing currentTime?)
• And/or make remove() have an overload with an optional parameter that lets the app optionally request the remove() operation to not actually do the remove if the remove would remove the GOP currently being decoded and played, and instead give some exception - perhaps with some indication of the earliest or latest PTS (depending on playbackRate>=0 or not) of coded frames in the currently decoding+playing GOP.
  • This alternative would alleviate all 3 of the complications listed for the first alternative, above.

I'd really welcome other proposals for how to alleviate this MSE web application difficulty. @jdsmith3000 @mwatson2 FYI - please chime in.

[jyavenard]

One of the issue with all those synchronous methods is that it's synchronous. Also, what is currently being played currentTime is typically behind what is currently being decoded and as such what is currently being accessed in the sourceBuffer.

For example the Windows H264 system decoder has a latency of about 35 frames. It needs to be fed 35 frames before outputting one. So when you playback start, currentTime is 0, yet the position in the sourceBuffer is already at least 35 frames ahead. The internal player also typically decodes ahead of time to ensure that it keeps a buffer full and handle intermittent hiccup, so that adds say another 10 frames buffered.

It is very difficult as such to link the data that is being played/presented with the data the sourcebuffer is actually processing. The current working GOP may not be the current GOP being presented.

So I don't see how you could query some sourcebuffer internals to get information related to currentTime. And doing so synchronously makes it even worse in a multi-threaded world used by agents implementation.

What about simply preventing removal of data that is currently being used by the sourcebuffer regardless of currentTime value.

That is now, remove could return an error such as "data in use" if the user attempt to remove a current GOP.

The problem with this, is that some JS player (like YouTube) will clear the entire sourcebuffer when the user change the resolution. That always remove the current GOP.

So we could make that new behaviour of remove() optional. Which means that it will likely never be used :)

[wolenetz]

I'm considering handling this instead as part of a new eviction policy that evicts anything buffered up to, but not including, the most recent random access at or prior to the next coded frame that will be decoded in playback. The issue tracking various eviction policy additions is #232.

[mwatson2]

I propose we close this and handle under #232.

[wolenetz]

I agree.