Open davemevans opened 7 years ago
@bbcrddave I understand there is a use-case for this for live services, where the desired behaviour when the buffer runs empty is not to fall behind "real-time" but to pause and then jump forward once new data is available.
But could you elaborate on the trick play and low latency playback use-cases ?
sounds similar to https://github.com/w3c/media-source/issues/21 (and https://github.com/w3c/media-source/issues/133)
The 'low-latency' use case is essentially the problem you describe where data may not be available in time for whatever reason and staying at the live edge is deemed more critical than having moving pictures. My choice of words was poor because this isn't about decode latency (so not the same as #21 etc) but staying at the live edge at all times.
For trick play, it may be difficult to download data required to maintain the buffer as playback speed increases. It may be useful to append only some segments (every xth for example), despite the potential UX problems, or even partial segments where possible, and leave gaps in the buffer which are played through.
Thank you for filing this issue. While partially a duplicate of #133 and #21, I'll keep this one independent in VNext to support the specific use case mentioned here.
I think we generally agree there may be a need for low latency. I can also imagine that trick play might benefit from skipping segments during appends, but as you note, the app could probably do this perhaps just as well. Can you provide evidence of issues here that are difficult to solve with the current MSE API?
Using MSE to broadcast live video streams has great potential but lacks some key features that must be addressed to really be a viable solution. First the ability to skip over missing frames (play through gaps) is a key feature that would need to be addressed. Second the ability to append sourcebuffers with video data received from one of many video sources (passing the permission from one participant to another to transmit video), currently I have to tare-down a sourcebuffer and rebuild it every time the video is passed, causing a flash (no desirable). I tried using 'Sequence' mode to solve these issues, but got unexpected video drift that was worse than what I tried to solve.
@wolenetz do you know if this feature request will be included in the work being done for the referenced #232 ? That's tracked by https://bugs.chromium.org/p/chromium/issues/detail?id=963717
1) How much tolerance the MSE-extended HTMLMediaElement.buffered will have to be able to coalesce any SourceBuffer gaps (e.g. "play through any gap that is smaller than 0.75 seconds") and not stall, 2) and/or what HTMLMediaElement playback will do when it encounters a gap (after (1)'s coalescence, if any) with a buffered range later (such as play silence for missing audio, frozen last frame for missing video while letting clock advance at playbackRate? Or seek to the next point where all active tracks have buffered media? Or seek to the earliest next point where an active track has buffered media (e.g. for commonly jagged A/V segment buffered range starts) and play silence/last frame for any other track until it also gets buffered.
This has been discussed multiple times (FOMS 2018, FOMS NYC March 2019, various F2F including recent TPAC Sep 2019 media WG F2F), and will likely become a vNext feature to incubate soon.
Per https://github.com/whatwg/html/issues/6359, we'll likely need to help UAs and apps understand and customize audio vs video-specific playback behavior on network underflow (buffered range gap in each track, here) as part of this solution.
Briefly discussed during 8 Nov 2022 Media WG meeting: https://www.w3.org/2022/11/08-mediawg-minutes.html#t05
This came up again at FOMS2023. It's absolutely a valid issue that still needs to be solved. Meanwhile, there is an ugly hack that might help coalesce unintended gaps in Chromium SourceBuffers: the coalescence heuristic currently used there is based on the largest frame duration buffered so far in that particular track in that particular SourceBuffer. Auto-coalescence of gaps can thus be achieved by manipulation of the inputs to this heuristic:
Yes, this is ugly: non-interoperable. Not guaranteed to continue working. Adds overhead to time-to-first-frame (async buffering and removal). But since this issue has not yet been fixed, and it is hurting MSE users, this hack might help at least on Chromium based browsers.
This came up again at FOMS2023. It's absolutely a valid issue that still needs to be solved. Meanwhile, there is an ugly hack that might help coalesce unintended gaps in Chromium SourceBuffers: the coalescence heuristic currently used there is based on the largest frame duration buffered so far in that particular track in that particular SourceBuffer. Auto-coalescence of gaps can thus be achieved by manipulation of the inputs to this heuristic:
- For each track, first append a simple keyframe with huge duration. Ensure it is buffered. Then remove it from the SourceBuffer (or overlap it in your first appends to that track).
- This should trick the heuristic.
Yes, this is ugly: non-interoperable. Not guaranteed to continue working. Adds overhead to time-to-first-frame (async buffering and removal). But since this issue has not yet been fixed, and it is hurting MSE users, this hack might help at least on Chromium based browsers.
Thanks for this, really interesting. I would like to share my observations:
I created a media segment with a single IDR frame and a duration of 4 seconds. Note that earliest_presentation_time= 720000
so I am setting sourceBuffer.timestampOffset = -8;
to start from 0 in the SourceBuffer:
Now simply appending the corresponding init segment and the above media segment to the SourceBuffer leads to a buffered range from 0 - 4 as expected. But when removing the buffered range and hitting "play" the current time of the video element is still progressing although there is no valid buffered range:
As a comparison, Firefox is behaving as expected:
I also did not manage to replace the buffered IDR frame with new content. The new content only becomes visible after the segment with the IDR frame has finished (after 4 seconds)
It looks like a simple seek (e.g. seeking to 0.1) solves the previously described problem:
As @wolenetz described Chrome is playing over gaps in my tests afterwards. In this example, there was a gap between 2-4 seconds:
For comparison Firefox stalls after 2 seconds:
I put my findings in a blog post: https://websites.fraunhofer.de/video-dev/being-trapped-in-a-gap-with-big-buck-bunny/ : Happy to get feedback, maybe I also missed something in my implementation and the gap workaround can be used without a seek.
I put my findings in a blog post: https://websites.fraunhofer.de/video-dev/being-trapped-in-a-gap-with-big-buck-bunny/ : Happy to get feedback, maybe I also missed something in my implementation and the gap workaround can be used without a seek.
@dsilhavy Nice write-up! It looks like the "ugly hack" is working for you for video. Seek is understandably a "fix" for the Chromium media pipeline having started prerolling the decoder with the (later removed) media. In this case, the prerolling is aborted by a seek to a time that is not currentTime. Without the seek / preroll abort, the Chrome media pipeline will still accept the newly buffered content into the timeline, but the renderer will throw away decode output with earlier presentation time than what was expected to follow the prerolled content. At least, I think this is what is happening.
To avoid needing to seek, try buffering the fake content elsewhere in the timeline (say, far in the future), then remove it, optionally adjust the MediaSource duration, and then append the real content. How far in the future? This is another point of lack of interop precision. I'd suggest more than thrice the manufactured frame's duration, and no less than 10 seconds, to hopefully avoid prerolling any decode of the fake content.
While considering the "ugly hack" further, I recalled more ugly details. The hack will likely lead to issues even now in Chromium when attempting to auto-coalesce gaps in an audio track's buffered ranges for manifold reasons:
There are likely other caveats, further motivating getting this spec issue fixed and implemented.
A couple other notes:
In order to support use cases such as trick play, as well as very low latency playback, it may sometimes be useful to be able to allow the media time to keep incrementing at all costs whether or not there is any media to be played.
Ways in which I can imagine this working could be: a) signal a SourceBuffer as potentially containing sparse data and not to stall the timeline during unbuffered ranges (exact behaviour to be defined eg hold last frame) b) explicitly 'append' an 'empty' range
I appreciate that this may be better solved by the media element, or at application level, but I think it is at least worth considering in MSE.
I intend this is to be considered a VNext use case.