search

videojs / http-streaming

HLS, DASH, and future HTTP streaming protocols library for video.js
https://videojs-http-streaming.netlify.app/
Other
2.5k stars 424 forks source link

Support smoothQualityChange on HLS when using MSE #1224

Closed eocanha closed 2 years ago

eocanha commented 2 years ago

Description

The current "fast quality switching" policy which clears the buffers and replaces then as fast as possible with others with the new quality is good for fast quality switching on regular desktop systems, which have a software decoder and plenty of CPU.

However, on set-top-boxes using embedded browsers like WPE WebKit, this causes an internal "flush and reenqueue" on the playback pipeline, which is emptied and has the buffers with new quality enqueued and quickly decoded until the old playback position is reached again. This causes a big stress on the decoder to catch up, and it's even worse on devices with hardware decoders and custom audio-video synchronization mechanisms.

Users of video.js on those systems would benefit a lot on having the smoothQualityChange setting honored when using Media Source Extensions.

This issue is currently affecting a user deploy using video.js v7.8.1 (http-streaming 1.13.2). For some reason, they can only use published video.js versions, and that's why it would be interesting to have the problem solved on the http-streaming 1.X branch (at least).

Sources

This is the place where an already buffered segment is selected for load and replacement, causing a "flush and reenqueue" in the browser implementation: https://github.com/videojs/http-streaming/blob/1.x/src/segment-loader.js#L870

Steps to reproduce

The steps to reproduce the issue involve having WPE running on a set-top-box, which isn't trivial. Still, I'll try to focus on comparing the effects of fast quality change vs. smooth quality change.

  1. Open an HLS stream in an environment with variable network conditions and wait until a quality change happens.

  2. Observe in the logs how the fast quality change policy overwrites already buffered segments (the log lines are edited to fit in 80 columns):

    VHS: SegmentLoader[main] [3-Stream(04)/index.m3u8] [0 => 3.9599999999999973] VHS: SegmentLoader[main] [3-Stream(04)/index.m3u8] [4 => 7.959999999999997] VHS: SegmentLoader[main] [4-Stream(05)/index.m3u8] [4 => 7.959999999999997] VHS: SegmentLoader[main] [4-Stream(05)/index.m3u8] [8 => 11.959999999999997] VHS: SegmentLoader[main] [4-Stream(05)/index.m3u8] [12 => 15.96] VHS: SegmentLoader[main] [4-Stream(05)/index.m3u8] [16 => 19.96]

Results

Expected

The expected segment selection for a smooth playback in WPE in a set-top-box should be this one:

VHS: SegmentLoader[main] [3-Stream(04)/index.m3u8] [0 => 3.9599999999999973] VHS: SegmentLoader[main] [3-Stream(04)/index.m3u8] [4 => 7.959999999999997] VHS: SegmentLoader[main] [4-Stream(05)/index.m3u8] [8 => 11.959999999999997] VHS: SegmentLoader[main] [4-Stream(05)/index.m3u8] [12 => 15.96] VHS: SegmentLoader[main] [4-Stream(05)/index.m3u8] [16 => 19.96]

I understand that these kind of smooth quality switches are undesired for powerful desktop systems, though.

Error output

No errors.

Additional Information

videojs-http-streaming version

videojs-http-streaming 1.13.2

videojs version

video.js 7.8.1

Browsers

Platforms

Other Plugins

Other JavaScript

welcome[bot] commented 2 years ago

👋 Thanks for opening your first issue here! 👋

If you're reporting a 🐞 bug, please make sure you include steps to reproduce it. We get a lot of issues on this repo, so please be patient and we will get back to you as soon as we can. To help make it easier for us to investigate your issue, please follow the contributing guidelines.

gkatsev commented 2 years ago

Hi, thanks for opening up the issue. One thing to note is that the "fast quality change" stuff don't apply to regular playback. The option smoothQualityChange was there so that when a user took (or an API was used to take) an explicit action (going fullscreen or manually changing renditions), we would flush the buffer and re-download from the current position instead of continuing to the end of the buffer. This means that this option should not be used for regular playback operations, when the adaptive bitrate algorithm (ABR) is running. During normal playback, we should be downloading and appending at the end of the buffer only, and when we switch renditions we should still be appending at the end. In some cases, like the one you noticed, during rendition switches, we don't have a good idea of where to start playback in the new rendition. At that point, we make a conservative guess, which helps minimize potential gaps which can cause playback stalls. Your proposed change makes this guess less conservative. However, it does significantly increase risk on other platforms of gaps coming up. In addition, given that it's a few months shy of 2 years since we last touched the 1.x release line, I'd be less inclined to make updates to it. Is newer Video.js/VHS an option? Does the problem manifest with that? You mentioned that only published versions of Video.js can be used, anything stopping from using the latest Video.js?

eocanha commented 2 years ago

During normal playback, we should be downloading and appending at the end of the buffer only, and when we switch renditions we should still be appending at the end. In some cases, like the one you noticed, during rendition switches, we don't have a good idea of where to start playback in the new rendition. At that point, we make a conservative guess, which helps minimize potential gaps which can cause playback stalls. Your proposed change makes this guess less conservative. However, it does significantly increase risk on other platforms of gaps coming up.

Oh, I thought that you were always overlapping the first segment on purpose just to force a faster quality switch. Isn't there any good way to know if the selected next media segment overlaps with an already downloaded one?

What is conservative from the point of view of avoiding content gaps can be aggressive (in our case) from the point of view of how fast the switch happens (and the troubles it creates to the internal browser decoders). Anyway, I understand that the general behaviour should be tailored to the main platforms where your userbase is, not to a specific platform like ours.

Still, could a new configuration variable (with a different name from smoothQualityChange, I still don't know which one) be used in order to control the next segment to choose for the new quality? That way, it would be up to the video.js user (embedder) to choose the conservativeness of the quality change.

In addition, given that it's a few months shy of 2 years since we last touched the 1.x release line, I'd be less inclined to make updates to it. Is newer Video.js/VHS an option? Does the problem manifest with that? You mentioned that only published versions of Video.js can be used, anything stopping from using the latest Video.js?

I can't answer that question myself, but will forward the question to our user. In any case, the problem can also be reproduced in the latest video.js v7.17.1 (this time I took the logs from Chrome for simplicity, but the overlapping segments are there anyway):

VIDEOJS: DEBUG: VHS: SegmentLoader[main] > Appended segment [1/2309] segment start/end [0 => 6.64] startOfSegment [0] duration [6.64] timeline [0] selected by [getMediaInfoForTime (currentTime 0)] playlist [2-...3.m3u8]
VIDEOJS: DEBUG: VHS: SegmentLoader[main] > Appended segment [2/2309] segment start/end [6.64 => 12.48] startOfSegment [6.64] duration [5.84] timeline [0] selected by [mediaIndex/partIndex increment] playlist [2-...3.m3u8]
VIDEOJS: DEBUG: VHS: SegmentLoader[main] > Appended segment [2/2309] segment start/end [6.64 => 12.48] startOfSegment [6.64] duration [5.84] timeline [0] selected by [getMediaInfoForTime (bufferedEnd 12.48)] playlist [3-...4.m3u8]
VIDEOJS: DEBUG: VHS: SegmentLoader[main] > Appended segment [3/2309] segment start/end [12.48 => 19.64] startOfSegment [12.48] duration [7.16] timeline [0] selected by [mediaIndex/partIndex increment] playlist [3-...4.m3u8]

The original code where I was applying the fix in the proposed patch can be followed up to the latest http-streaming version. It would be really great to be able to find some polite way to avoid overlapping segments during a quality change, even if that means delaying the change to the next segment.

gkatsev commented 2 years ago

Since it's happening in latest Video.js/VHS, we can definitely consider having an option for this there, we're probably not going to backport it.

If we know whether we're overlapping content, maybe the option should be to never overlap content on a rendition change.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.