CMAF Low Latency Chunks

[fadomire]

Hello,

i wanted to know if it is planned to support CMAF Low latency chunks ?

I saw reference to that in the following ticket comparing dash.js and shaka, but no status for shaka

1351

thanks for the work by the way

[theodab]

We will look into this feature. We will add it if we can, but we'll need to investigate how it would work with MediaSource.

[fadomire]

thanks for the reply

it may be interesting to check how dashjs made their implementation when the subject come in the roadmap

one thread discussing low latency specifics : https://github.com/Dash-Industry-Forum/dash.js/issues/1474

[PatrikCarlander]

Is it possible to priorities CMAF support? CMAF is what the OTT industry asking for now. At NAB-Show (April 6 - 11, 2019 | Las Vegas) we have to show CMAF with Dashif instead of Shaka and it's sad.

[PatrikCarlander]

The live stream simulator has available test vectors which expose low latency chunking. This stream has 8s segments with 1s chunks:

https://vm2.dashif.org/livesim-chunked/chunkdur_1/ato_7/testpic4_8s/Manifest300.mpd

It is possible to set the ato and chunkdur to floating point numbers like chunkdur_0.5/ato_7.5/ in the URL. The original content is 30Hz, so setting ato to 7.9 and chunkdur to 0.1 is also possible.

[joeyparrish]

Hi, everyone. Just to clarify: CMAF is already supported. Low-latency CMAF will be added later this year.

[PatrikCarlander]

It was Low-latency CMAF we asked for. thanks for the reply

[joeyparrish]

@PatrikCarlander, yes, but I was clarifying based on your first comment "Is it possible to priorities CMAF support". But I think we are all on the same page now.

[Romain-Bernier]

Any update about Low Latency CMAF ? Thanks

[joeyparrish]

We haven't begun work on this yet. We anticipate working on this in Q3 2019.

[ramoncaldeira]

+1 for low-latency DASH

[PatrikCarlander]

Q3 2019 is ending soon, is this feature postponed? LL is a highly requested feature from our customers. We need this in Shaka to get LL in chromecast too.

[ismena]

Hi @PatrikCarlander So, there's good news and there's bad news. The bad news is, we got entangled in pre-work for this feature (cleaning up and optimizing our hls code), so we weren't able to start actual work when we'd hoped to. The good news is, that the pre-work is almost done (I think I saw the last optimization chunks go into into code review last week :)). We've already allocated time for LL and the developer has started research. I'll sync with them next week (after the research) to see if we can give you a better estimate. I will also do my best to cajole our TL @joeyparrish into making LL one of our biggest priorities for the rest of the year.

[PatrikCarlander]

This can be a good input to define the scope. ExoPlayer Low Latency design doc

[michellezhuogg]

Hello @PatrikCarlander ,

I saw that both Community Hls spec and Apple Hls spec are mentioned in the document. Which one are you looking forward to use?

[PatrikCarlander]

@michellezhuogg For a start ULL-DASH, after that we will see if Apple LLHLS will replace Community LHLS.

[quanghuy1288]

waiting for hls cmaf low-latency :)

[buhlaze]

I am confused about cmaf. Is Cmaf just a mpeg-dash manifest with Availabilitytimeoffset added to it? I want to use it instead of mpeg-dash. Also, how do you make a CMAF manifest with ffmpeg with or without shaka player? I have looked everywhere and can't find an example of how to do so.

[leandromoreira]

@buhlaze CMAF is a container (much like .TS or .MKV or yet .MP4)

[buhlaze]

low latency CMAF, in order to take advantage of it, does it just involve adding Availabilitytimeoffset to your dash manifest in the BASEURL with segmented video X number of seconds and a player that supports low latency, for example LowLatencyEnabled? Could I please get an explanation?

[leandromoreira]

@buhlaze you can read more about it (there is even a talk that I think you should watch).

[joeyparrish]

Shaka Player v2.5 and v2.6 (upcoming) don't support low-latency mode yet. We'll be working on it soon, though.

[sridhard]

Hi,

Any updates on this feature?

[joeyparrish]

Sorry, but there's nothing new to report at this time.

[joeyparrish]

Work is underway and scheduled for v3.1.

[joeyparrish]

Forked from https://github.com/google/shaka-player/issues/2505#issuecomment-641766885 : @avelad wanted information on our plans for low-latency DASH.

I find it hard to follow your diagram. I found this one from Frauenhofer to be much easier to read:

[joeyparrish]

(Original context of the diagram: https://websites.fraunhofer.de/video-dev/dash-js-low-latency-streaming-with-cmaf/)

In this diagram, the availabilityTimeOffset gets us fetching segment 4 instead of segment 3, even without using fetch and ReadableStream to pipeline data to MediaSource. This reduces latency from 8 seconds to 3 seconds in this example.

The difference with ReadableStream is that we would be able to start playback after one chunk instead of one segment, so startup latency would be reduced by this. However, since DASH (unlike LL-HLS) does not give separate URIs for chunks, we can only get approximately chunk-based streaming with ReadableStream, and it comes at a cost that we don't have to pay with HLS.

If I fetch a chunk from an HLS playlist through a partial segment tag, I know that is a complete chunk. I don't have to use ReadableStream, and I can treat it like any other segment. I can run response filters on it, I could potentially transmux it (does LL-HLS allow for TS?), and I know I could start playback with that data alone.

When I start fetching a DASH segment with ReadableStream, I don't get any guarantee of the boundary of the data when I read from that stream. It may be a complete CMAF chunk, or 0.62 chunks, or 1.8 chunks, etc. It's an arbitrary number of bytes. So response filters must be disabled.

In addition, we have to change our network plugin interface to allow for ReadableStream. We haven't done the work to determine if that can be done in a backward-compatible way.

So before we commit to the cost of disabling response filters and transmuxing, as well as redesigning a plugin interface, we want to do the most minimal proof-of-concept to measure the benefit.

[avelad]

@michellezhuogg, I have seen your last commit and I have a question, why use a low latency configuration parameter instead of applying it whenever the manifest has low latency?

[joeyparrish]

We decided to use a flag for low-latency mode for now, partly for backward compatibility reasons. When we implement DASH low latency, for example, using ReadableStream to pipeline data to MediaSource, this will necessarily mean disabling segment response filters and transmuxing, which might surprise developers or break some applications.

So low-latency mode will be default to false and apps will need to opt-in and effectively acknowledge that they are giving up certain other features.

If we see low-latency-enabled content and the flag is set to false, though, we should probably issue a warning log pointing to a tutorial explaining these tradeoffs and how to opt-in.

[avelad]

@michellezhuogg, I have looked at the commits of the implementation and I have doubts ...

In the commit https://github.com/google/shaka-player/commit/a534dca88351466164b64389530ab53dcc6ffb39 it is mentioned that in order to use low latency you have to set inaccurateManifestTolerance to 0 and change the value of rebufferingGoal.

I understand that inaccurateManifestTolerance could be set to 0 automatically when the lowLatencyMode setting is set to true, otherwise this can be done at the application level and is not a problem.

The problem is that setting a rebufferingGoal to 1 partial segment may not be known in advance and even less when there are multiple providers at the same time.

Since the lowLatencyMode configuration is already there, I understand that all this should be changed with this config.

Is there something that I am not understanding well?

[joeyparrish]

Since the lowLatencyMode configuration is already there, I understand that all this should be changed with this config.

We could choose to also change the inaccurateManifestTolerance and rebufferingGoal settings when lowLatencyMode is set to true, or to ignore them instead. But it's hard to see what choice would follow the principle of least surprise.

If we override or ignore those settings in low latency mode, would developers be surprised that those settings changed silently or were ignored? Or would developers be more surprised that they have to change additional settings to make low latency mode effective?

We have already decided to disable response filters for segments in low latency mode, since we will use ReadableStream for DASH and not fetch entire segments that could then be filtered. Is this surprising? Would this decision align better with changing/ignoring inaccurateManifestTolerance and rebufferingGoal automatically, since we already have to ignore response filters for low-latency segments?

We would be happy to have everyone's feedback on this before we launch v3.1.

The problem is that setting a rebufferingGoal to 1 partial segment may not be known in advance and even less when there are multiple providers at the same time.

Any small, non-zero value should be fine. You don't have to know the exact size. For example, you could set it to 0.001. If we fetch a partial segment, that will put us over the rebuffering goal and playback will begin as soon as that first partial segment is consumed.

[avelad]

For me if response filters are disabled in DASH, I would expect the setting to be applied by default when low latency is set.

Another option is to add a new configuration autoConfLowLatency that if true, apply the configuration change automatically.

[joeyparrish]

@michellezhuogg and I have taken some time to revisit our decision on configuration.

TL;DR: Developers will only have to change one setting (streaming.lowLatencyMode) to activate low-latency mode. We'll change the others automatically, but let you change them back if you want.

Our original thought was that it would surprise people to change their configuration (inaccurateManifestTolerance and rebufferingGoal) automatically, and that they might still want to be in control of those values for one reason or another. We definitely didn't want to ignore those values in low latency mode, which would take control away from developers.

But we've decided that instead of requiring developers to set them to make low-latency mode work, we will instead make a change to the low-latency mode setting log a warning and automatically change inaccurateManifestTolerance to 0 and rebufferingGoal to 0.001, as well. If a developer wants control of those settings, they can set them in the same or subsequent call to configure(), but they won't be surprised to have to set three settings instead of one. The common case will be easy, and developers can still have control if they want it.

Since the demo app applies all settings in the control panel at once (in some cases, such as demo/main.js in drmConfiguration_()), the demo app will need to have this logic, as well. When the user changes the lowLatencyMode toggle to true, the UI for inaccurateManifestTolerance and rebufferingGoal will need to change, as well.

docs/tutorials/config.md also needs to be updated. I think in light of the simplifications proposed here, the "low latency" section of that doc could just be removed.

[avelad]

I have another case that I would like to know how it is treated. If the low latency mode is set and the stream is not low latency, what will be the behavior of the player?

[joeyparrish]

If the content is not low-latency, some things will not be improved. Others may be, depending on the encoding of the content. And enabling low-latency mode will come with a cost in that some features will become unavailable.

For DASH, non-low-latency content would be a stream without availabilityTimeOffset. The availability of segments will be as it always was, without this extra adjustment indicated in the manifest.

For HLS, non-low-latency would be a stream without partial segments or delta updates. Only full playlists and full segments would be requested.

For both DASH & HLS, with low-latency mode enabled, response filters for segments would be skipped, which is the main cost of low-latency mode.

The main benefit of low-latency mode, independent of the manifest or playlist features, would be using ReadableStream to process chunks as they become available. Each pair of moof+mdat box within the segment would be processed as it becomes available.

If the segment only contains a single moof+mdat, then there is no benefit from ReadableStream. Also, if the platform does not have the Fetch API, XHR will be used instead, and there will be no benefit. IE11 does not have Fetch, and some embedded devices may not have it, either. All other desktop and mobile browsers seem to have Fetch. https://caniuse.com/#feat=fetch

[joeyparrish]

We believe low-latency support is now feature complete, though there is still some discussion in PR #2861 about further improvements.

[kanongil]

The preload hints don't support the BYTERANGE-START attribute. It incorrectly uses a non-specced BYTERANGE attribute.

[joeyparrish]

@kanongil, thanks for pointing that out! It has now been fixed in the master branch.

[vlee-harmonicinc]

It is great that shaka player support ll-hls! Apart from delta update (_HLS_skip), is there any plan for Blocking Playlist Reload (_HLS_MSN, _HLS_part) as well? https://tools.ietf.org/html/draft-pantos-hls-rfc8216bis-08#section-6.2.5.2

[joeyparrish]

Our internal streaming model requires us to update all playlists, rather than just the media playlist for the currently-selected streams. So we decided that blocking playlist updates might not make sense for Shaka Player at this time.

@michellezhuogg, let's discuss this Tuesday and double-check our assumptions to make sure we made the right call on that.

[joeyparrish]

@michellezhuogg and I are not sure if we made the right decision about blocking playlist updates. We originally decided not to do them because we're updating all media playlists.

It might work to block on a media playlist update, even if we are going to update all playlists serially. If all are going to be blocked on the same segment number, for example, the first one would block, and subsequent playlist requests would not, since that segment would already be ready in all the others by the time the first was ready.

At least, that's what we're thinking now.

I'm also wondering if there are other configs that would need to be changed to accommodate this. Manifest request timeout comes to mind, but we could also override that per-request in the HLS parser without changing the configuration per se.

Any thoughts on this? Anyone interested to experiment with the HLS parser to see what breaks if we block?

[vlee-harmonicinc]

A client MUST NOT assume that segments with the same Media Sequence Number in different Variant Streams or Renditions have the same position in the presentation; Playlists MAY have independent Media Sequence Numbers. https://tools.ietf.org/html/draft-pantos-hls-rfc8216bis-07#section-6.3.2

For example Apple's LL-HLS CMAF stream have different duration and MSN for audio and video. Blocking one media playlist still works if playback latency large enough to tolerate the difference, but probably not a prefect design.

I tested implement blocking playlist reload via saving next MSN and next part number in streamInfo and marking preload in segmentReference. It seems works, but there is some other issue related to lint/compile :P

By the way, how about #1936: Lazy-load HLS media playlists on adaptation ?

A client MUST load the Media Playlist file of every Rendition selected for playback in order to locate the media specific to that Rendition. But, to prevent unnecessary load on the server, it SHOULD NOT load the Playlist file of any other Rendition.

Lazy-load is not mandatory and not directly related to this issue, but LL-HLS better have lazy-load in practice. Since the partial segment of LL-HLS is short, update frequency of LL-HLS is much higher than standard HLS. For example, if playing a LL-HLS stream with 2 audio 5 video and 1s partial duration, updating all playlist requires downloading. >7 media playlist per second, while only 2 selected playlists are necessary.

[joeyparrish]

The problem with lazy-loading is Shaka Player's current architecture. Manifest parsers don't know what the currently-selected stream is or when it changes, so there is no good way to know which playlist needs to be updated. We designed manifest parsing to be completely decoupled from streaming, but it seems that in this case, they are too decoupled.

[avelad]

I think it would be nice to add a table in the readme of the features needed for low latency to work. For example in DASH it is necessary the ReadableStream API

[maximk-1]

Hi shaka team! Do you plan to release LL-DASH support this year?

[joeyparrish]

@htleeab wrote:

For example Apple's LL-HLS CMAF stream have different duration and MSN for audio and video. Blocking one media playlist still works if playback latency large enough to tolerate the difference, but probably not a prefect design.

Based primarily on this, and on our current architecture (manifest parsers completely independent of streaming), @michellezhuogg and I have decided not to add blocking playlist updates for now. If a feature meant to reduce latency on updates ends up adding latency because of how it interacts with our existing architecture, it's best not to do it for now.

@mkozlov1 wrote:

Do you plan to release LL-DASH support this year?

Well, we certainly planned it that way, but has anyone else noticed how 2020 has gone so far? We're doing what we can with very limited resources (team members temporarily allocated to other internal priorities). We intend to wrap up the features (basically done now) and close enough serious bugs to feel confident about the stability of the master branch before we release v3.1. We hope that can be accomplished before the end of the year, but if it takes until January 15th to make v3.1 usable, we'd rather have the release work as intended than have it technically come out before midnight on Dec 31.

With that, I think our initial LL feature work can be called complete. We're open to discussions of blocking playlist updates and other features with accompanying architecture changes at a later date.

Thanks!

[michellezhuogg]

Hello everyone,

The Low Latency HLS and Low Latency Dash features are ready for testing on our Nightly build demo: https://nightly-dot-shaka-player-demo.appspot.com/demo/#audiolang=en-US;textlang=en-US;lowLatencyMode=true;uilang=en-US;panel=HOME;build=uncompiled We would like to get a few sample manifests for testing, or any bug report before our official release of v3.1. Thank you!

[kanongil]

While it seems to be somewhat workable with LL-HLS on a good connection, the ABR experience quickly breaks down. There are several minor/major issues, but I won't list them, as it seems that this has hardly been tested. Note that using the Chrome network throttling is not representative of real-world issues, since it always triggers a navigator.connection 'change' event, which resets the estimator.

I will mention one non-ABR problem: Playback does not respect the INDEPENDENT=YES flag, and starts from whatever fragment, causing playback issues.

Anyway, I probably won't be using this as long as it doesn't support blocking playlist reloads.

[michellezhuogg]

@kanongil ,

Thank you for your feedback! Would you please share some sample low-latency manifests with us, so that we can test and triage the minor/major issues that you mentioned? You can send them to shaka-player-issues@google.com if they're private. We'll also look into the problem with INDEPENDENT=YES flag. Thank you!

[michellezhuogg]

Closing the issue due to inactivity. If anyone has further questions or comments, please open another new issue. Thank you!