Support containerless formats

joeyparrish commented 4 years ago

Have you read the FAQ and checked for duplicate open issues? Yes

Is your feature request related to a problem? Please describe.

Some HLS content uses containerless formats, such as raw AAC or raw MP3. In these formats, there is no timestamp information. This not only complicates the building of the segment index in the HLS parser, but it requires special timestamp management at the MediaSource level because these formats implicitly force their SourceBuffer into sequence mode.

Without special handling, the timestamps are never right after seeking, even if we synthesize correct SegmentReference timestamps during the playlist parsing.

Describe the solution you'd like

We should skip these formats in the HLS parser's timestamp discovery stage. We should also mark these streams as being containerless. Finally, StreamingEngine should use this containerless flag to manage timestamps at the MediaSource level (set timestampOffset every time the buffer is cleared, and include the first appended segment reference's start time).

Describe alternatives you've considered

We could reject containerless formats, but I'd prefer to improve our support for them for the sake of legacy HLS content.

Additional context

Related to #1083 (raw AAC). Sequence mode docs: https://developer.mozilla.org/en-US/docs/Web/API/SourceBuffer/mode Spec text around sequence mode: https://www.w3.org/TR/media-source/#dom-appendmode-sequence

avelad commented 3 years ago

@ismena , can you consider this issue for your next roadmap planning? (github label) There is a lot of legacy content to support and this would be very useful.

ismena commented 3 years ago

Yes!

johansvedberg commented 3 years ago

Does the addition of the P2 label mean that this will be worked on in the near future?

theodab commented 3 years ago

I wouldn't count on it. At the moment we have 21 P1 issues and 88 P2 issues to get through, and we're a small team.

Ankita20 commented 2 years ago

Hello

Is this issue resolved? We are trying to playback hls video on shaka player but not able to do so because of this issue. Any updates on by when this will be resolved and which version of shaka to use then?

theodab commented 2 years ago

This issue is not currently being worked on. We do not have any estimates for when it will be resolved.

Honestly, given how much our team size has shrunk since this issue was filed, realistically it might be best to wait for community contributions. I'll change the tags on the issue.

joeyparrish commented 2 years ago

Quick correction: @michellezhuogg started this work before going on leave, and we may be able to complete the first phase of it in the next few weeks. I need to plot out our schedule and priorities for the near future, as we've had a lot of changes internally lately.

joeyparrish commented 2 years ago

The bulk of this feature work should be merged very soon. We may need to follow up with a couple minor changes after that, and of course, we'll need testing and feedback from the community. Looking forward to releasing this in v3.4!

avelad commented 2 years ago

@theodab One question about your commit https://github.com/google/shaka-player/commit/36d0b5484fad68dc1d640fbddf2fae3e1eb7169b

According to https://developers.google.com/web/updates/2016/03/mse-sourcebuffer (Cache: https://webcache.googleusercontent.com/search?q=cache%3A2Lf2cEE2UB4J%3Ahttps%3A%2F%2Fdevelopers.google.com%2Fweb%2Fupdates%2F2016%2F03%2Fmse-sourcebuffer%20&cd=1&hl=es&ct=clnk&gl=es) sequence mode is supported in Chrome 50+, has it been tested to work in Tizen 3 (https://developer.samsung.com/smarttv/develop/specifications/web-engine-specifications.html)? Tizen 3 (2017) uses Chrome 47.

theodab commented 2 years ago

Oh, I'm not sure. I came in to finish up this CL since @michellezhuogg is on leave. I didn't test it on Tizen myself, at least?

avelad commented 2 years ago

@joeyparrish , can you help us with this?

avelad commented 2 years ago

Thinking, another option is to detect if it supports the sequence mode when registering the HLS parser, so on devices that don't support it they can use the native player (src=).

joeyparrish commented 2 years ago

We don't have access to a Tizen TV right now, due to some issues with our lab space at the office. Someone else would have to test that right now. Sorry!

joeyparrish commented 2 years ago

@avelad, @Ankita20, @johansvedberg, please try the latest code from master or the nightly build on appspot to see if sequence mode is working correctly for your HLS streams, especially HLS live streams. The code is feature-complete now, and we would like to release v3.4 when we have positive signals from the community. Thanks!

avelad commented 2 years ago

Using Chrome 47 (same as Tizen 3) on browserstack, the following stream doesn't work: https://d3rlna7iyyu8wu.cloudfront.net/skip_armstrong/skip_armstrong_multi_language_subs.m3u8

https://user-images.githubusercontent.com/8983024/153154720-0f49b819-b3ed-4577-b437-c2f7228ceeb3.mp4

There is also a problem in the Demo version that containerless content is not supported (happens in all browsers)

Note I have tested on the latest versions of Chrome, Safari, Firefox, and Edge, as well as Tizen 5, WebOS 6, and the above VOD stream works fine on all of them.

theodab commented 2 years ago

Oh. I forgot to update shaka.media.MediaSourceEngine.probeSupport to probe support for containerless formats.

avelad commented 2 years ago

I tested the next stream: https://nightly-dot-shaka-player-demo.appspot.com/demo/#audiolang=es-ES;textlang=es-ES;uilang=es-ES;asset=https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8;panel=CUSTOM%20CONTENT;build=uncompiled

I have found that if I try to play it I get the error:

If I disable the video (config), the audio plays correctly.

Another issue, the subtitles are not shown, and it seems like a regression, in 3.3.1 they work correctly.

3.3.1 https://shaka-player-demo.appspot.com/demo/#audiolang=es-ES;textlang=es-ES;disableVideo=true;uilang=es-ES;asset=https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8;panel=CUSTOM%20CONTENT;build=uncompiled;vv

Nightly: https://nightly-dot-shaka-player-demo.appspot.com/demo/#audiolang=es-ES;textlang=es-ES;disableVideo=true;uilang=es-ES;asset=https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8;panel=CUSTOM%20CONTENT;build=uncompiled;vv

theodab commented 2 years ago

Yeah, I've been looking into that.

The closed captions in that test stream don't parse correctly, it's true, but I don't think it's related to sequence mode. For example, if you run in a version before we started using sequence mode, the error still happens. https://v3-2-3-dot-shaka-player-demo.appspot.com/demo/#audiolang=es-ES;textlang=es-ES;uilang=es-ES;asset=https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8;panel=CUSTOM%20CONTENT;build=uncompiled I think this is a bug with the CEA 708 parser, instead. I'll open a separate GitHub issue for it.

As for the subtitles, they are shown, they are just offset by 10 seconds. That Apple sample content has a 10 second gap at the start of the presentation. Now that we are using sequence mode the gap is effectively normalized away, but we don't normalize the time of the captions so they end up starting 10 seconds after they should. We need to ignore the X-TIMESTAMP-MAP tag in the webvtt files when in sequence mode, I think?

theodab commented 2 years ago

Actually, the captions error might be the same as #3659? Not sure, I'll have to check the content they provided for that to see if it looks like the same problem. If it is the same, I shouldn't open a new issue.

avelad commented 2 years ago

The root cause is the same (I think), but in the open issue it talks about H265 and here it is H264.

joeyparrish commented 2 years ago

We have an internal report of an HLS live stream with A/V sync issues in the master branch. @theodab is looking into it.

After discussing HLS live implementation with @JulianDomingo, here are things we should consider doing next:

Parse EXT-X-PROGRAM-DATE-TIME and use it for A/V sync. Since HLS will now always use sequence mode in MSE, we should make sure our assigned timestamps are aligned with EXT-X-PROGRAM-DATE across streams.
Extrapolate EXT-X-PROGRAM-DATE-TIME if it's missing. The spec says we should extrapolate forward from segments with EXT-X-PROGRAM-DATE-TIME to those without, based on segment duration. I think we could also extrapolate backward if needed, to approximate the wall-clock-time of the first segment if it doesn't have one in the playlist for some reason.
If EXT-X-PROGRAM-DATE-TIME is missing, use a combination of target duration and media sequence number to approximate alignment between playlists.
Add a config to force us to ignore EXT-X-PROGRAM-DATE-TIME and only use sequence numbers. Anecdotally, a similar options was useful for certain content in the legacy Cast player.

avelad commented 2 years ago

@theodab I've seen that support for EXT-X-PROGRAM-DATE-TIME has already been added, what's missing?

joeyparrish commented 2 years ago

I think we're ready to close this, but we could use some feedback from the community before launch. @avelad, @johansvedberg, and anyone else reading this, have you been able to try the new implementation with your own live streams?

avelad commented 2 years ago

I have tried several HLS Live streams and I see a correct result in all of them.

joeyparrish commented 2 years ago

Great news! Thank you. We will close this issue, then.

If anyone has content that doesn't work correctly after v4 is released, please file a new issue with links to playlists.

shaka-project / shaka-player

Support containerless formats #2337