Low-Latency: Initialization

[haudiobe]

Problem: After initialization media decoder, it is important that as soon as the first media is added, decoding and presentation is started as quickly as possible. Initialization may be done ahead of time.

Content: CMAF content with presumably sufficient information to initialize codec

Test: Add CMAF Header. Wait. After adding content and pressing play, start of playback must be below a certain threshold.

Potential problem: Is CMAF Header sufficient to initialize codec - also check CMAF specification.

[jpiesing]

This is simple to do badly and hard to do easily. The current spec already includes language like this;

The start-up delay should be sufficiently low, i.e., TR [k, 1] – Ti < TSMax.

But ...

• This says "should" so therefore will be ignored
• TSMax is not defined so an implementer who fails a test will just argue that the test is stricter than the spec
• This is in section 8, "single track media playback requirements".
• There's no equivalent in section 9 for CMAF presentations which is IMHO more important

The reason it's hard to do properly is that there doesn't appear to be any event in MSE that would make initialisation visible.

Right now the obvious possibility would seem to be running the same test with avc1 and avc3 with an artificial delay between when the initialization segment is appended and when the media data is appended. Then measure the time from when the first media data is appended to when the first content is visible.

[dsilhavy]

• Suggest to also keep in mind that the appended data should have a certain minimum length. Otherwise playback might not start because the platform requires more data.
• Suggest to use HTML5 video element ready states as an indicator for when playback is initialized: https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/readyState. This needs to be investigated further in terms of "which state change is triggered once the init is appended"
• I also find it hard to define TSMax. For instance, I would expect the playback start time to increase depending on which encoding profile is used? H.264 high profile might lead to longer decoding times?

[jpiesing]

• Suggest to also keep in mind that the appended data should have a certain minimum length. Otherwise playback might not start because the platform requires more data.

This is not about decoding but about initialization. Suppose decoder initialisation would take 1.5s. Some implementations might initialise the decoders when the initialization segment is appended. Other implementations might delay initializing the decoders until some media segments have been appended. On an implementation where the decoders are initialized without waiting for media segments then playback will be at least 1.5s closer to the live edge than implementations where the decoders are only initialized when some media segments have been appended. On implementations where decoders are only initialized with media segments,

• Suggest to use HTML5 video element ready states as an indicator for when playback is initialized: https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/readyState. This needs to be investigated further in terms of "which state change is triggered once the init is appended"

I fear there's no easy way to know if the decoder(s) actually has been initialized. For video, it may be possible to determine when re-initialization occurs by measuring when the last frame of a previous stream is taken down but I can't see anything for initialisation or for audio.

• I also find it hard to define TSMax. For instance, I would expect the playback start time to increase depending on which encoding profile is used? H.264 high profile might lead to longer decoding times?

Decoder initialization should be independent of encoding profile.

[dsilhavy]

• Suggest to also keep in mind that the appended data should have a certain minimum length. Otherwise playback might not start because the platform requires more data.

This is not about decoding but about initialization. Suppose decoder initialisation would take 1.5s. Some implementations might initialise the decoders when the initialization segment is appended. Other implementations might delay initializing the decoders until some media segments have been appended. On an implementation where the decoders are initialized without waiting for media segments then playback will be at least 1.5s closer to the live edge than implementations where the decoders are only initialized when some media segments have been appended. On implementations where decoders are only initialized with media segments,

• Suggest to use HTML5 video element ready states as an indicator for when playback is initialized: https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/readyState. This needs to be investigated further in terms of "which state change is triggered once the init is appended"

I fear there's no easy way to know if the decoder(s) actually has been initialized. For video, it may be possible to determine when re-initialization occurs by measuring when the last frame of a previous stream is taken down but I can't see anything for initialisation or for audio.

• I also find it hard to define TSMax. For instance, I would expect the playback start time to increase depending on which encoding profile is used? H.264 high profile might lead to longer decoding times?

Decoder initialization should be independent of encoding profile.

Thanks, understood. As you mentioned before, avc1 vs. avc3 might also cause a difference then?

[haudiobe]

part of new test cases.