Clarify/remove 'stalled' and 'progress' events for MediaSource

[chcunningham]

Note on priority: Happy to see any spec change pushed to MSE vNext. v1 is nearing cut date and IMO this isn't critical.

'Stalled' makes good sense for non-MediaSource playbacks where the UA is responsible for the fetching the media. For MSE, the web app does the fetching and MSE is unaware that the fetch is being made and cannot know whether its progressing or stalled. The MSE spec states that "the download" or "bytes received" (in resource fetch algorithm) refer to data passed in via appendBuffer() and appendStream()". With this model, it is not possible for MSE to experience a stall (appends don't stall).

MSE spec makes no special mention of 'stalled' and when to emit (if ever) is ambiguous given the lines about "the download" and "bytes received" above. At the moment Chrome raises stalled if an MSE app goes 3 seconds without appending. This leads to confusion. I propose that we should not emit stalled for MSE (similar to the decision to not emit suspend for MSE in Issue 24).

When to emit 'progress' is also not clear. Raising this during an append is not as confusing as Chrome's behavior for 'stalled', but one could argue that raising progress is still confusing (given that app control the fetch) and also not useful for app developers. Apps know when they've called append and they can clearly know the outcome/completion of that append by other events/states (e.g. updateend event, SourceBuffer.buffered). I propose that we should also not bother to emit progress.

This is also a call for app developers to point out MSE use cases for these events that I may have overlooked.

One argument that I've heard is that some players may have UI that relies on these events dating back pre-MSE implementation. If this is a real concern then we have to decide on an implementation for these events that is meaningful. Showing a stall indicator because no append has been made in the last 3 seconds is not a great UI. Apps may append less frequently than that and still deliver a perfect playback experience. Additionally, apps would be better served basing any network UI on the network fetch they make prior-to appending to MSE.

[chcunningham]

Sorry, closed by mistake

[chcunningham]

reference: #24 fyi: @wolenetz @foolip @karlt @mounirlamouri

[wolenetz]

Thanks for filing this, @chcunningham. I suppose 'stalled' and 'progress' might make more sense to retain for MSE if we retained appendStream(). However, for V1 MSE, appendStream() is currently marked at-risk.

@jdsmith3000 , @mwatson2 Do you agree with my evaluation that this should be triaged to VNext, especially given the current tight MSE v1 timeline and our triage process? This would also afford implementors and web authors the opportunity to chime in on perhaps some useful 'stalled' or 'progress' MSE use cases in the interim (or provide evidence that 'stalled' and 'progress' indeed are underdefined/useless in MSE HTMLMediaElements). Please update my triage result as you deem fit.

[dmlap]

One data point: in video.js we trigger synthetic progress events based on content fetching already since that corresponds more closely to the definition in the HTML spec. I don't think stopping native progress or stalled events for MediaSource content would have an impact on us.

[wolenetz]

Thank you, @dmlap - that confirms @chcunningham 's hypothesis that apps should really take over 'progress' and 'stalled' logic in MSE cases.

Regarding potential use case for MSE 'stalled' and 'progress' if we use appendStream(), linking #14 to follow-up on that.

[ghost]

For MSE based playback, if HTML does not emit the stalled/progress events, the use case of a third app listening on HTML events for video element, will be broken/inconsistent. I feel, there should be a notification back to HTML, mandated by the standard, upon which HTML standard should generate these events.

[chcunningham]

swapnai - under what specific conditions do you think these events should be emitted for MSE?

[wolenetz]

Also, @swapnai, would those conditions possibly need to differ across MSE apps such that the MSE app should instead have the opportunity to own the app-specific logic and fire their own 'progress' and 'stalled' events at the mediaElement?

[ghost]

Sorry for the delay. Here are my comments:

Use-case: third-party video analytics and optimization solutions

Stalled events are currently part of the quality of experience analytics features of the HTML video element spec, providing insight into the performance and reliability of its internal content downloader components.

When stalled event triggers definition is left up to downloader/MSE component’s implementors, video analytics solutions need to establish their own API contract outside of W3C specs to either enforce a proper stalled event definition or identify each implementation to allow for proper interpretation of the stalled event.

Strengthening the contract between video element and MSE specs instead would ensure reliable stalled events across content downloader implementations and maintain interoperability of video analytics solutions.

[chcunningham]

Thanks @swapnai

For MSE in Chrome, the events are randomly tossed out as a result of code cruft. At this time they should not be interpreted to have any meaning for MSE (they're fine to use for non-MSE though).

In the discussion above we are at a loss for how give these events meaning for MSE. I understand your desired use case, but from my POV you're asking us to report on the status of something we (by-design) cannot know about.

In the comments above, video.js mentions that they (who manage the download) create synthetic progress/stalled events. I think this is the best path for your use case. You can lobby for the apps you're monitoring to do similar to video.js.

@wolenetz - can you help me add additional stake holders? shaka? edge/ghecko?

[ghost]

@chcunningham, This is inherently a w3c standard definition issue and the solution is possibly be to re-think the flow of events or to extend to include generic event capability to MSE based implementation as well. Since players are only downloading and chunking up the content, so the browser can play it in a format browsers know to play, the playback lifecycle should still be reported fully by the browser. There should be a clear distinction between downloading and playback events. The w3c standard should include events, an MSE implementation need to emit for the download and chunking part ( ones the browser would do for native content type) to comply with w3c. This will ensure uniformity and interoperability.

[chcunningham]

Since players are only downloading and chunking up the content, so the browser can play it in a format browsers know to play, the playback lifecycle should still be reported fully by the browser.

In my view the playback lifecycle is fully reported by the browser. However, it is not possible for MSE report on the state of the download that it isn't managing (at least in the case of appendBuffer).

The w3c standard should include events, an MSE implementation need to emit for the download and chunking part ( ones the browser would do for native content type) to comply with w3c.

I don't think MSE must emit these events in order to "comply". For instance, it is already decided not to emit the suspend event for the same reasons I'm giving above for stalled/progress to not be emitted.

[ghost]

Sorry, I meant the entity responsible for downloading and chunking should emit those events. NOT the browser. @huchunming Should I edit my last post, to make it clear?

[riksagar]

I would not expect those events to be removed, not purely because MediaSource were being used. I think that adds unnecessary complexity for client players when you selectively remove the events based on the media delivery protocol (e.g., DASH 'v' single MP4 file over HTTP).

My player than supports DASH and progressive download won't want to treat sources differently. If I use those events as UI triggers, it seems it would add unnecessary complexity. Now I need to propagate knowledge of the source throughout more of the application.

However, MSE does need a better way to provide 'progress' notifications to the MediaElement, esp. in the situation where the fragments being downloaded are large and take several seconds to download. That way, the behavior of "progress" and "stalled" can be made consistent across HTTP and MediaSource based streams.

[chcunningham]

However, MSE does need a better way to provide 'progress' notifications to the MediaElement...

@riksagar, this would likely require a new API where apps tell MSE how the download is progressing. For apps, calling such an API would be at least as complex as synthesizing the progress/stalled events. If your player sits above a library like video.js which synthesizes the events, then you can treat everything the same. If your rolling your own MSE code, you could synthesize the events. Either way, UI code can be re-used.

However, for V1 MSE, appendStream() is currently marked at-risk.

@wolenetz, how is appendStream doing these days? Also, you marked this as "needs author input" - does that mean my input or spec author input?

[paulbrucecotton]

how is appendStream doing these days?

See ISSUE-14. Due to lack of progress on the Streams API we have removed the "at risk" feature from MSE.

[wolenetz]

@chcunningham: Needs web author input

On Sep 6, 2016 5:57 PM, "Paul Cotton" notifications@github.com wrote:

how is appendStream doing these days?

See ISSUE-14 https://github.com/w3c/media-source/issues/14. Due to lack of progress on the Streams API we have removed the "at risk" feature from MSE.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/w3c/media-source/issues/88#issuecomment-245143041, or mute the thread https://github.com/notifications/unsubscribe-auth/ALCtw4jDFtGx5VroCqXlSeItBBkESHxvks5qngvsgaJpZM4IwYw- .

[chcunningham]

Friendly ping on this. @wolenetz what are the obligations for soliciting / blocking on further input.

[wolenetz]

This sounds like something ripe for incubation in MSE vNext. Next steps would be to propose incubation via WICG (this may be something I'll take up, pending vNext work prioritization among other similar features).

[chcunningham]

Taking a closer look, I realize that the current form of the MSE + HTML5.x specs is such that stalled events would not be emitted by in MSE use case.

From MSE spec, 2.4.1 Attaching to media element

If the resource fetch algorithm was invoked with a media provider object that is a MediaSource object or a URL record whose object is a MediaSource object, then let mode be local, skip the first step in the resource fetch algorithm (which may otherwise set mode to remote) and add the steps and clarifications below to the "Otherwise (mode is local)" section of the resource fetch algorithm.

This means MSE's resource slection algo skips all of "If mode is remote" in the HTLMMediaElement resource fetch algorithm, which includes skipping over all mentions of stalled timer and stalled event. So for MSE, stalled timer is never started, and stalled events are never raised (at least, by the spec). Given the above, I will update Chrome to not fire stalled for MSE. FYI Firefox folks - you have the same bug as us. Haven't checked the others (easy repro steps here).

One lingering cleanup is that the current MSE recommendation also includes this NOTE:

An attached MediaSource does not use the remote mode steps in the resource fetch algorithm, so the media element will not fire "suspend" events. Though future versions of this specification will likely remove "progress" and "stalled" events from a media element with an attached MediaSource, user agents conforming to this version of the specification may still fire these two events as these [HTML51] references changed after implementations of this specification stabilized.

The wording "may" still fire is compatible with my plans to not fire. But now that the HTML5.1 spec has stabilized (and salled is still only for remote), we might update this note. @wolenetz - should we track that separately and close this bug?

[wolenetz]

@chcunningham tracking that note's cleanup separately SGTM - just refer to your comment, above, from a new spec issue.

[wolenetz]

I suggest that, for V2, we scope this part for inclusion: "The wording "may" still fire is compatible with my plans to not fire. But now that the HTML5.1 spec has stabilized (and salled is still only for remote), we might update this note. @wolenetz - should we track that separately and close this bug?"

The rest of the previous discussion led to this result, and we have separate issue tracking potential API for MSE to let app know when underflow/stall is approaching, which in my opinion is a better fit for an MSE spec solution to helping apps understand when underflow might be approaching (https://github.com/w3c/media-source/issues/40).

So, I'll triage this for V2 to update the existing note in the spec.