Revisit how fatal and non-fatal media network/decode/abort errors work

[foolip]

Pointed out by @ddorwin in https://github.com/w3c/encrypted-media/issues/100#issuecomment-157551930 but a problem not specific to EME.

If my interpretation is correct, there is no explicit algorithm for reporting MEDIA_ERR_DECODE once the "entire resource gets loaded and kept available" even if the entire resource has not yet been decoded/played yet to determine that the media data is not corrupt.

This matches my reading of the spec as well, but the problem isn't just that it's not allowed to report decode errors after the resource fetch algorithm has finished, it's also that decode errors during playback are transient. If one seeks into an area that cannot be decoded, many media frameworks will simply skip ahead until it finds some data that it can decode. Even if the decode error is reported, after seeking to a good area, it doesn't make much sense for mediaElement.error.code attribute to still be MEDIA_ERR_DECODE.

Next step: investigate what implementations actually do when playing past bad data, both while the fetch is still ongoing and when it is not. A "render media data" algorithm as suggested by @ddorwin could make sense.

[foolip]

@wolenetz and I discussed this a little bit in https://code.google.com/p/chromium/issues/detail?id=472253#c5

there is no way for mediaElement.error to revert back to null other that the media element load algorithm, which is invoked by load() and setting src/srcObject. If you think it would make sense, we could change the spec to perhaps clear it after seeking.

I'm not sure if there are other ways of recovering from an error, does it ever make sense to report an error and then just keep decoding until you get to good data again?

[wolenetz]

From web developer perspective, it would certainly be useful to have some mechanism for discovering non-fatal media parsing/decoding/playback errors. In Chrome, we've been adding these as DEBUG and INFO logs to chrome://media-internals (in addition to various debug-build log messages that pre-existed our more recent efforts to expose these logs in release builds).

For fatal errors, Chrome is tracking addition of custom error log entries to the devtools console in https://code.google.com/p/chromium/issues/detail?id=472253.

IIRC, Edge/IE has something like "msExtendedError" that helps indicate to web apps more detail like this, though I'm not sure if that error is populated for just fatal errors, or if it might also be populated for recoverable issues. @jdsmith3000 for comment on this piece at least.

[foolip]

IIRC, Edge/IE has something like "msExtendedError" that helps indicate to web apps more detail like this, though I'm not sure if that error is populated for just fatal errors, or if it might also be populated for recoverable issues. @jdsmith3000 for comment on this piece at least.

Fascinating! I can confirm that Edge has a MediaError.prototype.msExtendedCode. Oddly enough it's a number, e.g. for a 404 resource it's -1072889839. Weird :)

[foolip]

I've changed the title to broaden the scope. In https://github.com/whatwg/html/pull/509#issuecomment-174967812 and https://github.com/whatwg/html/pull/509#issuecomment-177914176 I complain about how these errors work, and this issue is about making sense of it.

• After reaching networkState HAVE_CURRENT_DATA or greater, are there actually any fatal errors in implementations, errors that make it impossible for the media element to ever play again without starting over with load()? As written, there's nothing to prevent seeking or playing after the errors are reported.
• For non-fatal network or decode errors, how should they be reported? There could be any number of such errors during the lifetime of a media element, but we have only a single error attribute which is only assigned once.
• Can network and decode errors change duration, buffered and seekable? The spec doesn't say so, but it would make sense for a file found to only contain garbage after time x to clamp the timeline to x.

My rough idea is to limit the error attribute to truly fatal errors, so that attempting to do anything with the media element while the error is a no-op, or could throw/reject with InvalidStateError. I suspect that only MEDIA_ERR_SRC_NOT_SUPPORTED would be included in this group. For non-fatal errors, we could add a way to report them with better detail, at least for same-origin data.

[dmlap]

Another wrinkle to consider: the behavior of media with multiple video or audio and corrupt or inaccessible non-main tracks. Falling back to the main track seems like reasonable behavior but it would definitely be helpful to indicate the error to application code so the change can be reflected to users.

[foolip]

Do you mean for example audio tracks where they seems OK from the metadata, but where during playback or when trying to switch to that track one finds that it can't be decoded? Any idea what browsers would do in that situation today?

[dmlap]

Yep, that's it. I don't have an example of in-band alternate video/audio tracks to test with but I do know that Safari will trigger MEDIA_ERR_DECODE if segmented WebVTT subtitles are specified in an m3u8 but they turn out to be inaccessible. When correctly configured, those sort of subtitles would be exposed as text tracks. That solution works but it seems like overkill to me.

[foolip]

For audio and text tracks, multiple tracks can be enabled, so I guess in those cases simply disabling the track again would work. For a video track I'm not sure, it seems like just keeping the previous track would be best. A fatal error in any of these cases seems like overkill. @jernoble, can you comment on any of this for Safari?

[jernoble]

In the case of out-of-band text tracks, we could probably fail more gracefully. But for in-band audio, video, and text tracks, we may not have enough information from the media engine to successfully disable the offending tracks and continue playback.

[benwiley4000]

I'm not sure if this is the correct thread, but have you considered revisiting the ABORT_ERROR fired when mediaElement.play() is called then mediaElement.pause() is called after, before the media had had a chance to load and play? The play() request was interrupted by a call to pause()

Also, when skipping quickly through different songs, the same issue happens: Uncaught (in promise) DOMException: The play() request was interrupted by a new load request.

There are plenty of media playback applications on the web that do not catch() mediaElement.play promises (after all, it's a recently added API), and these applications nonetheless "work" just fine. However the appearance of a big red uncaught promise error in the console gives the impression that something is wrong and it must be fixed. That could be particularly alarming if a developer is consuming a third party library which emits this error message.

I'm not sure what the best solution might be. It's certainly not safe to assume a media element is playing just because audio.play() was called. However many if not most applications which display "playing" UI when a play is initiated will make a corresponding UI changes when a pause happens. In cases like this, the promise handling can feel like ceremonious boilerplate.