guidelines on integration of web media APIs with hardware video & audio decoders

[jpiesing]

A number of issues that seem to cause inter-operability pain with web media apps relate to the integration of the web media APIs with hardware video & audio decoders.

We could address them in WAVE but addressing them in somewhere else might be better. I think it's unrealistic to try & get the WHATWG HTML spec changed but perhaps a W3C Note with some guidelines on the subject might be both useful to the industry and achievable.

Below is a list of issues which I've encountered which might be suitable to cover with some W3C guidelines.

Where there are multiple different answers in real world media devices, I'd hope it's possible to at least document them & perhaps recommend one for new implementations.

---

• When play() is called on a media element that needs to use a hardware decoder, at which step in which algorithm does the UA attempt to “claim” that hardware decoder? What happens if the hardware decoder is being used by 1) something outside the HTML5 UA or 2) another HTML5 media element (in the same page) that is playing or 3) another HTML5 media element (in the same page) that is stopped or 4) another HTML5 media element in a page that’s loaded in another tab in the browser? If the HTML5 UA is unable to “claim” the hardware decoder, how would the method call fail and what state would the media element be left in?
• Same as the previous question but for load()? Would the method call fail like play(), stall in one of the earlier algorithms, (e.g. not get beyond HAVE_METADATA even when all the data would actually be available) or succeed? If a call to load() would not fail, what would be visual representation of a video element without a hardware video decoder to decode the first frame of the video?
• Is it possible to pre-load one video element while another is playing without disturbing the playback or would calling load automatically “claim” the hardware decoder and interrupt the pre-existing playback?
• If it takes 250ms-500ms to initialise a hardware video/audio decoder, where in the various algorithms would this delay happen? With MSE, would this be expected to happen in reaction to the initialisation segment being written to the SourceBuffer(s) or could it only happen when the first media data is written?
• What does setting audioTrack.enabled do if only one audio stream can be decoded at one time & another audioTrack is already being decoded? Does it fail or “take” the decoder from the track that’s playing?
• If the hardware media pipeline has a 80-150-250-750ms latency/delay between encoded samples entering the beginning and decoded data leaving the output & being visible/audible, what time is measured by the MediaElement currentTime property? When would the ended event be fired – when the data for the last frame/sample enters the pipeline or perhaps 750ms later when it leaves the media pipeline? Does it make any difference if video and graphics take separate paths through some of this pipeline and are then combined to pass through the remainder of it. (TVs typically have a low latency mode for gaming where the latency could be <40ms or even <20ms. Media consumption is however typically done with the media pipeline configured differently and having a higher latency. 750ms is however an extremely pessimistic example).
• If a video element is used to play content to the natural end, do hardware decoders remain "claimed" by that video element? When are they "released"?
• Are any of the above different if the source of the media element is the URL of a media resource on the network rather than MSE?
• If an app wants to seek to somewhere in a media resource other than the start, how far can it get without hardware decoders being available? Does it fail, succeed or stall at that point? (See step 12 of the seeking algorithm which refers to decoding (my emphasis) - "Wait until the user agent has established whether or not the media data for the new playback position is available, and, if it is, until it has decoded enough data to play back that position. ")

"hardware decoder", "claim" and "released" are shorthand and not intended to be precise or restrictive. Some hardware platforms may have a single video decoder block that can decode more than one stream at one time under some circumstances.

[jpiesing]

@johnluther @JohnRiv @tidoust @chrisn @haudiobe

[tidoust]

Documenting usual behavior in real world media devices and listing recommendations for developers seems a good thing. If a W3C Note seems useful, the Media and Entertainment Interest Group could perhaps host the effort, provided IG participants support it and are willing to work on the document. Should I raise this within the IG right away or should we wait until WAVE gets a chance to discuss it?

For what it's worth, the IG charter already mentions the possible publication of "Primer or Best Practice documents to support web developers when designing applications", so no need to recharter the group for it to publish a note as suggested.

[jpiesing]

Documenting usual behavior in real world media devices and listing recommendations for developers seems a good thing.

What I had in mind for recommendations were ones aimed at those organisations porting / integrating an existing HTML5 UA onto a media device hardware + operating system - not at developers. I'm sure there are issues for developers too - feature detecting how the integration has been done where this makes a difference.

If a W3C Note seems useful, the Media and Entertainment Interest Group could perhaps host the effort, provided IG participants support it and are willing to work on the document. Should I raise this within the IG right away or should we wait until WAVE gets a chance to discuss it?

I'd like to have one round of discussion in WAVE first - at least that way I have the possibility of participating.

For what it's worth, the IG charter already mentions the possible publication of "Primer or Best Practice documents to support web developers when designing applications", so no need to recharter the group for it to publish a note as suggested.

That could be seen as not including documents aimed at browser integrators :(

[chrisn]

I agree this would be useful, and welcome. I wouldn't rule out proposing clarifications to HTML though, and perhaps creating a guidelines document is a good step towards that.

That could be seen as not including documents aimed at browser integrators :(

This is under "Other non-normative documents" as a "such as", so it's not an exclusive list. I agree with @tidoust that this could be done under the current charter.

It's possible someone at my organisation would want to contribute to these guidelines.

[tidoust]

Ah, I had misunderstood the intended audience, indeed, and so my reference is misleading at best... I don't think the shift makes much difference though. The scope section does mention clients and devices (and thus by extension those who implement these clients, directly or indirectly).

[jpiesing]

Another possible subject for the list is the usage of hardware media decoders by the Web Audio API and how that interacts with media APIs.

• If a Media Element has the hardware decoder "claimed", what happens if there's an attempt to do something with the Web Audio API which also needs the hardware decoder? Which "wins" and what happens to the other?
• And the converse - if the Web Audio API has the hardware decoder "claimed", what happens if there's an attempt to do something with a Media Element that also needs the hardware decoder? Which "wins" and what happens to the other?

[jpiesing]

I've just updated the problem description to expand concerns around the load method and add concerns about seeking.

[sandersdan]

Here is how I see these question from the point of view of Chrome's media stack (primarily the video element) and a future that includes WebCodecs.

Current situation:

• The web does not have explicit resource management for media. This is a consistent problem and we employ a number of tricks and optimizations to work around it, which seems to be a qualified success.
• Chrome's primary intervention is to discard codecs for idle players, then resume playback by an internal seek-to-previous-keyframe when required. (We call this Suspend/Resume, it is unrelated to network suspend.)
• We may pause playing video to force it to become idle. A variety of heuristics may be used to select players for this.
• I would very much like to see explicit mechanisms added. For WebCodecs I expect that codec instances will be fallible, allowing us to take their resources at any time. A more sophisticated approach is likely required for new use cases, such as background transcoding. (Eg. some way to inform apps when resources become available.)

Resource availability:

• Hardware codecs are not our only constrained resource. CPU memory, CPU cycles, DMA bandwidth, GPU memory, and network bandwidth are all impacted.
• We do not, in general, have information about platform (hardware OR software) codec availability/constraints. We merely try to obtain them, and this can fail. Failure cases are typically decode errors, and they may be recoverable via load().
• Loading to HAVE_METADATA requires acquisition of a codec to decode the poster frame. play() from a lower readyState will force acquisition of a codec.
• On Android we may obtain codecs lazily and aggressively release codecs when idle. We release all codecs when switching tabs (or between apps) to make them available to other processes. We may fake loading progress when a poster image is available.

Specific questions from above:

• Preloading: we are able to decode at least two videos concurrently on all of our browser platforms.
• Latency/time of acquisition: we do not acquire codecs at a defined time, and it will vary by platform and video element configuration. MSE may acquire a codec once an initialization segment is available, but may be deferred. New codecs may be acquired at any time to handle codec changes, fallback, or to implement resource optimizations.
• currentTime: currentTime is the target time for rendered output. There is additional latency in the output path, but decoding latency is not a factor.
• seeking: seeking is deferred in our implementation during suspend, but a seek does not complete while in this state.

[chrisn]

Many thanks for sharing this, @sandersdan.

[jpiesing]

Hardware codecs are not our only constrained resource. CPU memory, CPU cycles, DMA bandwidth, GPU memory, and network bandwidth are all impacted. Preloading: we are able to decode at least two videos concurrently on all of our browser platforms.

Media devices may only be able to decode one video at the same time or may only be able to decode two with limitations. Many media devices would not have the memory capacity or bandwidth to decode two UHD streams. In some silicon, only one decoder is connected to the decryption block.

[sandersdan]

Media devices may only be able to decode one video at the same time or may only be able to decode two with limitations.

Indeed. I specified "browser platforms" above primarily because Chromecast is different.

[jpiesing]

Media devices may only be able to decode one video at the same time or may only be able to decode two with limitations.

Indeed. I specified "browser platforms" above primarily because Chromecast is different.

I wasn't sure of your definition of "browser platforms". Thanks for the clarification and for the original details.

[chrisn]

@sandersdan Do you think some of these issues could be addressed via clarifications in the HTML spec, or are there cross-vendor implementation differences which would make specifying expected behaviour not practical for these resource contention scenarios?

Some issues seem more straightforward to clarify in the spec, e.g., definition of currentTime.

I'd be interested to explore further the idea of adding explicit mechanisms.

[sandersdan]

The spec as written assumes infinite resources, and web pages do not typically implement any recovery mechanisms to handle cases where that turns out not to be true. Our workarounds have to be careful to maintain the illusion (basically always eventually emit the expected events) or there will be breakage.

Note that we are not always successful at maintaining the illusion. For example, we can't seek to the previous keyframe in a realtime stream, and therefore can't use suspend/resume transparently in that case.

We change our workarounds somewhat routinely, but there may be a core set of principles we can document. (Approximately 'always make forward progress eventually, fake events if you have to'.)

Comparatively recent changes like MediaSession (media controls in notifications) and the play() promise have helped, in that new players are more flexible when the browser asserts control over video elements. In the past a majority of player implementations would call play() in a loop if we didn't enter a playing state, but that's not a problem we have with modern players.

I am hopeful that with the right mechanisms now, there will come a future time where we can drop the illusion and just fail playback when resources are constrained. When using software decode or the Page Lifecycle API we are nearly there now.

[JohnRiv]

FYI, as you can see from the flurry of references above, I've migrated the issues listed here to specific, individual issues in the new me-media-integration-guidelines repo