search

w3c / me-media-integration-guidelines

Media Integration Guidelines
https://w3c.github.io/me-media-integration-guidelines/
Other
2 stars 2 forks source link

Claiming a hardware decoder with load() #2

Open JohnRiv opened 4 years ago

JohnRiv commented 4 years ago

Migrated from https://github.com/w3c/webmediaporting/issues/30:

Similar to #1, when load() is called on a media element that needs to use a hardware decoder, at which step in which algorithm does the UA attempt to “claim” that hardware decoder?

What happens if the hardware decoder is being used by 1) something outside the HTML5 UA, or 2) another HTML5 media element (in the same page) that is playing, or 3) another HTML5 media element (in the same page) that is stopped, or 4) another HTML5 media element in a page that’s loaded in another tab in the browser?

If the HTML5 UA is unable to “claim” the hardware decoder, how would the method call fail and what state would the media element be left in?

Would the method call fail like play(), stall in one of the earlier algorithms, (e.g. not get beyond HAVE_METADATA even when all the data would actually be available) or succeed?

If a call to load() would not fail, what would be visual representation of a video element without a hardware video decoder to decode the first frame of the video?

fghilardi commented 4 years ago

Chrome's Media Stack (c/o @sandersdan from https://github.com/w3c/webmediaporting/issues/30#issuecomment-610056448 and https://github.com/w3c/webmediaporting/issues/30#issuecomment-610628219)

We do not, in general, have information about platform (hardware OR software) codec availability/constraints. We merely try to obtain them, and this can fail. Failure cases are typically decode errors, and they may be recoverable via load().

Latency/time of acquisition: we do not acquire codecs at a defined time, and it will vary by platform and video element configuration. MSE may acquire a codec once an initialization segment is available, but may be deferred. New codecs may be acquired at any time to handle codec changes, fallback, or to implement resource optimizations.

Loading to HAVE_METADATA requires acquisition of a codec to decode the poster frame. play() from a lower readyState will force acquisition of a codec.

The [HTML] spec as written assumes infinite resources, and web pages do not typically implement any recovery mechanisms to handle cases where that turns out not to be true. Our workarounds have to be careful to maintain the illusion (basically always eventually emit the expected events) or there will be breakage.

Note that we are not always successful at maintaining the illusion. For example, we can't seek to the previous keyframe in a realtime stream, and therefore can't use suspend/resume transparently in that case.

sandersdan commented 4 years ago

To clarify, Chrome's answers are the same as I wrote in #1. Depending on the preload attribute and potential interventions used, load() may not require that any frames are decoded at all.