search

w3c / media-capabilities

Media Capabilities API
https://w3c.github.io/media-capabilities/
Other
78 stars 32 forks source link

multiple stream decoding #113

Open jpiesing opened 5 years ago

jpiesing commented 5 years ago

I'm sure I've seen this mentioned somewhere but couldn't find it & would prefer it didn't get lost.

A number of media devices support decoding more than one stream at one time. Unfortunately in some (many?) cases it's not as simple as it appears.

  1. Some devices may have multiple decoders whose capabilities are identical
  2. Some devices may be able to decode one UHD stream and one HD stream at any time - i.e. the second decoder simply doesn't support UHD ever but can be used at any time regardless of what the first decoder is doing. This may be due to the interface to the second decoder.
  3. Some devices may have dynamic capabilities. e.g. they can decode one UHD or two HDs at the same time but not two UHDs. This may be due to the amount of RAM available or the bandwidth into the RAM.

Unfortunately in the real world, I get the impression that the non-identical capabilities are more common - at least in devices that support UHD. The cost of dual UHD decoding (RAM size, bandwidth) does not seem to be justified by the (limited) benefits so device makers don't include it.

Using multiple media decoders can improve client-side advertising use-cases and it would be good if this could be included in the media capabilities.

The codec switching API doesn't really help if it would take a device between 0.5s and 1s to switch codec. A good user experience really needs either 1) a second media decoder all prepared and ready to go at the right moment or 2) chip providers to extensively re-write their device drivers - and their customers to do a lot of testing to find the regressions this introduces.

mounirlamouri commented 5 years ago

CC @jernoble @jyavenard @chcunningham

I believe this question was raised in the past and at least as far as I know, the answer hasn't changed: handling multiple stream is out of scope because it would be really hard to give reliable answer for N streams and it's a fairly small use cases. Maybe the folks in CC have a different opinion now that they have also implemented the API.

jpiesing commented 5 years ago

handling multiple stream is out of scope because it would be really hard to give reliable answer for N streams and it's a fairly small use cases

I agree it's hard but not that "it's a fairly small use cases". Multi-stream decoding really helps with client-side ad insertion when codec switching of a single decoder may take between 0.5s and (in extreme cases) 2s.

chcunningham commented 5 years ago

Re: codec switching, the fancy new approach is to call sourceBuffer.changeType(), re-using the existing MSE and media element. Documenting where these switches is supported falls to the transition() API (not spec'ed, name may change, active discussion in #102). Its my intention to specify that "supported" codec transitions must be seamless (imperceptible to the user). If it takes .5s, thats way too long and would be considered unsupported.

thinkski commented 1 year ago

Now that Chrome supports HEVC decoding, but only when there is hardware support for it, I think this will be a more pressing problem and should be reconsidered. I imagine HEVC hardware decoding limitations are more deterministic (and lower) than AVC decoding limitations and should be easier to describe. For instance, the configuration to be evaluated could be an array of (width, height, framerate, codec) tuples and should return a boolean on whether supported or not.