Questions on Scalability Mode and Media Capabilities

[aboba]

From bc-lee:

I have questions about Scalability Mode and Media Capabilities:

Let's say that a user agent advertises through the Media Capabilities API that the support of the encoding configuration below is power efficient.

{
  type: 'webrtc',
  video: {
    contentType: 'video/H264',
    width: 640,
    height: 480,
    bitrate: 100000,
    framerate: 30,
    scalabilityMode: "L1T3"
  }
};

However, when the user agent actually creates the encoder with the given Scalability Mode, it finds that the encoder does not support the given Scalability Mode.

In such a scenario, what would be the proper ways to handle this situation? Should we not create the encoder? Ignore the scalabilityMode? Or do something else? What if the power-efficient encoder is the only encoder for that codec?

Also, if the user agent can only determine whether a specific scalability mode is supported or not after the first few frames are encoded, what would be the proper ways to handle this situation? Should we not advertise the scalability mode? Or do something else?

[aboba]

"However, when the user agent actually creates the encoder with the given Scalability Mode, it finds that the encoder does not support the given Scalability Mode."

[BA] WebRTC does not directly enable creation of encoders or decoders, nor does it provide direct control over hardware acceleration. Media Capabilities API is by nature "optimistic". That is, if it indicates that an encoding configuration is supported, or that it is "powerEfficient", this is not a guarantee. If hardware resource are unavailable, WebRTC will failover to software which might not be "powerEfficient" and might not support higher profiles. Currently, applications don't provide an indication of software failover.

SVC modes are typically more widely supported in software than hardware, so that it is unlikely that a scalabilityMode would only be supported in hardware but not software.

WebCodecs does enable direct construction of encoders, but capability discovery is handled via isConfigSupported API, not Media Capabilities.

In WebCodecs, if isConfigSupported indicates that a configuration is supported, then encoder construction should not fail.

[aboba]

Discussed at the Tuesday, November 14 MEDIA WG meeting (minutes).