Inconsistent SVC metadata between WebCodecs and WebRTC Encoded Transform API

[aboba]

Currently the EncodedChunkMetadata dictionary includes SvcOutputMetadata, which only supports temporal scalability (temporalLayerId).

However, within the Encoded-Transform API, RTCEncodedFrameMetadata includes its own (incompatible) SVC metadata.

Here is a comparison of the two approaches: https://docs.google.com/presentation/d/1lFAUSvApbBYfBNJH_xcRW0YjD0aF5T1ZqjyyDJMesJw/edit#slide=id.g1a4ac56601a_3_0

Can we define the complete set of SVC metadata in WebCodecs and then have it inherited by Encoded Transform?

Related: https://github.com/w3c/webrtc-encoded-transform/issues/170

[dalecurtis]

+1 for consistency, but defer to @djuffin

[Djuffin]

It's unusual to add fields unless there is at least one implementation that makes use of them. But in this case compatibility with Encoded Transforms seems like a pretty good reason to add these fields.

I think we can even rename SvcOutputMetadata into something more relevant to video frames, Output doesn't seem to convey a lot of meaning outside of webcodecs' context.

[aboba]

The Encoded Transform metadata has now been updated. However, it still is not completely compatible with the WebCodecs extended SVC metadata proposal.

[dalecurtis]

ed note: @aboba to send PR and @Djuffin to review.

[aboba]

A summary of where we are now.

Here is the RTCEncodedVideoFrameMetadata dictionary:

Here is what is in WebCodecs Section 6.7:

Here is what was in the extended WebCodecs SVC metadata proposal:

[aboba]

@pthatcherg @alvestrand Can you take a look?

[chrisn]

Minutes from 11 Apr 2023 Media WG meeting: https://www.w3.org/2023/04/11-mediawg-minutes.html#t03