search

w3c / media-capabilities

Media Capabilities API
https://w3c.github.io/media-capabilities/
Other
78 stars 33 forks source link

Discuss how to reconcile colorGamut & transferFunction with ISO 23001-8:2016 #152

Open vi-dot-cpp opened 4 years ago

vi-dot-cpp commented 4 years ago

Hi everyone. chcunningham@ and I would like to (re)open up discussion with respect to how MediaCapabilities.decodingInfo() should handle discrepancies between colorGamut & transferFunction input and and the color information implicit in mime types like VP9 and AV1.

Take the following example:

navigator.mediaCapabilities.decodingInfo({
    type: 'file',
    video: {
      contentType: 'video/webm; codecs="vp09.02.10.10.01.09.16.09.01"',
      ...
      colorGamut: "srgb",
      transferFunction: "srgb",
    },
  })

The mime type implies BT2020 primaries and SMPTEST2084 EOTF, but colorGamut and transferFunction are specified as BT709. How should UA validate this input? Thanks everyone -- best wishes to everyone.

tidoust commented 4 years ago

Discussed during the Media WG call on 12 May 2020.

Resolution: reject the promise when there are conflicting inputs, details can be discussed back in the issue (see minutes)

vi-dot-cpp commented 4 years ago

Thanks tidoust@. jernoble@ rightly pointed out that this reconciliation between properties and mime types should occur broadly. For HDR properties though unique challenges may exist due to how colorGamut and transferFunction are defined.

For instance, chcunningham@ noted that mime types can specify an obscure transfer function not defined by the MediaCapabilities TransferFunction enum.

chcunningham commented 4 years ago

@jernoble, thoughts on the last comment from Vi?

jernoble commented 4 years ago

I don't think that it changes the outcome. If a MIME type specifies an obscure transfer function, the UA must be able to say "supports: false/true" using only that MIME type already, as the transferFunction property is optional. And if the client specifies a non-matching transferFunction, that should have the same outcome:

So lets look at the forward compatibility story:

If we add new values to the transferFunction enum, how will clients know what values are supported in a given UA? Clients can try one of the new transferFunction enum values, and they should get a TypeError exception calling into decodingInfo(). So clients have a path to discovering that UAs don't "understand" a given transferFunction value.

The remaining question is how the client knows whether it must provide transferFunction, colorGamut, etc, or whether that information is derivable from the MIME type. Let me throw out a straw man.

We already have MediaCapabilitiesDecodingInfo.configuration which contains the configuration as understood by the UA, to allow clients to detect UA support for new fields in MediaConfiguration. One way to answer this question would be for UAs to "fill in" the configuration object with the information provided by the MIME type. E.g., if the client passed in {video: { contentType: 'video/webm; codecs="vp09.02.10.10.01.09.16.09.01"'; width: 1080; height: 720; bitrate: 10000; frameRate: 24 } }, the resulting configuration object would also include {...; hdrMetadataType: X; colorGamut: Y; transferFunction: Z } as derived from the MIME by the UA.

If we had this kind of API, clients which have a MIME-type that specifies HDR properties can discover that they don't need to specify the HDR properties separately. In supporting UAs, the HDR properties will be present in the resulting configuration. In non-supporting UAs, or with MIMEs that don't specify HDR properties, they know that they must provide the HDR properties separately (because those values are missing from the resulting configuration, and they have a way of determining whether the values they've derived are understood by the UA (via the TypeError above).

mounirlamouri commented 4 years ago

Removing the agenda label as I forgot to to that after the previous meeting.

dalecurtis commented 11 months ago

While this seems resolved via the WG and implemented by Safari, the spec text hasn't yet been updated. Chrome will implement reporting mismatched color spaces as unsupported like Safari does.

markafoltz commented 4 weeks ago

The PR updates the spec to reflect the implemented behavior in https://github.com/w3c/media-capabilities/issues/152#issuecomment-1786265925.

FWIW I think it's more logical to check this in the steps to validate a video configuration. However, an invalid configuration rejects the Promise returned by decodingInfo with a TypeError, and that's not what was implemented.