HDR support

[ZmGorynych]

Video Color Space Interface currently supports SD (BT.601) and HD (BT.709), but not PQ and HLG. Syntax-wise, it seems that the values are taken from x264 colorimetry command line parameters. I would suggest using values defined in the CICP spec (see H.273 sec 8.1-8.3). Alternatively, the list of x265 colorimetry command-line parameters will contain the necessary human-readable values.

Also, how can we support backwards-compatible HDR dynamic metadata such as DolbyVision profiles 8.x and Samsung HDR10+? In both cases the underlying stream is a perfectly valid HDR10 stream while additional metadata from either one is used to enhance the experience. This way a single stream can be decoded as HDR10, DolbyVision, and HDR10+ by different decoders.

[sandersdan]

The values are aligned with the H.273 spec, but HDR and WCG are/were out of scope for WebCodecs v1.

The primary blocker at this time is lack of HDR <canvas> and supporting web platform features like ImageBitmap.

[chcunningham]

Re: Canvas HDR, the Color-on-the-web CG is tracking that deliverable with a strawman here.

Lets use this issue to track the WC spec work. Off the top of my head, we want:

1. Additional color space values
2. Higher bit depth video pixel formats

@sandersdan, what else?

[sandersdan]

Those are the most important; we'll also need to update the relevant codec registrations, and verify that the codec and format detection APIs are sufficient.

[sandersdan]

Proposal

This covers the most basic changes required to support HDR and WCG video in WebCodecs. This is becoming feasible to implement in Chrome as the infrastructure for <canvas> is becoming available (c.f. https://groups.google.com/a/chromium.org/g/blink-dev/c/S609KGTkSSk/m/1EW4CWn3BgAJ).

Pixel Formats

• I420P10, I420P12, I420AP10, I420AP12, I422P10, I422P12, I422A, I422AP10, I422AP12, I444P10, I444P12, I444A, I444AP10, I444AP12
  • These are all the YUV formats supported by libavif that are not currently specified.
• P010, P012, P016
  • These are the NV12-like formats likely to be supported by hardware decoders.
• RGBAF16
  • Matches the high-bit-depth <canvas> format (RGBA 16-bit float).

The above list excludes:

• XR30, XB30
  • These formats pack 10-bit RGB samples into four bytes. I didn't see these being used by any codecs, but they are sometimes used as capture formats.

Alternatively

If we add a new depth field to VideoColorSpace, the 16-bit fixed point formats could all be merged. They are bit-compatible in the sense that interpreting one using an incorrect bit depth is difficult to distinguish, so this may be an overall improvement in compatibility.

• I420P16, I422P16, I444P16, I420AP16, I422AP16, I444AP16
• P016
• RGBAF16

Color Primaries

• bt2020 (also used by BT.2100)
• smpte432 (AKA display-p3, used by <canvas>)

Transfer Functions

• linear: Used by <canvas> srgb-linear.
• smpte2084 (AKA bt2100-pq)
• arib-std-b67 (AKA bt2100-hlg)

The above list excludes:

• bt2020-10, bt2020-12: these are the same as the existing smpte170m

Matrix Coefficients

• bt2020-ncl

The above list excludes:

• ycocg, ictcp: I am unsure if these are being used in practice.

HDR Metadata

I am not currently proposing any HDR metadata. Typically HDR metadata includes per-frame light levels and the mastering color volume. I am uncertain about the future-compatibility of current ways of expressing this metadata, and have been told that it’s not a trusted input to tonemapping anyway.

[chrisn]

Thanks @sandersdan. I suggest seeking input from Color on the Web CG, which has a good number of HDR experts, and developed the HDR <canvas> approach. https://github.com/w3c/ColorWeb-CG/issues/7

[dalecurtis]

Do we even need the 16bit formats if we have a depth field? Wouldn't they just be I420, I420A w/ depth=16?

[sandersdan]

They could be, but I think that is a compatibility hazard. The 16-bit formats are all binary-compatible but the 8-bit formats are not, and existing code would not be expecting these new formats under the existing names.

[dalecurtis]

I'm not sure that's a practical compatibility issue since you can't decode HDR w/ WebCodecs today, you'd have to go through a video element to VideoFrame which I suspect has fairly limited usage (if not nonexistent, especially with HDR content)

[ccameron-chromium]

FYI, some early user of the Canvas HDR spec wanted to use it for editing PQ video, and, to match rendering between a canvas and a video, they needed to specify HDR metadata. We've added a CanvasSmpteSt2086Metadata option when configuring HDR to allow further experimentation here. It strikes me that we will probably want to query this information from a video stream (and it may be that WebCodecs and Canvas should want to use the same type here).

[sandersdan]

Thanks Chris, I think we can use CanvasSmpteSt2086Metadata directly.

[dalecurtis]

Editor's call: Prefer explicitly including depth on pixel format since its what all other platforms do and since there's not too many of them. Open to strong reasons for depth.