Proposal: require auxiliary image items and sequences set color_range to 1 (i.e., full range)

[wantehchang]

I'd like to propose that we add the following constraint to auxiliary image items and sequences in Section 4 of the AVIF specification:

• The color_range field in the Sequence Header OBU shall be set to 1.

AV1's color_range field is equivalent to the VideoFullRangeFlag in ITU-T H.273. A color_range value of 1 means full range.

Auxiliary image items and sequences (alpha or depth) do not need to be limited range. So setting their color_range field to 0 introduces distortion unnecessarily and incurs range adjustment overhead.

Optional: the AVIF specification could also recommend that auxiliary image items and sequences set the color_primaries, transfer_characteristics, and matrix_coefficients fields to the UNSPECIFIED values, or just set the color_description_present_flag field to 0.

[leo-barnes]

Do we also want to constrain them to use a linear transfer function? Or are there any use cases where it actually makes sense to have auxiliary images use some other transfer function?

[wantehchang]

Hi Leo,

Do we also want to constrain them to use a linear transfer function? Or are there any use cases where it actually makes sense to have auxiliary images use some other transfer function?

AVIF has only two kinds of auxiliary images: alpha and depth. I believe the transfer characteristics function does not apply to alpha and depth. In this issue I also propose the following:

Optional: the AVIF specification could also recommend that auxiliary image items and sequences set the color_primaries, transfer_characteristics, and matrix_coefficients fields to the UNSPECIFIED values, or just set the color_description_present_flag field to 0.

Wan-Teh

[leo-barnes]

AVIF has only two kinds of auxiliary images: alpha and depth. I believe the transfer characteristics function does not apply to alpha and depth.

Do you explicitly state that in the spec? I think MIAF added that alpha should have linear transfer function specified (since unspecified may get us into some default that is most likely not linear). We were at least discussing adding it. I'm assuming that AVIF inherits that unless you explicitly state that it shouldn't.

[kodawah]

Setting color_range = 1, and transfer = linear makes a lot of sense to me. We could recommend setting matrix = identity, and primaries = sRGB too, perhaps depending on use case

[wantehchang]

Hi Leo,

Thank you for your comment.

I think MIAF added that alpha should have linear transfer function specified (since unspecified may get us into some default that is most likely not linear). We were at least discussing adding it. I'm assuming that AVIF inherits that unless you explicitly state that it shouldn't.

I did not find this mentioned in MIAF (ISO/IEC 23000-22). MIAF does not contain the word "linear". I also inspected all occurrences of the word "alpha".

An AVIF decoder will use the decoded Y plane of an alpha auxiliary image as the alpha plane of the master image. The AVIF decoder will not perform YUV-to-RGB conversion of the decoded output of the alpha auxiliary image. Therefore, the color_primaries, transfer_characteristics, and matrix_coefficients fields in the sequence header OBU of the alpha auxiliary image are all ignored by the AVIF decoder. Since they are all ignored, we can save three bytes by setting the color_description_present_flag field in the sequence header OBU to 0. (See AV1 specification, Section 5.5.2. Color config syntax.)

[leo-barnes]

I did not find this mentioned in MIAF (ISO/IEC 23000-22). MIAF does not contain the word "linear". I also inspected all occurrences of the word "alpha".

We discussed it, but I'm not sure it got added.

An AVIF decoder will use the decoded Y plane of an alpha auxiliary image as the alpha plane of the master image. The AVIF decoder will not perform YUV-to-RGB conversion of the decoded output of the alpha auxiliary image. Therefore, the color_primaries, transfer_characteristics, and matrix_coefficients fields in the sequence header OBU of the alpha auxiliary image are all ignored by the AVIF decoder. Since they are all ignored, we can save three bytes by setting the color_description_present_flag field in the sequence header OBU to 0. (See AV1 specification, Section 5.5.2. Color config syntax.)

This may be a misunderstanding on my part. I didn't realize until right now that you're probably talking about what goes into the AV1 bitstream rather than in the container. If we don't specify a colr box for the auxiliary image in the MIAF container, we will most likely fall into the default discussed in #84. Or does that default only apply to the main image?

[wantehchang]

Hi Leo,

I didn't realize until right now that you're probably talking about what goes into the AV1 bitstream rather than in the container.

Yes, I am talking about what goes into the AV1 bitstream (specifically, the sequence header OBU) of an auxiliary image, rather than in the container.

If we don't specify a colr box for the auxiliary image in the MIAF container, we will most likely fall into the default discussed in #84. Or does that default only apply to the main image?

The default discussed in #84 only applies to the main image.

Note: A colr box can be associated with any image in an AVIF file. However, as the two AVIF image dumps (libavif.log and libheif.log) in Chromium issue 1115483 comment 9 shows, neither libavif nor libheif generates a colr box for the alpha auxiliary image.

Also, if a colr box for an alpha auxiliary image is specified, it is ignored by libavif (the reference implementation of AVIF).

[cconcolato]

MIAF 7.3.5.2 says:

The following semantics apply to alpha planes:

• sample value 0 means that the co-located pixel in the master image is transparent (i.e. will not be displayed);
• the maximum sample value (e.g. 255 for 8-bit sample values) means that the co-located pixel in the master image is opaque, i.e. fully covers the background image;
• the sample values of the alpha plane divided by the maximum value (e.g. by 255 for 8-bit sample values) provides the multiplier to be used to obtain the intensity for the associated master image.

So it essentially says it's full range. It does not talk about depth maps though.

It also says:

Depth maps and alpha planes should be encoded in monochrome format (i.e. 4:0:0 chroma format) if possible; if they are encoded in colour, they shall be encoded in a colour format with a luma plane and chroma planes, e.g. as 4:2:0 YCbCr; in which case only the luma plane is relevant, and the chroma planes shall be ignored by the MIAF renderer.

[cconcolato]

We have an agreement to add text to the spec to indicate that

• for alpha planes and depth maps:
  • full range must be used.
• for alpha planes:
  • nclx colr boxes or CICP values in the bitstream if present shall be ignored

[wantehchang]

Cyril, I reviewed https://github.com/AOMediaCodec/av1-avif/pull/101. It looks good. Note the following:

You wrote:

We have an agreement to add text to the spec to indicate that

[...snipped...]

• for alpha planes:

  • nclx colr boxes or CICP values in the bitstream if present shall be ignored

PR #101 does not mention the CICP values in the sequence header OBU in the bitstream. We could recommend omitting the CICP values in the sequence header OBU by setting the color_description_present_flag field to 0.