How can we determine in the file is RGB or YCbcr without going into the data

[rbouqueau]

Several 'shalls' in the specification refer to using the same "coding format", 'chroma sampling format" or "offset".

The issue is that in "7.3.6.4 Colour information property", the MIAF specification says:

If a coded image has no associated colour property, the default property is defined as having colour_type equal to 'nclx' with properties as follows: – For YCbCr encoding, sYCC should be assumed as indicated by colour_primaries equal to 1, transfer_characteristics equal to 13, [...] – For RGB encoding, [...]

Thus the specification provides default values for both options without specifying how to know which applies without going into the data.

This dual existence of implicit "YCbCr or RGB" also creates some confusion on conversions (and may have other yet-unspotted side effects). For example HEIF (6.6.1) says "This document does not require the colour space of the input images for grid or overlay derived image items to be identical. Derived specifications could constrain the colour space to avoid the need for colour space conversions in the derivation process." but MIAF (7.3.6.4) says "If a derived image has no colour information, then the default colour information is defined by the derivation" and then (7.3.11.4) "All input items to a grid shall have the same explicit or default colour information.". A consequence of the current text is that we could have a YCbCr (e.g. HEVC) master image with a RGB (e.g. JPG) derivation that implies a reader conversion but no conformance message. I am not sure this is what we intend.

The issue blocks the implementation of some conformance rules.

[dwsinger]

The problem, as I understand it, is that if we're missing file-format level information (and we are, as we're trying to establish defaults in the absence of info), then one is digging in a codec-specific way into the codec info, e.g. looking for VUI information in AVC/HEVC, to find the matrix coefficients. And at that point, why are we inventing defaults at the file format level when we've found low-level truth in the bitstream? If we have found matrix coeffs, we've also found colour primaries and transfer function, no?

[denoualf]

@dwsinger, Then, would an indication that the defaults apply only when digging into codec-specific way does not provide any hints help?

[rbouqueau]

I am not sure the solution should rely in the text but in ISOBMFF.

If we step back and think about what we are trying to achieve, a robust implementation should be able to deal with MIAF data without knowing deeply about the data it handles. This is one of the big strength of ISOBMFF (you can fragment, basically encrypt, or DASH content your implementation don't understand). My 2 cents: to keep that we need to expose the RGB/YCbCr information at the ISOBMFF level (either with a new box, a new version of the 'colr' box or whatever better solution experts may come with).

[cconcolato]

@dwsinger what happens if what you find in the bitstream is CP/TC/MC=2? How would you know if it's YUV or RGB?

[dwsinger]

@dwsinger what happens if what you find in the bitstream is CP/TC/MC=2? How would you know if it's YUV or RGB?

you don't; any more than if VUI is missing (in AVC). at that point, you guess. I think the usual guess is that it's "like JPEG", but we're into speculation...

[cconcolato]

MIAF should be clearer and fixed by:

• either mandating a colr box with either an ICC profile; or an nclx type and CICP values other than 2.
• or clearly saying that if the colr box is omitted, the bitstreams assumed to be CP=1/TC=13/MC=1

I also wonder if a fix is not needed in HEIF or ISOBMFF or a reminder in the renderer model in MIAF to clearly indicate that colr overrides the image bitstream information if any. Currently ISOBMFF says:

If colour information is supplied in both this box, and also in the video bitstream, this box takes precedence, and over-rides the information in the bitstream.

Note the use of "video" bitstream.

[cconcolato]

On this topic, you might be interested by this related thread on AV1/AVIF: https://github.com/AOMediaCodec/av1-avif/issues/84

[cconcolato]

Based on the discussion linked above and some others, here is the summary of my understanding (covering a bit more than this issue):

[cconcolato]

The recommendation to resolve the ambiguity would be to default MC=1 when no meaningful information is in the nclx or bitstream. @dwsinger can you confirm this matches what you expect (for HEIF and others)? @aklemets I think it matches what you reported in the AV1 thread. Can you confirm?

[aklemets]

Yes, we default to MC=1 when there is no meaningful information in nclx or in the bitstream.

However, I don't have any special casing for MC=2. So, if nclx or the bitstream specifies MC=2, I am not really sure what happens. I think the HEVC decoder will use MC=1 (BT.709) when the matrix coefficients are Unknown/Unspecified, but I will have to double-check if that assumption is correct. Is the matrix coefficients being explicitly signaled as 2 a realistic scenario that we have to worry about?

[joedrago]

Speaking from the tools side of things:

avifenc currently doesn't make any assumptions about what the enduser requesting the encoding wants to use for those values, so unless a value is explicitly requested, they might signal 2/2/2 for CICP. I can obviously change the defaults, but they could still explicitly set 2/2/2 unless I blocked it (errored out).

(As a note, I also fall back to MC==1 when the values are unspecified in the bitstream & nclx, so if everyone else also does, I'd rather just formalize that default/fallback than create more error conditions on encode and decode.)

[ichiro-ando-ai]

My understanding is below. AVC/HEVC 4:2:0 and 4:2:2 profiles is only YCbCr coding. JPEG is only YCbCr coding. We can determine that it is YCbCr coding by MIAF profile, at least MIAF AVC Basic/MIAF HEVC Basic/MIAF HEVC Anvanced. Only the case of MIAF HEVC Extended profile, it may not sure that RGB or YCbCr even if analyzing HEVC bitstream. Because it may use HEVC 4:4:4 profile that can use RGB and YCbCr. Is it right?

[dwsinger]

I think you are right, and this is a problem. We tell people what the default is, so that they don't have to dig deep into the bitstream to find the color information. But the default determination is based on an if statement — if(is YcrCb) then (x) elif(is RGB) then (y). But knowing if it's YCrCb means exactly what we were trying to avoid — digging into the bitstream.

(The If...elif is also missing a final "else").

[ichiro-ando-ai]

If the MIAF profile is MIAF AVC Basic, MIAF HEVC Basic, or MIAF Advanced, only YCbCr coding is available. So, for these profiles, the default is sYCC only. If the MIAF profile is a MIAF HEVC extension, it may not sure that RGB or YCbCr even if analyzing HEVC bitstream. Therefore, for MIAF HEVC Extended, the default must not be defined and we must describe color property.

From the above, my idea is following, to avoid digging into the bitstream: ・Default color property is the only sYCC ・In case of MIAF HEVC Extended, describing color property must be required

How is it?

[Naoya-Katoh]

I agree and support Ando-san's idea; ・Default color property is the only sYCC (for Basic and Advanced) ・In case of MIAF HEVC Extended, describing color property is required

[rbouqueau]

I understand that trying to infer the color encoding is appealing. But I think it would make the system weaker than we can make it.

To cite myself from this thread:

a robust implementation should be able to deal with MIAF data without knowing deeply about the data it handles. This is one of the big strength of ISOBMFF (you can fragment, basically encrypt, or DASH content your implementation don't understand).

[cconcolato]

I think the HEVC decoder will use MC=1 (BT.709) when the matrix coefficients are Unknown/Unspecified, but I will have to double-check if that assumption is correct.

@aklemets Have you been able to verify if your assumption was correct?

[ichiro-ando-ai]

I checked HEVC specification (2019-11).

When the colour_primaries syntax element is not present, the value of colour_primaries is inferred to be equal to 2 (the chromaticity is unspecified or is determined by the application).

When the transfer_characteristics syntax element is not present, the value of transfer_characteristics is inferred to be equal to 2 (the transfer characteristics are unspecified or are determined by the application).

When the matrix_coeffs syntax element is not present, the value of matrix_coeffs is inferred to be equal to 2 (unspecified).

[cconcolato]

@ichiro-ando-ai Thanks for the confirmation. I understand this as meaning that the application can do whatever it wants. Are you suggesting that in MIAF we change the default 1/13/?/1 to 2/2/2/1 ?

[ichiro-ando-ai]

No. I would like to just share the information that we may not be able to determine the color format used in the coded image, even if we dig deep into the bitstream. Therefore we should have some definitions. In current MIAF specification, two color encoding types (YCbCr encoding and RGB encoding) are considered. I think this is the problem. It is better that we have just one default. We generally use YCbCr encoding for 4:2:0 and 4:2:2. Therefore I think that it is better that YCbCr encoding is default at least in the case of 4:2:0 and 4:2:2 profiles. In addition, sYCC is the default color format for YCbCr encoding as defined by the current MIAF specification. How is it?

[Naoya-Katoh]

I agree that RGB encodings are rarely used in MPEG/JPEG, and therefore, we can consider "it is better that YCbCr encoding is default".

NK

[cconcolato]

I agree with Romain, we should try to avoid being codec specific. MIAF is applicable to other codecs than HEVC. I would agree to having a default only for YCbCr. I'm even wondering if we shouldn't require the 'colr' box to be always present (no default).

sYCC is the default color format for YCbCr encoding as defined by the current MIAF specification

There is no consensus yet on that part.

[dwsinger]

I think https://github.com/MPEGGroup/MIAF/issues/2#issuecomment-643576312 may be on the right lines.

EITHER the profile restricts to YCC (matrix), whereupon we can say what the default color is (independent of codec) OR the profile allows RGB, whereupon the color information box is mandatory.

Occasionally a default is going to be wrong; we already advise it SHOULD be present, so it's probably only going to be left out when the file writer didn't know and couldn't guess.

[cconcolato]

The MIAF group discussed this issue and resolved to say in the upcoming amendment text that:

• the color property (whether defaulted or explicit) always takes precedence over the bitstream value
• when it's defaulted it means 1/13/5 (or 6)/1. RGB requires an explicit color box (not a possible default anymore, resolving this issue).
• we still have to discuss what 2/2/2 means if used in the colour property. I'm closing this issue. Reopen or open a new one if needed.