HEIF: How do we indicate that coded item contains RGB+alpha?

[leo-barnes]

Future codecs may have native alpha support since it may give significant coding gains when you need to encode both RGB and alpha. How would this be indicated in HEIF? Having a four channel pixi (or two for grayscale + alpha) could potentially be used, but the spec would need to explain somewhere that this is how to interpret it.

[denoualf]

HEIF defines auxiliary items in section 6.4.5 (typically for alpha plane): would this be an option?

[leo-barnes]

HEIF defines auxiliary items in section 6.4.5 (typically for alpha plane): would this be an option?

The idea here is that future codecs may support native alpha. In other words rather than having one YUV item and one A item, you would have one YUVA item (or RGBA if using identity matrix). Having the codec understand alpha could give you much better compression ratio and would allow for easier incremental decoding.

So, assuming we have a future codec that supports YUVA/RGBA, how do we indicate that the coded item contains both YUV and alpha?

[leo-barnes]

Discussed during MPEG 143. Conclusion was that pixi is the right place to signal this, but the proposed structure was not flexible enough.

[bradh]

I think this is an issue worth revisiting soon. Uncompressed (23001-17) supports native alpha, and I understand it is possible for HEVC and JPEG 2000 (although likely not that common, yet).

Could re-use of the 23001-17 cmpd box be a solution? It wouldn't necessarily tell you what order the components were in, and the current semantics are that there can be components that are referenced, but the structure might still work as a descriptive item property.

[leo-barnes]

Could re-use of the 23001-17 cmpd box be a solution? It wouldn't necessarily tell you what order the components were in, and the current semantics are that there can be components that are referenced, but the structure might still work as a descriptive item property.

It's missing some stuff that is really required (premultiplied alpha for example). I'm trying to address this with an updated pixi property that can also signal stuff like floating point.

[bradh]

Could re-use of the 23001-17 cmpd box be a solution? It wouldn't necessarily tell you what order the components were in, and the current semantics are that there can be components that are referenced, but the structure might still work as a descriptive item property.

It's missing some stuff that is really required (premultiplied alpha for example). I'm trying to address this with an updated pixi property that can also signal stuff like floating point.

Premultiplied alpha is a good point. Possibly that could go into uncompressed too?

I realise it is very late in the cycle for 23001-17, but if pixi is going to hold the component format, possibly we can change the balance between what is (currently) uncC and cmpd, and then just not use cmpd. Once -17 goes out, that will be a harder change to make though.

[leo-barnes]

I'm hoping to bring an updated pixi proposal to MPEG 145, along with a proposal to add more generic subsampling and "chroma" location information to CICP so that we have that stuff in a centralized place rather than duplicated everywhere. We can certainly discuss there how it ties into 23001-17.

[cconcolato]

@leo-barnes any update on this one?

[leo-barnes]

The pixi proposal has landed in the WD of AMD3, so I think this one can be closed.