HEIF: How to indicate that coded item contains floating-point

[leo-barnes]

There are now a variety of codecs that support floating-point compression (OpenEXR, JPEG XL). Sooner or later HEIF will get bindings for such a codec. How do we indicate that the output of the coded item is floating point? We can indicate the bit-depth by using the pixi property, but AFAIK there is no way of indicating that it is floating point.

[denoualf]

What you wrote above @leo-barnes could make think that it relates to and could go into codec configuration... but actually pixi may be a better option and indeed the current version does not allow this.

[leo-barnes]

@denoualf

could go into codec configuration

It most likely will be specified in the codec config, but so is the bit-depth. We really want to have a codec agnostic way to indicate at the container level that the data is floating point and not integers. MIAF for instance requires a pixi property to be present. But what does that property even mean (as currently specified) if the data is floating point?

It feels like we need a v1 of the pixi that allows specifying that the data is not actually integers.

[y-guyon]

MIAF for instance requires a pixi property to be present. But what does that property even mean (as currently specified) if the data is floating point?

If colr is present with colour_type=nclx, the current definition of the full_range_flag clips the output samples to a range of positive integers for specified MatrixCoefficients:

ISO/IEC 14496-12 (HEIF)

12.1.5.3 Semantics

full_range_flag carries a VideoFullRangeFlag as defined in ISO/IEC 23091-2

ISO/IEC 23091-2 (CICP)

8.3 Matrix coefficients

[...] VideoFullRangeFlag specifies the scaling and offset values applied in association with the MatrixCoefficients. When not present or not specified, the value 0 for VideoFullRangeFlag would ordinarily be inferred as the default value for video imagery. [...] — If VideoFullRangeFlag is equal to 0, the following applies: — If MatrixCoefficients is equal to 0 or 8, Formulae (21) to (23) apply: R = Clip1Y ( ( 1 << ( BitDepthY − 8 ) ) * ( 219 * E′R + 16 ) ) [...] — Otherwise, if MatrixCoefficients is equal to 1, 4, 5, 6, 7, 9, 10, 11, 12, 13, or 14, Formulae (24) to (26) apply: Y = Clip1Y ( Round( ( 1 << (BitDepthY − 8 ) ) * ( 219 * E′Y + 16 ) ) ) [...] — Otherwise, if MatrixCoefficients is equal to 2, the interpretation of the MatrixCoefficients code point is unknown or is determined by the application. — Otherwise (MatrixCoefficients is not equal to 0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, or 14), the interpretation of the MatrixCoefficients code point is reserved for future definition by ITU-T | ISO/IEC. [...] [other rounding and clipping formulae]

AV1-ISOBMFF for example says The sample entry SHOULD contain a 'colr' box with a colour_type set to 'nclx'.

Allowing negative integer or floating point samples will require amendments in the specified MatrixCoefficients, using Unspecified as MatrixCoefficients, or introducing other MatrixCoefficients.

Maybe colr is a better fit for specifying int/float than pixi.

---

It feels like we need a v1 of the pixi that allows specifying that the data is not actually integers.

There should also be some way of signaling negative integer samples.

[leo-barnes]

Good points!

I think adding the signaling to pixi would work as long as the data is RGB rather than YUV. In other words, MatrixCoefficients needs to be 2 or Identity (can't remember the value, is it 1?).

But we may need some wording to indicate that this is how to interpret it until colr or CICP is updated to handle float/signed integers.

[leo-barnes]

Discussed during MPEG 143. Conclusion was that pixi is the right place to signal floating-point, but a more flexible approach than what was proposed is preferred (i.e. floating-point can be signaled per channel).

[leo-barnes]

@y-guyon point on CICP needing an update for use with floating point values needs to be discussed. Potentially we should open a separate issue for that since CICP is outside of HEIF.

[cconcolato]

@leo-barnes @y-guyon what should we do with this issue? Can we close?

[y-guyon]

I think this can be closed for the same reason as https://github.com/MPEGGroup/FileFormat/issues/81#issuecomment-2178365840.

[leo-barnes]

Yup, closing.