Feature request: Support unclamped floating point RGB data, and HDR JXR images

[tongyuantongyu]

With gainmap feature support, most colorspace manipulation functionalities are implemented in libavif. It may worth consider supporting floating point RGB input data with values outside [0, 1] range (Linear TC in H.273 clamps value), which is common for HDR data.

A good example is scRGB: scRGB uses BT709 primaries and a linear transfer characteristics where 1.0 maps to 80 nit. Windows uses scRGB for composition so HDR content captured on Windows usually are in this colorspace.

Xbox Game Bar and GeForce Experience are two common tools used to take HDR screenshot (of games usually), and they both directly save captured float data in JXR format, as Windows natively supports it. As a showcase for floating point RGB support, and providing a tool to convert HDR screenshot to AVIF for sharing, it may also worth consider supporting loading JXR images in avifenc.

Here are some example JXR files from Microsoft: https://aka.ms/hdr-backgrounds-wip

[y-guyon]

Thank you for the report.

consider supporting floating point RGB input data

Regarding the API, there is avifRGBImage:

https://github.com/AOMediaCodec/libavif/blob/4865c1ce145f919838f416469223eca142d24b58/include/avif/avif.h#L919

Unfortunately converting from F16 RGB to YUV is unimplemented.

with values outside [0, 1] range

I believe this is possible with today's API and some sample manipulation:

• Clamp samples in [0:1] range to make the base image
• Use full-range samples to make the destination image
• Call avifRGBImageComputeGainMap() with these two images to obtain the YUV gain map
• Implement avifImageRGBToYUV() for F16 or convert the base image to integers
• Encode these two into an AVIF image

Regarding avifenc, what would the input file format be for that use case (except JXR)?

consider supporting loading JXR images in avifenc

avifenc does not even support WebP input. In my opinion JXR input would be an even more niche use case.

What would be your options as of today to go from JXR to AVIF? From AVIF to JXR?
Gain maps in libavif being a recent experimental feature, I expect going through something like JXR -> SDR image + HDR image (PNG format?) -> avifgainmaputil combine -> AVIF. Assessing the current necessary steps and the minimal libavif additions to simplify these is the priority.

[kmilos]

Regarding avifenc, what would the input file format be for that use case (except JXR)?

Perhaps more widely used unclamped float options are EXR, TIFF/DNG and PFM (the latter requiring no dependency really as it's so simple). And JXL now maybe as well.

[tongyuantongyu]

I believe this is possible with today's API and some sample manipulation:

I'm considering both encoding normal HDR image and image with gainmap. Seems you are only considering gainmap encoding?

For normal HDR image encoding, the missing part is converting between float RGB and unorm RGB images (of different primaries and transfer function, most of which already implemented in #1861 and #1674). (Optionally we may also support direct conversion from float RGB to YUV). Gainmap computation would happily accepts float RGB and skip some work, since the computation happens on such unclamped linear float data.

Clamp samples in [0:1] range to make the base image

That's probably a bad idea, and I think we're not planning to do tone mapping in libavif.

What would be your options as of today to go from JXR to AVIF? From AVIF to JXR?

https://github.com/joedrago/colorist wrote by Joe can convert float JXR to HDR AVIF, but it haven't been updated for 2 years. I personally have a fork with some necessary updates making it work with the newest libavif. https://github.com/brion/hdrfix tone-maps float JXR to SDR JPG/PNG. The reference library jxrlib provides a JxrDecApp that converts JXR to BMP/TIF, but only support integer format. imagemagick's JXR support calls JxrDecApp through command line, so it can't deal with float JXR either.

As for AVIF to JXR (with float RGB format) there's no such tool (and probably no such need).

JXR -> SDR image + HDR image (PNG format?)

Let me rewrite this to JXR -> Float RGB -> UNorm RGB (HDR)

• Float RGB -> UNorm RGB (HDR): Is the first part of the proposal.
• JXR -> Float RGB: Unfortunately there's no notable tool capable of converting float JXR to some other float image format. One have to write some code to do it.

EXR, TIFF/DNG and PFM

I wonder if there is reasonable way non-professional user can produce images of these formats, like JXR's situation? If they always come from professional workflow, maybe libavif is not the best place handling them.

---

For JXR support, maybe we can make it optional and only available on Windows. Windows system API provides support for JXR, so no extra dependency is needed in this case.

[y-guyon]

Thank you for the explanation.

Seems you are only considering gainmap encoding?

Indeed.

For normal HDR image encoding, the missing part is converting between float RGB and unorm RGB images (of different primaries and transfer function, most of which already implemented in https://github.com/AOMediaCodec/libavif/pull/1861 and https://github.com/AOMediaCodec/libavif/pull/1674). (Optionally we may also support direct conversion from float RGB to YUV). Gainmap computation would happily accepts float RGB and skip some work, since the computation happens on such unclamped linear float data.

Assuming this is done both for "normal" HDR (which is PQ or HLG, right?) and for gain maps, there is little benefit in that feature unless avifenc accepts input file formats supporting float RGB (such as JXR), correct? Or do you foresee uses of that feature through the libavif API?

For JXR support, maybe we can make it optional and only available on Windows. Windows system API provides support for JXR, so no extra dependency is needed in this case.

Another entry in ext/ is not such a big deal in my opinion. Interfacing with the added library API and thoroughly testing it is usually the pain point. I doubt adding platform-dependent code will help in the long run.

[tongyuantongyu]

which is PQ or HLG, right?

HLG has more nuance due to it being scene referred. In the first step, I intend to only support PQ.

Or do you foresee uses of that feature through the libavif API?

One usage I can imagine would be screenshot tools. Those JXR screenshots store exactly what Windows API returns, so tools can feed the same data to libavif and get HDR screenshot, but this time in AVIF format.

Other cases dealing with HDR content would mostly be in professional workflows, so I expect more advanced tools being used.

Interfacing with the added library API and thoroughly testing it is usually the pain point.

For the float RGB support part, I was considering it too complicated to support in libavif. I bring this idea out now because gainmap support have most pieces implemented, and while we are here supporting this shouldn't be too much extra work. Gainmap itself is experimental though, so we may set back and wait gainmap to graduate from expermental state first.

For JXR support, I prefer strictly limiting our support to 4 channel interleaved RGBA F16 and F32 pixel formats only, which should drastically reduce the burden to support it. Putting JXR support in contrib as a standalone tool, dedicated for dealing with those screenshots is also an option.

[y-guyon]

converting between float RGB and unorm RGB images of different primaries and transfer function

If I understand correctly, this would be done using the following:

avifRGBImage floatRgb;
floatRgb->depth = 16;
floatRgb->isFloat = true;
// Set other fields accordingly

avifImage * unormRgb = avifImageCreate(floatRgb.width, floatRgb.height, /*depth=*/12, AVIF_PIXEL_FORMAT_YUV444);
unormRgb->matrixCoefficients = AVIF_MATRIX_COEFFICIENTS_IDENTITY;
// Set other fields accordingly, for example AVIF_COLOR_PRIMARIES_SRGB and AVIF_TRANSFER_CHARACTERISTICS_UNSPECIFIED
avifImageAllocatePlanes();

avifImageRGBToYUV(unormRgb, &floatRgb);

avifImageRGBToYUV() fails because of:

https://github.com/AOMediaCodec/libavif/blob/03a12f8aa325c55ea5bd228afa591f67632b018f/src/reformat.c#L236-L238

So replacing the AVIF_RESULT_NOT_IMPLEMENTED above by the actual implementation would convert from float RGB to unorm RGB. Would this be enough for the use case you described?

we may also support direct conversion from float RGB to YUV

This is the same code path as above, unless I missed something.

---

For JXR support, I prefer strictly limiting our support to 4 channel interleaved RGBA F16 and F32 pixel formats only, which should drastically reduce the burden to support it. Putting JXR support in contrib as a standalone tool, dedicated for dealing with those screenshots is also an option.

Some JXR reader in either apps/shared (new work, same license), contrib (new work, different license) or third_party (copypasted from another repo) would be fine with me. Are you planning to do that or just suggesting the feature addition?

[tongyuantongyu]

This is the same code path as above, unless I missed something.

I was considering a separated avifRGBImageFloatToUNorm function, and implement avifJXRReadRGB. This would avoid extra work for gainmap usage (RGB->YUV in avif*Read, then YUV->RGB in avifImageComputeGainMap). This also applies to avifJPEGReadRGB and avifPNGReadRGB, but this should better be discussed as a separated task.

Are you planning to do that or just suggesting the feature addition?

I'd like to implement this if we consider it worthwhile.

Some JXR reader in either apps/shared (new work, same license), contrib (new work, different license) or third_party (copypasted from another repo)

If we are going to support JXR in avifenc, we need to add jxrlib library in third_party. Otherwise I prefer adding a Windows only tool in contriband use Windows Image Component to read JXR, to make it simpler.

[y-guyon]

This would avoid extra work for gainmap usage

Performance is not critical yet there. For now I would favor simplicity for review, test coverage, maintenance and debugging easiness over fewer plane copies, which is usually a cheap operation compared to encoding.
So unless the added complexity is similar, please replace the AVIF_RESULT_NOT_IMPLEMENTED in the current API instead of adding new functions (when possible).

I'd like to implement this if we consider it worthwhile.

Feel free to do so, as you said HDR is quite experimental for now so even if it creates issues, it will just be for a few expert users.

If we are going to support JXR in avifenc, we need to add jxrlib library in third_party . Otherwise I prefer adding a Windows only tool in contriband use Windows Image Component to read JXR, to make it simpler.

I would prefer a platform-independent solution. Also most of our development and testing happen in Linux environments, so debugging there will be easier.