Possible to do ACES tonemapping with OpenColorIO's studio-config?

[ggarra13]

First, let me say that color science is not my area of expertise (neither is math !!! :)

One of my mrv2 users pointed out that my viewer is not handling HDR to SDR conversions properly. I am reading the FFmpeg metadata of the .mp4 file and trying to apply it to an OCIO pipeline.

Now I am trying to get a pipeline that works for OCIO2. I am trying to avoid using another library like libplacebo or similar.

This is what I have so far:

            std::cerr << "display/view tonemapping... data starts in Rec.2020 space" << std::endl;
            //
            // TONE-MAPPING
            //
            const image::HDRData& hdrData = p.tonemapOptions.hdrData;

            // Step 1: Color Primaries (Rec. 2020 to XYZ)

            OCIO::MatrixTransformRcPtr rec2020toXYZ = OCIO::MatrixTransform::Create();
            // Construct the 4x4 matrix (XYZ to RGB conversion matrix)
            math::Matrix4x4<double> Rec2020ToXYZ(
                   0.636958,  0.144617,  0.168881, 0.F, // Red
                   0.262700,  0.677998,  0.059302, 0.F, // Green
                   0.000000,  0.028073,  1.060985, 0.F, // Blue
                   0.000000,  0.000000,  0.000000, 1.F
            );
            rec2020toXYZ->setMatrix(Rec2020ToXYZ.e);

            OCIO::MatrixTransformRcPtr customPrimaryTransform = OCIO::MatrixTransform::Create();

            // Step 2: Use FFmpeg's custom primaries

            using image::HDRPrimaries;
            const std::array<math::Vector2f,
                  image::HDRPrimaries::Count>& v = hdrData.primaries;

            auto toXYZ = [](float x, float y) {
                            float Y = 1.0f;  // Luminance
                            float X = x / y * Y;
                            float Z = (1 - x - y) / y * Y;
                            return math::Vector3f(X, Y, Z);
            };

            // Convert chromaticity to XYZ
            math::Vector3f redXYZ   = toXYZ(v[HDRPrimaries::Red].x, v[HDRPrimaries::Red].y);
            math::Vector3f greenXYZ = toXYZ(v[HDRPrimaries::Green].x, v[HDRPrimaries::Green].y);
            math::Vector3f blueXYZ  = toXYZ(v[HDRPrimaries::Blue].x, v[HDRPrimaries::Blue].y);
            math::Vector3f whiteXYZ = toXYZ(v[HDRPrimaries::White].x, v[HDRPrimaries::White].y);

            // Construct the 4x4 matrix (XYZ to RGB conversion matrix)
            math::Matrix4x4<double> m(
                   redXYZ.x,   greenXYZ.x,   blueXYZ.x,   0.0,
                   redXYZ.y,   greenXYZ.y,   blueXYZ.y,   0.0,
                   redXYZ.z,   greenXYZ.z,   blueXYZ.z,   0.0,
                   whiteXYZ.x, whiteXYZ.y, whiteXYZ.z, 1.0
            );

            std::cerr << "Rec2020 to XYZ" << std::endl;
            for (int i = 0; i < 16; i += 4)
            {
                  std::cerr << m.e[i] << ", " << m.e[i+1] << ", " << m.e[i+2] << ", " << m.e[i+3] << std::endl;
            }
            customPrimaryTransform->setMatrix(m.e);

            // Step 2: XYZ to ACES (Assuming ACES2065-1 AP0)
            //             Any way not to hard-code the matrix?
            math::Matrix4x4<double> XYZ_to_ACES(
                            1.04981102,  0.00000000, -0.00009748, 0.F,
                            -0.49590302,  1.37331305,  0.09824004, 0.F,
                            0.00000000,  0.00000000,  0.99125202, 0.F,
                            0.F       ,  0.F       ,  0.F,        1.F);
            OCIO::MatrixTransformRcPtr xyzToACES = OCIO::MatrixTransform::Create();
            xyzToACES->setMatrix(XYZ_to_ACES.e);

            //
            // This step SEEMS wrong.  There are no PQ or HG spaces.  I am using the display spaces,
            // which I am not sure is correct.
            //
            // // Step 3: EOTF (PQ or HLG to Linear)
            //
            // OCIO::ColorSpaceTransformRcPtr eotfToLinear;
            // if (hdrData.eotf == 0) { // PQ
            //     eotfToLinear = OCIO::ColorSpaceTransform::Create();
            //     eotfToLinear->setSrc("Rec.2100-PQ - Display");
            //     eotfToLinear->setDst(OCIO::ROLE_SCENE_LINEAR);
            //     //eotfToLinear->setDst("Linear");
            // }
            // else if (hdrData.eotf == 1)
            // { // HLG
            //     eotfToLinear = OCIO::ColorSpaceTransform::Create();
            //     eotfToLinear->setSrc("Rec.2100-HLG - Display");
            //     eotfToLinear->setDst(OCIO::ROLE_SCENE_LINEAR);
            //     //eotfToLinear->setDst("Linear");
            // }

            // Step 4: Luminance Adjustment (MaxCLL, MaxFALL,
            //         Mastering Luminance)
            float maxLuminance = hdrData.displayMasteringLuminance.getMax();
            OCIO::ExposureContrastTransformRcPtr luminanceTransform = OCIO::ExposureContrastTransform::Create();
            luminanceTransform->setExposure(log2(100.0F / maxLuminance)); // Normalize to 100 nits SDR

            // Step 5: ACES Tone Mapping (RRT + ODT for Rec. 709 SDR)
            p.ocioData->transform = OCIO::DisplayViewTransform::Create();
            p.ocioData->transform->setSrc(OCIO::ROLE_SCENE_LINEAR);
            p.ocioData->transform->setDisplay("sRGB - Display");
            p.ocioData->transform->setView("ACES 1.0 - SDR Video");

            // Combine all transforms into a processor
            OCIO::GroupTransformRcPtr group = OCIO::GroupTransform::Create();
            group->appendTransform(rec2020toXYZ);
            group->appendTransform(customPrimaryTransform);
            group->appendTransform(xyzToACES);
            // group->appendTransform(eotfToLinear);
            group->appendTransform(luminanceTransform);
            group->appendTransform(p.ocioData->transform);

            p.ocioData->processor = p.ocioData->config->getProcessor(group);

            // ...etc...

[doug-walker]

Hi Gonzalo, before diving into discussing code, could you please let us know:

1. What non-linearity is used to encode your input (ST-2084/PQ?)?
2. Do you really need to support "custom primaries" for HDR video from ffmpeg? I would guess that there are really only three sets of primaries that are likely (Rec.709, P3-D65, and Rec.2020). OCIO has built-in conversions for those, so you may want to just check to see which one it is rather than supporting arbitrary possibilities.
3. What color space do you need to output (Rec.709?)?
4. What kind of HDR to SDR conversion do you think your users want?

Some of the stuff you are doing in your code is not necessary since OCIO already has built-in transforms that could be used. However, OCIO does not currently have specific HDR to SDR conversion options, so that part would need to be more bespoke. Depending on whether you need local tone-mapping or some proprietary algorithm such as Dolby Vision, it may or may not be possible to do that part in OCIO. There are LUTs available for licensing from various places that could be applied using OCIO. There are so many options, I'm not sure what the best published algorithm to recommend currently is, if you're looking to implement it yourself.

[ggarra13]

Thanks Doug. I'll try to answer the best I can.

The movie files in question are:

HDR: https://mega.nz/file/iGgxXLjL#gEsU5XGQNgRkXgWEbhqm0tZyD9azoY1T6YhqNi35OYI

SDR: https://mega.nz/file/jKYzUIZC#Nz34EVAv3SDPwu1atsTPvL-MdSSXPutTxmL5ikC3SZg

Here's a snapshot of all the HDR info I can get from FFmpeg:

Side data:
  Mastering Display Metadata, has_primaries:1 has_luminance:1 r(0.6800,0.3200) g(0.2650,0.6900) b(0.1500 0.0600) wp(0.3127, 0.3290) min_luminance=0.000100, max_luminance=1000.000000

Note that that metadata is for the stream. The movie in question also has frame HDR metadata which is:

1. What non-linearity is used to encode your input (ST-2084/PQ?)?

Not sure.

2. Do you really need to support "custom primaries" for HDR video from ffmpeg? I would guess that there are really only three sets of primaries that are likely (Rec.709, P3-D65, and Rec.2020). OCIO has built-in conversions for those, so you may want to just check to see which one it is rather than supporting arbitrary possibilities.

Ideally, yes, as FFmpeg's metadata in v7.0 now can change from frame to frame.

3. What color space do you need to output (Rec.709?)?

Rec.709, indeed.

4. What kind of HDR to SDR conversion do you think your users want?

For this example, one that matches Apple's videotoolbox more closely.
In a perfect world, I was shooting for ACES HDR to SDR conversion, as that should be the best one, so that even if it does not match exactly it would be the superior algorithm and hopefully the differences should be minimum.

[doug-walker]

I couldn't open the files you posted, but the Mastering Display Metadata is a description of the color gamut and dynamic range of the display the content was color graded on. It is not necessarily an indication of the color space that the movie is encoded in for transport. For example, the grading monitor typically has a P3 gamut but an HDR movie would typically be transported using Rec.2100 (HLG or PQ), which uses the Rec.2020 gamut. In short, you should not be converting color spaces based on that metadata. It is just info that might be useful for tone or gamut mapping, to know the extent of the source content.

Please keep in mind that ACES does not offer HDR to SDR conversion, per se. What it offers is a set of Output Transforms that convert scene-referred source data to various displays, both HDR and SDR. Yes, because the transforms may be applied in the inverse direction, you could convert HDR back to ACES2065-1 through one transform and then forward through an SDR transform. But in practice, I've not heard of anyone doing that.