Tone mapping: improve handling of very bright scenes/movies

[FoLLgoTT]

The masterings of different HDR movies vary a lot in terms of average brightness (not peak!). There are movies or scenes that have a significant amount of content >100 nits. These scenes are displayed too bright with the tone mapping algorithms of MPV when using a lower target-peak (e.g. 150) which usually fits well to most movies.

madVR has an option to handle that which is called "dynamic target nits". From my understanding this controls the strength for increasing the point where to start the tone mapping depending on the average brightness of the picture. In other words: it dynamically shifts the "reference white" to higher nits. Such an option seems to be missing in MPV.

I tried different tone mapping algorithms, but the problem seems to be in all of them. I usually use BT.2339, but Reinhard or BT.2446a have the same problem.

Examples of movies with high average brightness:

• Meg
• Aquaman
• Cars 3
• Starship Troopers
• Lion King (2019)

Example images ("Meg", chapter 7)

Histogram:

MPV with target-peak=auto (resulting in tone mapping to 203 nits):

madVR (target nits = 100, dynamic target nits = 30, resulting in tone mapping to 1224 nits):

MPV with target-peak=1220:

Expected behavior of the wanted feature

Better adaption on high average brightness.

Log file

log.txt

[haasn]

FWIW, I implemented histogram support (and dynamic measurement), so now we have these values to play with. The problem is that I have no idea what to do with these values:

One idea I had is to try and brute-force fit a curve onto the histogram values:

But this completely destroys the image. In this scene it isn't as bad as in others, but in most scenes it just completely demolishes the appearance of the image:

[dyphire]

@haasn To clarify, please tell me whether the correct way to use of the new spline in mpv is to use --hdr-compute-peak=no? When I use --hdr-compute-peak=no again for testing, I can always get more natural brightness content, and I can't see the obvious loss of detail before. edit: I got darker content with --hdr-compute-peak=no in other sample tests, which seems to be a wrong attempt.

Is it wrong to use --hdr-compute-peak=noauto/yes for testing before? If so, I will say that this patch works well.

[haasn]

Hmm, that sounds wrong, hdr-compute-peak definitely shouldn't be decreasing the quality of the result. But, if possible, I'd prefer to test on real content, not demo reels - the latter are notoriously bad.

Can you extract 1 frame of one of the bright problem scenes you identified in OP?

[quietvoid]

When I use --hdr-compute-peak=no again for testing, I can always get more natural brightness content, and I can't see the obvious loss of detail before.

The problem here is that the sample says it's 1000 nits, when it's not. So with hdr-compute-peak=no, it's assuming the frame doesn't go over 1000 nits so the curve is steeper.

Original: https://0x0.st/Hzfv.mkv 10 000 reencoded version: https://0x0.st/Hzft.hevc

[dyphire]

Hmm, that sounds wrong, hdr-compute-peak definitely shouldn't be decreasing the quality of the result. But, if possible, I'd prefer to test on real content, not demo reels - the latter are notoriously bad.

Can you extract 1 frame of one of the bright problem scenes you identified in OP?

If the real movie is used as the test sample, it should be good to use the original issue's example. The.Meg_test.zip

[dyphire]

The problem here is that the sample says it's 1000 nits, when it's not. So with hdr-compute-peak=no, it's assuming the frame doesn't go over 1000 nits so the curve is steeper.

Original: https://0x0.st/Hzfv.mkv 10 000 reencoded version: https://0x0.st/Hzft.hevc

You're right. Anyway, we should improve the current spline in extremely bright scenes, it now actively clamps the highlights and darkens the content like bt.2446a.

[haasn]

Okay, thanks, I see the issue in these samples comes from the excessively high scene average brightness, which is so high that we "sanity clamp" it. I'll think about how to handle this better.

[haasn]

Actually, simply changing the bounds of these sanity clamps helps a lot:

diff --git a/src/tone_mapping.c b/src/tone_mapping.c
index 4644f1c9..8cde283b 100644
--- a/src/tone_mapping.c
+++ b/src/tone_mapping.c
@@ -295,9 +295,9 @@ static void st2094_pick_knee(float *out_src_knee, float *out_dst_knee,
                              float adaptation)
 {
     // Knee point default, minimum and maximum
-    const float min_src_knee = 0.3f;
-    const float def_src_knee = 0.4f;
-    const float max_src_knee = 0.5f;
+    const float min_knee = 0.1f;
+    const float def_knee = 0.4f;
+    const float max_knee = 0.8f;

     float src_min = pl_hdr_rescale(params->input_scaling,  PL_HDR_PQ, params->input_min);
     float src_max = pl_hdr_rescale(params->input_scaling,  PL_HDR_PQ, params->input_max);
@@ -306,11 +306,11 @@ static void st2094_pick_knee(float *out_src_knee, float *out_dst_knee,

     // Choose default scene average brightness to be a fixed percentage of the
     // value range, override with (clamped) HDR10+ metadata if available
-    float target_avg = def_src_knee;
+    float target_avg = def_knee;
     if (params->hdr.scene_avg) {
         float scene_avg = pl_hdr_rescale(PL_HDR_NITS, PL_HDR_PQ, params->hdr.scene_avg);
         target_avg = (scene_avg - src_min) / (src_max - src_min);
-        target_avg = PL_CLAMP(target_avg, min_src_knee, max_src_knee);
+        target_avg = PL_CLAMP(target_avg, min_knee, max_knee);
     }

     float src_avg = PL_MIX(src_min, src_max, target_avg);
@@ -322,9 +322,8 @@ static void st2094_pick_knee(float *out_src_knee, float *out_dst_knee,
     // (neutral) line.
     dst_avg = PL_MIX(src_avg, dst_avg, adaptation);

-    // Hard-clamp at 20% from either edge to soften extreme cases
-    const float dst_knee_min = PL_MIX(dst_min, dst_max, 0.2f);
-    const float dst_knee_max = PL_MIX(dst_min, dst_max, 0.8f);
+    const float dst_knee_min = PL_MIX(dst_min, dst_max, min_knee);
+    const float dst_knee_max = PL_MIX(dst_min, dst_max, max_knee);
     dst_avg = PL_CLAMP(dst_avg, dst_knee_min, dst_knee_max);

     *out_src_knee = pl_hdr_rescale(PL_HDR_PQ, params->input_scaling, src_avg);

I forgot why I added them originally, actually, but I suspect the nu-spline no longer needs them.

[dyphire]

Actually, simply changing the bounds of these sanity clamps helps a lot:

diff --git a/src/tone_mapping.c b/src/tone_mapping.c
index 4644f1c9..8cde283b 100644
--- a/src/tone_mapping.c
+++ b/src/tone_mapping.c
@@ -295,9 +295,9 @@ static void st2094_pick_knee(float *out_src_knee, float *out_dst_knee,
                              float adaptation)
 {
     // Knee point default, minimum and maximum
-    const float min_src_knee = 0.3f;
-    const float def_src_knee = 0.4f;
-    const float max_src_knee = 0.5f;
+    const float min_knee = 0.1f;
+    const float def_knee = 0.4f;
+    const float max_knee = 0.8f;

     float src_min = pl_hdr_rescale(params->input_scaling,  PL_HDR_PQ, params->input_min);
     float src_max = pl_hdr_rescale(params->input_scaling,  PL_HDR_PQ, params->input_max);
@@ -306,11 +306,11 @@ static void st2094_pick_knee(float *out_src_knee, float *out_dst_knee,

     // Choose default scene average brightness to be a fixed percentage of the
     // value range, override with (clamped) HDR10+ metadata if available
-    float target_avg = def_src_knee;
+    float target_avg = def_knee;
     if (params->hdr.scene_avg) {
         float scene_avg = pl_hdr_rescale(PL_HDR_NITS, PL_HDR_PQ, params->hdr.scene_avg);
         target_avg = (scene_avg - src_min) / (src_max - src_min);
-        target_avg = PL_CLAMP(target_avg, min_src_knee, max_src_knee);
+        target_avg = PL_CLAMP(target_avg, min_knee, max_knee);
     }

     float src_avg = PL_MIX(src_min, src_max, target_avg);
@@ -322,9 +322,8 @@ static void st2094_pick_knee(float *out_src_knee, float *out_dst_knee,
     // (neutral) line.
     dst_avg = PL_MIX(src_avg, dst_avg, adaptation);

-    // Hard-clamp at 20% from either edge to soften extreme cases
-    const float dst_knee_min = PL_MIX(dst_min, dst_max, 0.2f);
-    const float dst_knee_max = PL_MIX(dst_min, dst_max, 0.8f);
+    const float dst_knee_min = PL_MIX(dst_min, dst_max, min_knee);
+    const float dst_knee_max = PL_MIX(dst_min, dst_max, max_knee);
     dst_avg = PL_CLAMP(dst_avg, dst_knee_min, dst_knee_max);

     *out_src_knee = pl_hdr_rescale(PL_HDR_PQ, params->input_scaling, src_avg);

I forgot why I added them originally, actually, but I suspect the nu-spline no longer needs them.

LGTM.

[dyphire]

I found a issue with peak detection. When the average brightness difference of frame switching is great, it will get obvious wrong results. test video: Aquaman_test.zip Problem screenshot: Expected behavior, correct peak detection results can be obtained by manually switching hdr-compute-peak=no and hdr-compute-peak=yes under this frame:

update: The last working version is https://github.com/haasn/libplacebo/commit/3c65e123399f588c30a9aeccd84e9105b03396af, I guess this commit https://github.com/haasn/libplacebo/commit/daed681e04e7ccd36b1302940ec2c2001b9b67ee broke it.

[quietvoid]

@dyphire I think this is fixed by https://github.com/haasn/libplacebo/tree/fix_peak. Specifically https://github.com/haasn/libplacebo/commit/d726b36d719dd61690c5b20b12a25286762120eb

It's definitely fixed in fix_peak for me. edit: I cherry-picked the commit and it indeed fixes the problem.

[haasn]

I think that what broke is actually the scene change hysteresis. I'll fix it. Can you open new issues for new issues, though?

[haasn]

Pushed a fix in https://github.com/haasn/libplacebo/tree/remove_peak

[dyphire]

@dyphire I think this is fixed by https://github.com/haasn/libplacebo/tree/fix_peak. Specifically haasn/libplacebo@d726b36

It's definitely fixed in fix_peak for me. edit: I cherry-picked the commit and it indeed fixes the problem.

Yes, I can confirm that fix it.