Quick Test Inquiry

[yazdinip]

Hello Andre,

This is a very interesting approach to HDR Image reconstruction. Thanks for the hard work. I did have a question. Using the test.py, without any previous training, I was able to test a few images. However the images look grey. Is this supposed to be like this? Sorry I am a bit of a noob in the area.

[AndreGuo]

Hi Yazdinia:

Thank for your interest, the answer will be two parts:

• Large container HDR/WCG will appear dim and desaturated when rendered by sRGB display;
• Your result is truly dimmer and more desaturated than it should be.

The reason lies in:

1. "HDR/WCG looks dim and dull when rendered as sRGB"

Since HDR/WCG (e.g. PQ/BT.2020) has a larger luminance and color container, the same real/scene luminance and color will have smaller normalized value in HDR/WCG than SDR (e.g. sRGB).

For example, 100nit is Y=1 in SDR, while only Y=0.58 in PQ HDR. In this case, if they are all rendered/displayed by a SDR monitor, SDR image will used 100% the luminance capacity of SDR display, while HDR can only utilize 58% (so appear dimmer).

The same goes for color/gamut. The same real/scene color will have less normalized saturation value in WCG (e.g. BT.2020) than BT.709/sRGB, so WCG will appear more desaturated/dull/muted than BT.709 when they are simultaneously rendered by same display.

• That's why it always says 'log film looks gray', because log by different camera all use larger luminance and color container (like HDR/WCG here), and you are watching it by sRGB display.
• You can also notice this phenomenon in page 95 of Brown ICCV23 ISP Tutorial, where same scene/real color in larger gamut RGB primaries appears more desaturated.
• To currently visualized HDR/WCG, you should use a monitor supporting e.g. PQ/BT.2020, set to this mode, and feed the same normalized picture (donot use Windows' HDR mode, it'll change pixel value).
• Say to this, I also want to mention that our HDR/WCG concept is a little bit different from: HDRNet who create HDR look still in SDR/sRGB luminance and color container, HDRCNN who reconstruct HDR luminance in linear container (ours is PQ non-linear container) while leave gamut along, and SIG17HDR who merge sever SDRs into HDR.

2. "Bad aesthetic performance by current dataset (especially its degradation model)"

From the picture you provided, it seems you tested the method using default params.pth.

This params.pth is trained by old HDRTV1K dataset, LQ SDR there is degraded from GT HDR/WCG by YouTube default down-conversion, which will enhance the luminance and color in SDR. In this case, the model will learn to decrease the luminance and saturation when converting SDR to HDR/WCG.

This is one of the shortcomings of the current dataset and one of the major concerns of our paper. We set params.pth as default just because most literatures are still using current dataset, and we want to ease researchers' comparison experiments with other literatures.

If you want a good look of result, please use params_3DM.pth (the one shown in paper) or params_DaVinci.pth.

[yazdinip]

Hi Andre,

Thank you so much for the quick and detailed response. Was able to cross reference your description with the sources mentioned, I now understand why the results are expected. I also tried the 3DM and DaVinci models. At first glance, the other models seem to show similar results. Sample:

Using 3DM:

Using Davinci:

Unfortunately, I don't have a monitor that can support the PQ/BT.2020 format and I have to rely on Windows HDR. For more context, I'm hoping to use your model to turn any SDR content on the screen such as Youtube to HDR. This is a very valuable use case as currently, even when a user enables Windows HDR on a supported monitor, most of the content is still on SDR. Would we be able to train the model for such scenario?

Many Thanks

[redrockvp]

Hi there, am relatively new to this scene, but I understand the above points about why it looks dimmer etc. What I don't understand, is how to actually display or get the image into a container that will display as HDR under Windows? If I open the tif file in Photoshop, it does not render as HDR, even though other HDR tifs from other methods do. I have a display capable of 1000 nits peak brightness HDR.

I'd love to try and implement this software into another pipeline, but would just like to understand what I'm missing first.

[AndreGuo]

Hi there, am relatively new to this scene, but I understand the above points about why it looks dimmer etc. What I don't understand, is how to actually display or get the image into a container that will display as HDR under Windows? If I open the tif file in Photoshop, it does not render as HDR, even though other HDR tifs from other methods do. I have a display capable of 1000 nits peak brightness HDR.

I'd love to try and implement this software into another pipeline, but would just like to understand what I'm missing first.

Usually PQ/BT.2020 format is for video, so I rec using ffmpeg to add some metadata to you video, so most video player will recognize and switch to HDR mode? (I don't know how specific video player cooperate with Window HDR mode, but anyhow we use ffplay to derectly play it on PQ/BT.2020 monitor, without turning on Window HDR mode), here is the sample script:

./ffmpeg.tvhdr.test.20220726.195461d.7d3e097.036cbeb.cpu/bin/set_vars.sh ffmpeg -y -i /mnt/data2/VQA2ITM/dataset/SDR/out/ferriswheel_SDR709_HEVC_QP25_50_10_%03d.tiff -t 0:10 -vf sdr2hdr=iprimaries=bt2020:itrc=smpte2084:ispace=gbr:irange=pc:primaries=bt2020:trc=smpte2084:space=bt2020nc:range=tv:format=yuv420p10:mode=passby:refer=scene -color_primaries bt2020 -color_trc smpte2084 -colorspace bt2020nc -color_range tv -tag:v hvc1 -vcodec libx265 -preset 6 -x265-params crf=10:no-info=1 -movflags faststart HDR10_out.mp4

For HDR images in tif format, it's also ok to use 'ffplay'.

[redrockvp]

Thanks for the feedback here. I will give ffplay a try and see what I can do.

A followup question though, would it be possible to just set the correct HDR metadata on the tif file via the Python code?

I have tried a few methods myself, but it doesn't seem to work correctly. It would also be incredibly handy to have options to output as avif or jpeg-xl (with the proper metadata) when you're just working with images. Then you'd have an output that could immediately be shared on platforms that support HDR, e.g. Instagram has now started supporting avif and jpeg-xl uploads and most phones are HDR capable already.

[AndreGuo]

Hi Andre,

Thank you so much for the quick and detailed response. Was able to cross reference your description with the sources mentioned, I now understand why the results are expected. I also tried the 3DM and DaVinci models. At first glance, the other models seem to show similar results.

Unfortunately, I don't have a monitor that can support the PQ/BT.2020 format and I have to rely on Windows HDR. For more context, I'm hoping to use your model to turn any SDR content on the screen such as Youtube to HDR. This is a very valuable use case as currently, even when a user enables Windows HDR on a supported monitor, most of the content is still on SDR. Would we be able to train the model for such scenario?

Many Thanks

Hi:

1. I donnot rec using windows HDR mode, I still donnot understand its pipeline, and its hard to calibrate. I rec using ffmpeg or other tool to add corect metadata to the video, so some video player will support? See the last comment for ffmpeg sample usage.

2. You mentioned that you want to convert SDR to HDR in display-end in real-time. Unfortunately, our neural network (NN) model is not lightweight enough, so it might not be able to run in real-time on common GPU with limited FLOPs. You can refer to more efficient method e.g. LUT (look-up table) or lightweight NN (might be integrated in some SNOY/Philips etc. TV, but we don't know the excat implementation).