search

hecomi / uDesktopDuplication

Desktop Duplication API implementation for Unity (only for Windows 8/10)
http://tips.hecomi.com/entry/2016/12/04/125641
MIT License
557 stars 97 forks source link

HDR displays are captured incorrectly. #46

Open Curtis-VL opened 3 years ago

Curtis-VL commented 3 years ago

HDR displays capture with incorrect colors: https://i.gyazo.com/9280c7dae0667a5eb7fe1a542f257638.jpg

I'm happy to test any changes you make if you don't have a HDR compatible display to test with. 👍

hecomi commented 3 years ago

Thank you for the report! I have one display that supports HDR. I'll check the bug with it.

Curtis-VL commented 3 years ago

Thank you for looking into this! :)

hecomi commented 3 years ago

I've tried to fix the problem..., but couldn't seem to solve now. Ref: https://github.com/psieg/Lightpack/issues/343 I've attempted a new API DuplicateOutput1 but couldn't get any good result. Even Windows Alt + Tab has incorrect colors now (when showing a window on HDR display in the Alt + Tab window on SDR display). Instead of this, I've added isHDR member to Monitor in the above commit. So please try to show a warning message or something when an user is using HDR display with your app if needed.

Curtis-VL commented 3 years ago

That'll help with the issue, thank you for your time!

Andon13 commented 3 years ago

I think there's a better way of handling this.

Rather than this new isHDR field, could you instead store the colorimetry info returned by IDXGIOutput6::GetDesc1 (...) for the monitor?

The color points aren't really necessary, since 9/10 the desktop's scRGB colorspace is going to be transformed to sRGB and they share the same white/red/green/blue color points.

With the additional info above, it's possible to run the duplicated image through a shader and rescale the image for sRGB output. Tonemapping would be preferable, but that's quite a bit more work, and simply scaling the range between MinLuminance / MaxLuminance into SDR is adequate for windows that aren't displaying actual HDR content.

Really the only thing you need to know once you've got the min/max luminance values is how luminance is encoded in scRGB. It's quite simple:

The RGB triplet 1.0, 1.0, 1.0 is 80.0 nits white, and luminance increases linearly such that 2.0, 2.0, 2.0 is 160.0 nits white.

The brightest pixel you're going to encounter with the desktop in HDR mode will have value 0.0125 MaxLuminance, and black is anything darker than 0.0125 MinLuminance.

Since the source is a linear colorspace and sRGB is not, you're probably going to need also to apply sRGB gamma after scaling for range.

TayouVR commented 3 years ago

I am working towards doing the above mentioned right now, I have now passed the values from the API calls down to the shader and need to find out how to convert the color space. I hope it won't be too hard. If all works well we might have something working by tomorrow. https://github.com/TayouVR/uDesktopDuplication/commit/f97aed5bd1f6d5d391f5f624b28869204055f32d

TayouVR commented 3 years ago

Really the only thing you need to know once you've got the min/max luminance values is how luminance is encoded in scRGB. It's quite simple:

The RGB triplet 1.0, 1.0, 1.0 is 80.0 nits white, and luminance increases linearly such that 2.0, 2.0, 2.0 is 160.0 nits white.

The brightest pixel you're going to encounter with the desktop in HDR mode will have value 0.0125 MaxLuminance, and black is anything darker than 0.0125 MinLuminance.

Since the source is a linear colorspace and sRGB is not, you're probably going to need also to apply sRGB gamma after scaling for range.

in my fork I have sent the necessary information to the shader to do this calculation, I however do not understand what calculations actually need to be performed.

With the additional info above, it's possible to run the duplicated image through a shader and rescale the image for sRGB output. Tonemapping would be preferable, but that's quite a bit more work, and simply scaling the range between MinLuminance / MaxLuminance into SDR is adequate for windows that aren't displaying actual HDR content.

I attempted to use the tonemapping code and logic from various other sources to achieve the desired effect, but my lack of knowledge about color spaces and an HDR display made it difficult.

In the end it does not seem to work and i don't think I can implement a proper color space correction. If anyone with the necessary knowledge would like to continue the work on it you can continue on the work I did here: https://github.com/TayouVR/uDesktopDuplication