[Enhancement] [Bug] Doesn't work with HDR

[Jugbot]

With HDR on, the image recognition simply doesn't work as well, often skipping artifacts and weapons. Would there be a way to automatically convert the image quality to something easier to read?

Display used: 4k @ 120Hz

[Cupcak3]

While HDR might seem a relevant clue in this, I suspect it is not. Another user reported rows/columns of items being skipped. I think I've tracked this bug down to being too strict of parameters for finding all the items in an inventory page. Your issue sounds similar to their problem. V1.1.1 prerelease hopefully fixes this issue.

[Jugbot]

Strange considering I've tried it on two displays, one with HDR enabled and another without. Another clue is that the screenshot in the program looks washed out:

[Jugbot]

This is Windows auto HDR btw (since genshin does not natively support it)

[Jugbot]

I turned off auto HDR and it worked so its definitely the increased bit depth that is messing with the scanning.

[Cupcak3]

Ah I see. Thanks for the screenshot. None of my monitors support HDR so I cannot work with this outside of theory crafting. I'm also not an expert in image processing with all it's formats and whatnot but at a cursory glance we're capturing all our screengrabs into 24bpp RGB formatted objects. Why that format? Honestly I don't even know but this might be what causes things to break when HDR is enabled. I'll leave this issue open for you or anyone else has the equipment and expertise to debug this.

[Jugbot]

https://docs.microsoft.com/en-us/windows/uwp/audio-video-camera/screen-capture

On systems with Windows HD color enabled, the content pixel format might not necessarily be DXGI_FORMAT_B8G8R8A8_UNORM. To avoid pixel overclipping (i.e. the captured content looks washed out) when capturing HDR content, consider using DXGI_FORMAT_R16G16B16A16_FLOAT for every component in the capturing pipeline, including the Direct3D11CaptureFramePool, the target destination such as CanvasBitmap. Depends on the need, additional processing such as saving to HDR content format or HDR-to-SDR tone mapping might be required. This article will focus on SDR content capturing. For more information, please see Using DirectX with high dynamic range Displays and Advanced Color.

Since I doubt tesseract is set up to handle 16bit image data, it looks like we would need tone mapping...

[Cupcak3]

Thanks for the article and snippet. It would appear that the format truncation is the source of the screenshots being washed out.

Like I mentioned before, I do not have an HDR capable monitor so I can't work on it much directly myself. I can, however, reorganize some of the code to be more unit test friendly to make for faster testing, but for this issue in particular I would need HDR-enabled screenshots of the various stages where we capture the screen.

[Jugbot]

I can take a look at fixing this myself, since it would be quite the hassle to get all those images to you.

As for a solution, here is a snippet that uses AForge.net for bit conversion https://github.com/selenur/ToolsGenGkode/blob/bc80dbed1b31f64f5b039292f2eb7ebfb87de73f/AForgeNetLib/Controls/PictureBox.cs

And here is the actual conversion function https://github.com/selenur/ToolsGenGkode/blob/bc80dbed1b31f64f5b039292f2eb7ebfb87de73f/AForgeNetLib/Imaging/Image.cs#L352

I have not set up a C# workspace in a while, so it will probably take me more than a weekend to test this out 😢

[Jugbot]

Or if you want, you can send me the exe and I can tell you if it works 😄

[Jugbot]

Ok so after a deep dive I think I am going to close this issue... it seems that windows' api does not give the option to screencap at anything beyond 8 bit channels (in my case 10bit), and there is no way to control the conversion. I can't seem to find a single program that exists that can screenshot in HDR except windows game bar 🪟 + ALT + PrtSc. Even the windows snipping tool cant do this properly 🪟 + Shift + S.

All hope is not lost however, as other HDR programs (such as chrome) are converted to 8bitRGB correctly, so this might be a bug with AutoHDR (which is still fairly new).

[Cupcak3]

So I've looked into it just a bit trying to understand what can be done about this. I'm not super familiar with pixel formats, especially not for HDR images. From your article excerpt DXGI_FORMAT_R16G16B16A16_FLOAT might be something like PixelFormat.Format48bppRgb as described in .NET's image format api which we are using for screen captures. If they are indeed analogs then changing to that pixel format won't be too hard to do.

The next problem we'd need to asses is making sure that we're using acceptable image formats for various Accord image filters that we use. Some of our image processing methods do not use Accord filters and they will probably need to be rewritten to use Accord instead. There are a number that we use and they do not all accept the same images with respect to image formats. 48bpp color images are likely not going to work with a majority of those filters. The grayscale filter does accept 48bpp color images and is a key step in our image processing so it may work as a good point for image format conversions.

Let me know what you think.