Add better HDR handling (+ for HDR color generation)

[FoxyOfJungle]

Is your feature request related to a problem?

Yes. Currently GameMaker is still in its infancy to support HDR correctly. I mean, the application_surface still only has a RGBA8 format and you need to use shaders to make objects emit.

There is also no preparation for color generation in HDR... they are all 8-bit only RGB(255, 255, 255).

Not having these features causes several post-processing effects to make the game look ugly (like Bloom, Godrays, water shine reflections..).

Describe the solution you'd like

Have a function to define whether the application_surface will be created using a texture format other than RGBA8.

For example, there could be a function to change the format of existing surfaces:

surface_format_set(application_surface, surface_rgba16float)

would cause the surface to be recreated with the new format.

---

Furthermore, I would like there to be functions that generate HDR colors, which can be used so that sprites/drawn things have emission.

Intensity basically works like this: To make the sprite glow, I set the Bloom threshold to 1.5 (HDR), and the particle emission to 2. Using a shader:

const float emission = 2.0;

vec4 col_tex = texture2D(gm_BaseTexture, v_vTexcoord) * v_vColour;
gl_FragColor = vec4(col_tex.rgb * emission, col_tex.a); 

Which is how I'm doing it at the moment (using a shader) which is not practical and causes batch breaks...

Something like: make_color_rgb_hdr(red, green, blue, intensity) would be great.

So this could be more simplified if the RGB colors received an intensity variable too, so you could do the same like this (example):

draw_set_color(make_color_rgb_hdr(255, 255, 255, 2));
draw_sprite(sprite, 0, x, y);
draw_set_color(c_white);

(Another option would to have a function like draw_set_emission(intensity)?)

This allows sprites to emit HDR colors, which prevents out-of-range pixels from being hit by post-processing effects like godrays:

The game's art becomes much more interesting with these features.

Describe alternatives you've considered

I currently need to use a custom pipeline that creates a 16-bit RGBA (HDR) surface to be used in a view, which allows me to make sprites recognized by bloom, for example:

Additional context

If you look at games like The Elder Scroll Oblivion, which don't have HDR, Bloom-type effects completely harm the game's image:

A practical example of this happening:

https://github.com/YoYoGames/GameMaker-Feature-Requests/assets/52144406/74b2e2a2-6d3f-41b3-a0b5-5021c4dc9959

In the end I correct Bloom by setting the threshold above 1, which makes the image much better. :)

[all500234765]

So you want an ability to switch format of your app_surface? An app_surface is a surface that will be sent to WDDM, X11, ..., and it can dictate the driver how it should be displayed.

Hence HDR Display Support...

HDR Display support is actually more rather than just setting app surface format to 16 or 10 bits and calling it a day.

There are multiple things to consider before doing anything with display HDR that i'd like to point out.

First. HDR is not about bloom/glow and it never was and never will be. The only thing that connects HDR to bloom or glow is through bad settings menus and poor understanding of this concept in general.

Oblivion case is rather special. It not really a lack of proper HDR. It's lack of sRGB gamma correction.

There are numerous articles about color spaces, gamma correction yada-yada-yada. And they are not wrong. Most of the time and the time when they start to talk about color spaces in general.

So what's the problem here with just flipping the format of the app_surface?

Well, the answer lies in the range of values, color spaces and how you represent bright colors on the Display Screen. I won't bother with detailed explanations here because there is a lot to cover in a single comment under issue i happend to stamble randomly pondering old GM search queries, so here's a quick breakdown in working with HDR Displays: 1) You linearize your textures and colors

• Meaning you apply power(x, 2.2) and pow(x, 2.2) in gml and shaders respectively.
• When you want to send a color to your shader you apply gamma correction (sRGB Gamma to Linear Gamma) power(color_get_red(...) / 255, 2.2)
• When you want to read a texture with color (if it doesn't contain info like how bright your color should be or something like that) pow(texture_color, 2.2) 2) Draw everything to a HDR Buffer (it's either app_surf or something else) 3) Now you have 16bpc buffer with HDR colors in BT. 709 Color Space with Linear Gamma 4.1) If user doesn't have HDR screen see 4.1, otherwise 4.2 4.2) Apply Tonemapping curve, you need to downscale your Hight Dynamic Range into SDR (Standart / Low Dynamic Range) to fit the screen's range. 4.3) Apply HLG or PQ curve, transform from BT. 709 to BT. 2020, call some theoretically new function for the game maker that will tell WDDM, X11, ... that your app_surface is in BT. 2020 PQ or BT. 2020 HLG.

That's sounds so complicated!

That's because it is, if you consider that app_surface to be your main surface to draw objects to. And notice that in steps 4.2 and 4.3 we still had to apply some transformation to the surface, meaning that we'd have to copy it right before the end, not ending this debate.

So what's the actual solution then?

See steps 1-3 and 4.2. The bloom effect should be calculated before 4.2 and applied as a step 5 in the pipeline. And you will have to create a separate HDR surface to draw objects to.

As a personal note here:

RGB, HSL, HSV, YUV, YCbCr - Color Model BT. 2020, BT. 709, sRGB - Color Space BT. 2100 (HLG/PQ), ST. 2084 (PQ), BT. 709, sRGB - Gamma Function

Notice that PQ, sRGB and BT. 709 has multiple entries. That's because those standarts describe both entries (into Color Space and GF).

There is actually more to all of this, but i am not going to elaborate since all of this is already quite boring to grasp.

For some starters there's some really good explanation by Acerola on part of this topic: https://www.youtube.com/watch?v=fv-wlo8yVhk

And here's explanation for the pipeline with ACES tonemapper: https://docs.nvidia.com/gameworks/content/devices/shield-hdr-dev-guide/hdr-dev-guide-nvidia-shield.htm

You can find some ready to use pieces of code online, just make sure they contain some conversion matrices and they output in BT. 709 (for 8 bit surface) or BT. 2020 (for 10+bit surface)

Source of arcane knowledge:

Lot's of samples, docs, research, lot's of curiosity and motivating rage to close this internal debate whenever HDR it self is worth it or not, and what it takes to implement correct HDR Display support.

Through grading in HDR is whole another topic of it self. https://www.youtube.com/watch?v=bYS-P2bF2TQ

[FoxyOfJungle]

I appreciate the long answer, while you are right about some things, it appears you misunderstood the topic of the subject being addressed.

First of all, GameMaker appears to have no Color Space management, everything is linear as far as I know. And even if it did, converting from linear space to gamma space has no direct relation to what I mentioned in the original topic.

My goal is not compatibility with HDR monitors, but features that facilitate the use of 16-bit float format textures, in which they can emit pixels above 1. For this, I can give you an example of the materials in 3D rendering:

I want to be able to make the pixels have an exposure above 1, so that Bloom detects this through a threshold, in which it will emit glow only in those pixels. GameMaker currently does not allow this with the current 8-bit application_surface.

Obviously I would use tone mapping to map the values to LDR, so as not to get unwanted visuals. I did this already with my Post-Processing project.

For this, the monitor being HDR or LDR, would not influence the objective. This is something that is rendered within the GM application and has no relation to the monitor (since the output should be always Low Range: 0 - 1).

The application_surface having more pixel precision doesn't seem like something problematic to implement, since we already have the texture formats currently and I emulated this, it's just not very practical to do it without it...

Please take into consideration that this would be an optional feature and not mandatory. I believe that many games that require post-processing effects would benefit from this feature.

For most games, application_surface being 8-bit seems sufficient. But to obtain special effects that require a more precise texture format, it is extremely useful for the surface to allow this.

For example, I did this below, for a materials system. Since I use 16-bit float textures for this:

https://github.com/YoYoGames/GameMaker-Feature-Requests/assets/52144406/3caaeb9f-07ad-430e-af61-a2d236974a9e

Note that I had to create a custom surface/texture and draw the models on it, manually. If it were possible to change the application_surface format, this would not be necessary, achieving great performance. The same is true for 2D games.

Regarding the Oblivion game, it's not really a gamma correction issue (actually that would help, but not at all). The Bloom effect simply does not have a threshold with a larger scale (above 1) so this is a limitation because it only reaches certain pixels, so it ends up reaching the light pixels in LDR, which means that if a character wears white clothing, it will shine like the sky, which is also very bright. That's why the image looks like this crap.

I know Acerola's channel and coincidentally I've seen this video xD... I spent a few months studying HDR... to be able to implement this in my asset, which also uses ACES, so I learned a lot about how to adapt this in GameMaker. My goal is for me to have better control over the the application_surface.

[all500234765]

I understood your in the first post already. Please re-read my answer, specifically So what's the actual solution then? part, this is what you actually want to do.

Making application_surface 10+ bit won't do what you actually want to accomplish here, as it will require making another application_surface from GM side (i.e. application_surface_present or something), so you will endup with the same thing as i said earlier.

TL; DR From what i know, GM can't do anything about making app_surface with higher bitness, as it will require color managment. The only way is to use separate HDR surface to which you will draw as an intermediate step. There is no other way as far as I'm aware of.

[FoxyOfJungle]

I was counting on the fact that GM's final rendering has output up to 1, but actually this may not be the case then. I remember that I had to adapt the Post-Processing FX to work correctly in HDR (avoid popping colors and things like that).

Yeah, it would be necessary to create another application_surface (which could be referenced by the same constant name).

[all500234765]

referencing 2 different things would be confusing i presume.