Viewport lost precision on HDR values

[danilw]

Godot version: 3.1.1 stable

OS/device including version: Linux/Windows

Issue description: Godot viewport is GL_RGBA16F with GL_HALF_FLOAT 0xffff should work in viewport, but set even 11111 become 11104 (look screenshot)

I do not understand where it come from, I want edit it to make it work "correctly", but even editing Godot source code(C++/shaders) does not fix it for me...

Steps to reproduce: set shader output color to vec4(65534.,11111.5,65533.,65532.); and print its result in other shader

Minimal reproduction project: https://danilw.github.io/godot-utils-and-other/FBO_test/FBO_test2.zip

[danilw]

Update: I dump shader(with large 32-bit values) look like something change const numbers to this vec4 m_tc=vec4((1.67772e+07/2.0), 8.38861e+06, 65535.6, 1.67772e+07); when original this line is vec4 tc=vec4((16777215./2.),8388607.5,65535.55,16777212.); this give me result that I see (on 32bit float test) (1.67772e+07/2.0)!=8.38861e+06 when original is equal (16777215./2.)==8388607.5

[danilw]

as I understand it come from shader_compiler_gles3.cpp because I know that Nvidia GLSL compiled do not change values like that (I launch shader in other launcher, it work correctly)

so how to dodge it in godot without editing source of shader_compiler (for example code that linked in first post)

int val=65535+min(iFrame,0); (this should work in all cases)

I test that simply int val=65535; also work

then COLOR=vec4(float(val),float(val),float(val),float(val)); will save correct 65535 value instead of something broken....

I think this is bug

[Zylann]

Aren't half types unable to accurately represent numbers such as 11111?

[danilw]

this is not because half or medium

godot GLSL pre-compiler do that (I think, and as I saw in my tests)

[randomness128]

The viewport is 16-bit floating point. The precision you're getting is exactly what is expected. https://en.wikipedia.org/wiki/Half-precision_floating-point_format

[danilw]

The viewport is 16-bit floating point. The precision you're getting is exactly what is expected.

sorry, this is not true download my example project that has vec4(65534.,11111.5,65533.,65532.); and output like on screenshot 65534 become 65504

then add this code int tmp=65534; and edit line with vec4 to vec4(float(tmp),11111.5,65533.,65532.);

output on screen will be 65534 as you send it ....so

[randomness128]

If the viewport is GL_RGBA16F, putting 65534 in it is impossible. 65534 should be rounded to either 65504 or infinity. If you're reading 65534, it either didn't come from the viewport or the viewport isn't GL_RGBA16F.

[danilw]

you right at this point look like I did test it on modified Godot with GL_RGBA32F then more correct about this bug is - float const 0xffffff rounded to '1.67772e+07' when used GL_RGBA32F viewport, if put 0xffffff as int const it will work and written to viewport output

[clayjohn]

As per @randomness128 comment above. The maximum allowable floating point number using half precision is 65504, so it is only natural numbers higher than that cannot be represented. This is not a Godot bug nor an error in the graphics API.