Color banding artifacts make atmosphere ugly in low light

[10110111]

See the following screenshot in 1:1 scale to understand the problem:

This banding can be smoothed by application of dithering. I've done a quick-and-dirty implementation of it just to see which places would need it, and below is a proof-of-concept patch I've come up with. I'm sure it can be improved, including a switch to toggle it on/off as well as enable/disable 6 BPP monitor support (which is currently an #ifdef). Also it might be better to simply do all the rendering to a higher-precision FBO texture (e.g. with GL_RGBA32F format) and then simply dither this render target when blitting to screen. But at least this patch makes the image much nicer (see the screenshot after the diff).

diff --git a/src/core/StelPainter.cpp b/src/core/StelPainter.cpp
index 529407e826..f3d66e5cc3 100644
--- a/src/core/StelPainter.cpp
+++ b/src/core/StelPainter.cpp
@@ -2027,12 +2027,36 @@ void StelPainter::initGLShaders()

    QOpenGLShader fshader2(QOpenGLShader::Fragment);
    const char *fsrc2 =
+"#version 120\n"
+"vec3 dither(vec3 c)\n"
+"{\n"
+"    const float bayerPattern[] = float[](\n"
+"        0,  32,  8, 40,  2, 34, 10, 42,  /* 8x8 Bayer ordered dithering  */\n"
+"        48, 16, 56, 24, 50, 18, 58, 26,  /* pattern.  Each input pixel   */\n"
+"        12, 44,  4, 36, 14, 46,  6, 38,  /* is scaled to the 0..63 range */\n"
+"        60, 28, 52, 20, 62, 30, 54, 22,  /* before looking in this table */\n"
+"        3,  35, 11, 43,  1, 33,  9, 41,  /* to determine the action.     */\n"
+"        51, 19, 59, 27, 49, 17, 57, 25,\n"
+"        15, 47,  7, 39, 13, 45,  5, 37,\n"
+"        63, 31, 55, 23, 61, 29, 53, 21);\n"
+"    float bayer=bayerPattern[int(mod(int(gl_FragCoord.x),8)+8*mod(int(gl_FragCoord.y),8))]/64;\n"
+#ifdef MONITOR_6_BPP
+        "    const float rgbByteMax=63.;\n"
+#else
+        "    const float rgbByteMax=255.;\n"
+#endif
+"    vec3 rgb=c*rgbByteMax;\n"
+"    vec3 head=floor(rgb);\n"
+"    vec3 tail=fract(rgb);\n"
+"    return (head+step(bayer,tail))/rgbByteMax;\n"
+"}\n"
+"vec4 dither(vec4 c) { return vec4(dither(c.xyz),c.w); }\n"
        "varying mediump vec2 texc;\n"
        "uniform sampler2D tex;\n"
        "uniform mediump vec4 texColor;\n"
        "void main(void)\n"
        "{\n"
-       "    gl_FragColor = texture2D(tex, texc)*texColor;\n"
+       "    gl_FragColor = dither(texture2D(tex, texc)*texColor);\n"
        "}\n";
    fshader2.compileSourceCode(fsrc2);
    if (!fshader2.log().isEmpty()) { qWarning() << "StelPainter: Warnings while compiling fshader2: " << fshader2.log(); }
@@ -2067,12 +2091,36 @@ void StelPainter::initGLShaders()

    QOpenGLShader fshader4(QOpenGLShader::Fragment);
    const char *fsrc4 =
+"#version 120\n"
+"vec3 dither(vec3 c)\n"
+"{\n"
+"    const float bayerPattern[] = float[](\n"
+"        0,  32,  8, 40,  2, 34, 10, 42,  /* 8x8 Bayer ordered dithering  */\n"
+"        48, 16, 56, 24, 50, 18, 58, 26,  /* pattern.  Each input pixel   */\n"
+"        12, 44,  4, 36, 14, 46,  6, 38,  /* is scaled to the 0..63 range */\n"
+"        60, 28, 52, 20, 62, 30, 54, 22,  /* before looking in this table */\n"
+"        3,  35, 11, 43,  1, 33,  9, 41,  /* to determine the action.     */\n"
+"        51, 19, 59, 27, 49, 17, 57, 25,\n"
+"        15, 47,  7, 39, 13, 45,  5, 37,\n"
+"        63, 31, 55, 23, 61, 29, 53, 21);\n"
+"    float bayer=bayerPattern[int(mod(int(gl_FragCoord.x),8)+8*mod(int(gl_FragCoord.y),8))]/64;\n"
+#ifdef MONITOR_6_BPP
+        "    const float rgbByteMax=63.;\n"
+#else
+        "    const float rgbByteMax=255.;\n"
+#endif
+"    vec3 rgb=c*rgbByteMax;\n"
+"    vec3 head=floor(rgb);\n"
+"    vec3 tail=fract(rgb);\n"
+"    return (head+step(bayer,tail))/rgbByteMax;\n"
+"}\n"
+"vec4 dither(vec4 c) { return vec4(dither(c.xyz),c.w); }\n"
        "varying mediump vec2 texc;\n"
        "varying mediump vec4 outColor;\n"
        "uniform sampler2D tex;\n"
        "void main(void)\n"
        "{\n"
-       "    gl_FragColor = texture2D(tex, texc)*outColor;\n"
+       "    gl_FragColor = dither(texture2D(tex, texc)*outColor);\n"
        "}\n";
    fshader4.compileSourceCode(fsrc4);
    if (!fshader4.log().isEmpty()) { qWarning() << "StelPainter: Warnings while compiling fshader4: " << fshader4.log(); }
diff --git a/src/core/modules/Atmosphere.cpp b/src/core/modules/Atmosphere.cpp
index 12953ed9ed..20798f554f 100644
--- a/src/core/modules/Atmosphere.cpp
+++ b/src/core/modules/Atmosphere.cpp
@@ -58,10 +58,34 @@ Atmosphere::Atmosphere(void)
    }
    QOpenGLShader fShader(QOpenGLShader::Fragment);
    if (!fShader.compileSourceCode(
+"#version 120\n"
+"vec3 dither(vec3 c)\n"
+"{\n"
+"    const float bayerPattern[] = float[](\n"
+"        0,  32,  8, 40,  2, 34, 10, 42,  /* 8x8 Bayer ordered dithering  */\n"
+"        48, 16, 56, 24, 50, 18, 58, 26,  /* pattern.  Each input pixel   */\n"
+"        12, 44,  4, 36, 14, 46,  6, 38,  /* is scaled to the 0..63 range */\n"
+"        60, 28, 52, 20, 62, 30, 54, 22,  /* before looking in this table */\n"
+"        3,  35, 11, 43,  1, 33,  9, 41,  /* to determine the action.     */\n"
+"        51, 19, 59, 27, 49, 17, 57, 25,\n"
+"        15, 47,  7, 39, 13, 45,  5, 37,\n"
+"        63, 31, 55, 23, 61, 29, 53, 21);\n"
+"    float bayer=bayerPattern[int(mod(int(gl_FragCoord.x),8)+8*mod(int(gl_FragCoord.y),8))]/64;\n"
+#ifdef MONITOR_6_BPP
+        "    const float rgbByteMax=63.;\n"
+#else
+        "    const float rgbByteMax=255.;\n"
+#endif
+"    vec3 rgb=c*rgbByteMax;\n"
+"    vec3 head=floor(rgb);\n"
+"    vec3 tail=fract(rgb);\n"
+"    return (head+step(bayer,tail))/rgbByteMax;\n"
+"}\n"
+"vec4 dither(vec4 c) { return vec4(dither(c.xyz),c.w); }\n"
                    "varying mediump vec3 resultSkyColor;\n"
                    "void main()\n"
                    "{\n"
-                    "   gl_FragColor = vec4(resultSkyColor, 1.);\n"
+                    "   gl_FragColor = vec4(dither(resultSkyColor), 1.);\n"
                     "}"))
    {
        qFatal("Error while compiling atmosphere fragment shader: %s", fShader.log().toLatin1().constData());

Screenshot of the same scene with the above patch:

[10110111]

Interestingly, on the RasPi3, the final image seems OK. No brightened texture squares:

But no dithering either: see the flat magenta-ish and greenish bands e.g. near the head of the fish. This is also confirmed by the fact that Bayer texture appears black.

[gzotti]

4 spaces, OK. I tried two backticks or up to 3 spaces... Still new here.

The shader does "something" on RasPi3 with the various modes, but no useful pattern. Here are 666, 888 and off: 888 is pretty good, the others show banding. Testing with that envvar next...

[10110111]

One more thing to try: in makeBayerPatternTexture, replace GL_RED with GL_LUMINANCE in both 4th and 7th arguments of glTexImage2D call. According to the ES 2.0 docs, GL_RED may not be acceptable in OpenGL ES 2.0.

[gzotti]

Sigh, OpenGL things are still not too flexible:

QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled

[10110111]

QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled

On which system/config did you get this? I remember getting it when Qt simply wasn't able to link to libEGL.so.

[gzotti]

System as described in our Wiki, https://github.com/Stellarium/stellarium/wiki/Raspberry-Pi

Driver version string: "OpenGL ES 2.0 Mesa 17.4.0-devel (git-7983adc60f)"
GL vendor is "Broadcom"
GL renderer is "VC4 V3D 2.1"
GL Shading Language version is "OpenGL ES GLSL ES 1.0.16"
MESA Version Number detected:  17.4

In any case, software OpenGL will have again 0.05 fps, so I don't think it's worth to start fixing that. Tried with GL_LUMINANCE, again vec3(bayer) is black.

I will now turn back to my other Windows systems with (hopefully) non Qt5.9, i.e. different ANGLE.

[gzotti]

Stupid, wih Qt5.10 we have #139.

[10110111]

One more thing to try on Mesa: set MESA_DEBUG=1 environment variable and look for any error messages emitted by Mesa in the console.

[gzotti]

I now tried with my netbook. AMD A4-1200 Radeon HD 8180, Qt5.9, Win10. This works reasonably well in OpenGL (4.5!), is unusable (GUI panels missing??) with ANGLE/DirectX9, but works well again with ANGLE/DirectX11. No texture squares, and patterned dither 565&666, smooth 888. Maybe it's the ANGLE D3D9 that is broken for Geforce and AMD GPUs when they can run OpenGL and D3D11? Need to test 2 more...

[gzotti]

OK, we have for the "return toColor(bayer).rgb":

Detected: OpenGL ES "2.0"
Driver version string: "OpenGL ES 2.0 (ANGLE 2.1.0.8613f4946861)"
GL vendor is "Google Inc."
GL renderer is "ANGLE (NVIDIA GeForce GTX 960M Direct3D9Ex vs_3_0 ps_3_0)"
GL Shading Language version is "OpenGL ES GLSL ES 1.00 (ANGLE 2.1.0.8613f4946861)"
VS Version Number detected:  3
PS Version Number detected:  3

Black screen.

And

Detected: OpenGL ES "2.0"
Driver version string: "OpenGL ES 2.0 (ANGLE 2.1.0.8613f4946861)"
GL vendor is "Google Inc."
GL renderer is "ANGLE (NVIDIA GeForce GTX 960M Direct3D11 vs_5_0 ps_5_0)"
GL Shading Language version is "OpenGL ES GLSL ES 1.00 (ANGLE 2.1.0.8613f4946861)"
VS Version Number detected:  5
PS Version Number detected:  5

compare with OpenGL:

Detected: OpenGL "4.5"
Driver version string: "4.5.0 NVIDIA 385.41"
GL vendor is "NVIDIA Corporation"
GL renderer is "GeForce GTX 960M/PCIe/SSE2"
GL Shading Language version is "4.50 NVIDIA"
GLSL Version Number detected:  4.5

(Don't bother the high resolution with GUI bar inside, I am playing with #102 as well.) Changing GL_RED to GL_LUMINANCE does not change the result. It's blue on OpenGL but green with D3D11.

I deactivated all the preliminary returns now, and ... errm.. whatever happened in the past 10 days, the colored texture boxes also have vanished in D3D9 mode. Maybe there was one mediump earlier which I then changed to highp, and now it works? But the black screen still seems wrong. And 888 dither pattern looks worse than undithered, at least for the constellation artwork. With Zodiacal light, on my notebook I see soft colored stripes. 666 is overall better.

This is turning crazy. Now I have an Intel HD4700 to test...

[gzotti]

OK, it's an Intel 4600, Qt5.9 like on my Win10 notebooks: OpenGL (4.3): Works. ANGLE/DirectX9: Terrible banding when set to 565 ANGLE/DirectX11: Works indistinguishable from OpenGL.

Maybe we should just conclude "Dithering may not work well on Angle/DirectX9" (and again, "update your drivers"...) After all, switching off looks like before.

[gzotti]

Thanks for fixing #139. On that Win7 notebook (Geforce580, Qt5.10) I now have similar results now like with the Intel GPU: OpenGL 4.5 and DX11 are fine, DX9 shows ugliest banding. No texture squares, though.

I think we can stop here and better turn to other issues. Objections?

[10110111]

If you can provide a Windows binary of current Stellarium git, I might try to find time to see what happens myself. Not sure though how you differentiate Qt versions – are they included with Stellarium? Or do you somehow have them installed system-wide?

In any case, if this works in normal cases (i.e. default mode, when the users don't turn on any workarounds which ANGLE seems to be), I'm fine with closing this issue.

[gzotti]

Alex makes weekly betas, available from Launchpad: https://launchpad.net/stellarium The Qt libraries are packed in the program directory.

The whole ANGLE business is a fallback for unfortunately far too many "average" users who are not aware of the fact that their GPUs need current drivers from other sources than the daily Windows update. It feels like about 40% support requests still relate to that nonsense, and we keep repeating ourselves in the Release FAQs. Some GPUs are meanwhile blacklisted by Qt because their OpenGL drivers are bad (e.g. early Intel HD-2000/3000 always start ANGLE mode, and probably even earlier Intels always would use MESA (software GL), although I don't have access to such old systems now). For others, users should explicitly launch ANGLE mode with the provided link, or ... update their drivers.

OK, we may close here, and if you are able to detect what's wrong (some rounding/ULP issue, most likely), please just call again!

[gzotti]

@10110111 unfortunately it seems this feature caused a bad memory leak! @guillaumechereau found:

OK, I did a long run with valgrind massif, and it seems the leak might
come from the call to 'makeBayerPatternTexture' in
StelPainter::drawFromArray.

Also, dithering on Odroid XU3 and C1 SBCs (OpenGL ES2) does not really work, has similar increased banding as DX9 ANGLE.

[10110111]

I suppose it's textures which leak. I supposed StelPainter only dies when GL context is destroyed. But it appears to be created and destroyed at every frame. I'll make a pull request to fix this one.

Also, dithering on Odroid XU3 and C1 SBCs (OpenGL ES2) does not really work, has similar increased banding as DX9 ANGLE.

Issue #131 has grown to a hard-to-manage length. Would it be better to track new problems as new issues? (You can always ping me there via the @ syntax.)

[gzotti]

Agreed. Thanks for the fast fix. And I may open a new issue about the GLES dithering stuff.

[danifernandez5]

hi, apologies for reviving this. Im with a 6bpp monitor and the band problem happens to me. Is very annoying. How can I easily solve? Im using the latest beta version and it appears in all options (table mode, angle mode, etc). gpu amd radeon rx vega 10; amd ryzen 7 3700u cpu, drivers 20.4.1; windows 10 64 bit.

[10110111]

@danifernandez5 By default dithering is disabled, so it's expected that you see banding. To get rid of banding, knowing that you have a 6bpp monitor, go to Configuration (F2) → Tools → Dithering. Choose 6/6/6 bits in this drop-down menu.

[danifernandez5]

Thank you very much! It worked perfect on 8/8/8. It's weird because my operating system says the color depth is 6 bit. Thanks again.