several shaders show artifacts in glcore

[LazyBumHorse]

I'm actually not sure if this is a shader problem or a RetroArch video driver (i.e. Slang shader translation) problem, but here's what's happening:

Using the glcore video driver, xbr-mlv4-multipass shows artifacts: It's fine in Vulkan: Here's glcore again: And here's the GLSL version in gl for another comparison:

I'm using a GTX 1060 6GB.

This may be related to #70 as well, though Vulkan is innocent this time.

[hizzlekizzle]

Looks like a glslang issue. Other reflections seem to be okay.

[LazyBumHorse]

Now that I know what to look for, I found several shaders that seem to be affected by this as well:

• xbr-lv3-multipass
• All hqx shaders
• aa-shader-4.0 (and probably -level2)
• scalefx has stray pixels similar to xbr-lv3-multipass

Can we break it down what they have in common? Maybe there's already an issue in the glslang's issue tracker.

[hizzlekizzle]

I'm not getting any issues with xbr-mlv4-multipass, scalefx or hqx, but I am getting rough edges on xbr-lv3-multipass. Using glcore with mesa driver on a Haswell integrated GPU.

[LazyBumHorse]

Do you mean this when you say rough edges in xbr-lv3-multipass? Noticeable at the font the most, I don't seem to get issues with the single pass xbr-lv3 version:

And are you sure you don't have any issues with the other examples? They aren't very pronounced for me but in direct screenshot comparison to glsl it becomes very obvious something is wrong, e.g. at first glance hq2x looks okay until you notice the 'ghosting' of the font and then the fact it doesn't smooth the pixels at all: vs glsl

scalefx is also very very subtle, here's another (upscaled) comparison, slang in glcore vs glsl in gl: Notice e.g. the darker lines between the gray bricks besides the king. vs glsl:

I want to try debugging it myself, but I don't even know how Slang or glslang you mentioned work nor what reflections precisely are.

[hizzlekizzle]

Here's what lv3-multipass looks like for me: hq4x looks okay here: The way slang works is it gets converted to SPIR-V, which then gets "reflected" back to whatever driver's native code via glslang. So, there are several places where inconsistency can sneak in, whether it's the normal endpoint driver issues we always have to deal with when shaders are involved, but also issues with glslang's reflection (rare, but it can happen).

[LazyBumHorse]

I think I found the issue! It was my graphics driver settings all along. I forced 16x anisotropic filtering in all programs, which surprisingly messes up the shaders in glcore. Now all the named shaders behave like they should.

Well, almost. aa-shader-4.0 and xbr-lv4-multipass still show some line artifacts. aa-shader-4.0 shows a rough line every 32 pixels (at least in the bottom half): xbr-lv4-multipass has seemingly a single rough line going through the bottom characters:

[LazyBumHorse]

I found the xbr-lv4-multipass issue so weird so I did some more testing. It appears to really be only one singular line artifact and it does appear in Snes9x glcore and vulkan (didn't know you could use vulkan with it, but apparently you can). I also couldn't replicate the issue in cores for other systems and I believe that is because of differing vertical resolutions. In higan you can set different internal resolutions and only if the vertical resolution is 224 (like in Snes9x), the single rough line appears. Maybe some kind of precision error? Rounding error? Off-by-one error? Compared to glsl, the 'slicing' appears to be different, you see what I mean when you switch between the images: glsl: I should have activated integer scale... Interestingly the artifact disappears when integer scaling, the horizontal 'slicing' still seems to be different though, which is weird - different rounding?

So a workaround would be to set the scaling of the last pass to a clean 4x or 5x instead of 'Don't care'.

---

The aa-shader-4.0 issue appears in glcore and vulkan and also seems to be dependent on the vertical resolution. 224 and 240 have issues, but 448 and 480 appear fine. Some other cores/resolution I tested appeared fine as well.

I thought this had something to do with the use of scale_type(...) = "viewport" and nearest neighbor scaling because 224 and 240 don't divide 1080 (and maybe the issue is just slightly hidden with 448 and 480), but using integer scale or setting scale_type to source and scale to 4x, the lines are still there.

Looking closer: glsl: This looks like an off-by-one error, all lines appear to shift down by 1 in Slang, except for every 32th (in 1080p) or 28th (with integer scaling, 896'p') line.

[hizzlekizzle]

yeah, rounding errors like that are very common. I usually deal with them by multiplying the texcoord calculation in the vertex by something like 1.0001 but I believe the issue is also GPU-dependent, so sometimes 1.0001 does it, other times I have to go up to 1.0004 or somesuch.

[LazyBumHorse]

Hm... that sucks. I guess you just can't expect pixel perfect output in every case, but I didn't think it would be that bad.

What's just so weird to me is that the code for the aa-shader-4.0 slang and glsl shaders are actually identical, except glsl is using precision qualifiers for OpenGL ES, which I'm not using, so...

Oh well, the original problem has been solved, xbr-lv4-multipass has a workaround and aa-shader-4.0 seems to only break in that particular resolution on my particular system in this particular universe, so the topic can be closed.

[hizzlekizzle]

I spotted another common culprit for rounding errors in aa-shader-4.0: floor(). Adding a small offset to it appears to have cleaned it up quite a bit! https://github.com/libretro/slang-shaders/commit/79c95a5036af4a38138dc8f924445a168eee4e54

[LazyBumHorse]

That works very well! The difference image is almost completely black. Only around the very top, there's still one line artifact, probably because the effect of the multiplication is too low at that point.

How it should look like: