BUG: Some shaders not applying correctly in "hybrid" screen view (nds emulation)

[candid-crow]

pixel art scaling(?) shaders don't apply correctly in "Hybrid" screen view for nds emulation

apologies if I've messed up the formatting. i've attached screenshots showing some examples of the bug.

I ran into this issue initially on my steam deck (gaming mode/native linux; retroarch 1.16.0; MelonDS (0.9.3)) but have since replicated it on my windows 10 desktop (retroarch 1.16.0; bug persists across cores MelonDS (0.9.3) as well as DeSmuME (git 4ee1bb1) as well as retroarch 1.10.3 out of curiosity).

...that is to say, i assume this is a retroarch bug because the problem persists regardless of which specific core is being used.

Description

When using any variation of the "hybrid" screen layout for nds emulation, some shaders do not apply correctly to all screens. When using any other screen layout (top only, bottom only, left/right, right/left, etc.) the shaders apply correctly.

Expected behavior

I expected that the loaded shader would apply to all (or both) screens when using the hybrid screen layout.

Actual behavior

For many shaders (not all) the loaded shader applies to the small screen(or screens, if "Hybrid Small Screen Mode" is set to "Duplicate") but not to the large screen. this occurs when using the MelonDS core, as well as the DeSmuME core.

For the DeSmuME core there is a way to make the shader apply to all screens, (setting "Hybrid Layout: Scale So Small Screen is 1:1px" to "OFF;") but this causes it to break on the small screens. Otherwise, the behavior is identical to the MelonDS core.

Steps to reproduce the bug

1. Launch RetroArch
2. in the retroarch menu settings/video/output/video set either gl or vulkan
3. Load Core: load one of the MelonDS or DeSmuME cores
4. Load Content: load an nds rom
5. load "scalefx-aa-fast.slangp" from shaders_slang/presets if using vulkan, or "scalefx-aa-fast.glslp" from shaders_glsl/presets if using gl. (the exact shader doesn't have to be only this, but this is a shader that exists in both formats, and the bug is present for this shader.)
6. set screen layout in the core options to "Hybrid Top" or "Hybrid Bottom"
7. observe that the shader applies to the small screen(s), but not the large screen.

Bisect Results

as far as i can tell as someone who is not terribly knowledgeable regarding shaders, the common element of shaders which do not apply correctly seems to be any that use pixel art scaling, notably anything with "xbr," "xbrz," or "scalefx."

for example, "lcd-grid-v2-nds-color.glslp" and "median3x3.glslp" apply correctly to all screens, but "fast-bilateral-super-xbr.glslp" and "scalefx.glslp" don't.

I first noticed the bug on my steam deck, as on my desktop I've never had a reason to use the hybrid screen layout.

After replicating the issue on my Windows 10 PC and testing across different cores, I also tested an older version of RetroArch (1.10.3) which I happened to have installed and found the same bug to be present.

I'm not sure if this is a known issue; I searched the open and closed issues for any mentions of "hybrid" and couldn't find anything relevant, but I find it hard to believe that I'm the first person to encounter this—maybe it's gone unnoticed until now because it only applies to a subset of shaders, and only in the hybrid screen layout?

regardless, it seems like it's been present as an issue for some time, though i only narrowed down the exact problem (that is, specifically a subset of shaders and only in the hybrid screen layout) as i was writing this and doing further testing.

Version/Commit

• RetroArch: 1.16.0/Git Version 6c2cc45

Environment information

• OS: Windows 10 Pro; SteamOS Holo (Stable Update Channel)

Screenshots

[hizzlekizzle]

Unfortunately, this is not a bug, which makes it difficult to "fix". When you use hybrid view, it scales one of the screens up or down, and the shaders you've listed all have a common mechanism: they look at a pixel and its surrounding neighbors and try to identify patterns based on what they find. Their pattern-recognition no longer works once the original pixel has been scaled to the size of 2 or 4 or whatever actual pixels.

I made a couple of "ds-hybrid-view" shaders (one for the simple-but-nice 'sabr' shader and one for the much more complex 'scalefx' shader) located in the 'handheld' directory that will take the default NDS stacked image and do a fake hybrid view with the scaler applied on the big image.

[candid-crow]

Unfortunately, this is not a bug, which makes it difficult to "fix". When you use hybrid view, it scales one of the screens up or down, and the shaders you've listed all have a common mechanism: they look at a pixel and its surrounding neighbors and try to identify patterns based on what they find. Their pattern-recognition no longer works once the original pixel has been scaled to the size of 2 or 4 or whatever actual pixels.

I made a couple of "ds-hybrid-view" shaders (one for the simple-but-nice 'sabr' shader and one for the much more complex 'scalefx' shader) located in the 'handheld' directory that will take the default NDS stacked image and do a fake hybrid view with the scaler applied on the big image.

oooooohkay, gotcha. i was wondering if it was strictly a "bug" or more a consequence of the way the shaders worked, so thank you for explaining it.

...out of curiosity, is this (that is, needing to use a work-around for hybrid view) documented somewhere that i missed? i found the description of the shaders you mentioned here on github, but did I miss a mention somewhere that you need to do it this way?

in any case: thank you for creating the "ds-hybrid-view" shader; i was happily surprised to find it even works with integer scaling, so, kudos.

i ended up appending the base "ds-hybrid-view" shader to my existing shader configuration, and now that i have the aspect ratios and such configured correctly, that works!

side note: ik this isn't a help forum and i apologize if this is pushing the boundaries of that, but since you're the creator of the shader, i wanted to ask: is there a way to fix the touch/stylus functionality? currently it's very misaligned. i looked breifly for settings that bounded where the input was, but i couldn't find anything. i imagine it might just be the cost of simulating hybrid view this way, but i wanted to ask in case it was a solved problem. anyway, thank you again!

[hizzlekizzle]

Correct, it's a result of the shader warping the image and, unfortunately, there's nothing to be done about it. The shader system and the core's touch coordinate system are completely separate with no way to communicate what they're doing to the other end.

[candid-crow]

Correct, it's a result of the shader warping the image and, unfortunately, there's nothing to be done about it. The shader system and the core's touch coordinate system are completely separate with no way to communicate what they're doing to the other end.

thank you for confirming that. i was pretty sure, but i figured that you could say for certain one way or another.

[Luciofrenetico]

Unfortunately, this is not a bug, which makes it difficult to "fix". When you use hybrid view, it scales one of the screens up or down, and the shaders you've listed all have a common mechanism: they look at a pixel and its surrounding neighbors and try to identify patterns based on what they find. Their pattern-recognition no longer works once the original pixel has been scaled to the size of 2 or 4 or whatever actual pixels.

I made a couple of "ds-hybrid-view" shaders (one for the simple-but-nice 'sabr' shader and one for the much more complex 'scalefx' shader) located in the 'handheld' directory that will take the default NDS stacked image and do a fake hybrid view with the scaler applied on the big image.

Thanks for the shader, it works perfectly, I noticed a problem, in "ds-hybrid-scalefx.glslp" there is no "Show both small screens" option in the parameters, an option that is in "ds-hybrid-scalefx.slangp" but this doesn't work for me. Could you add this option to the first shader too? Really thanks!