ruby: Add source frame scaling via `screen` functions

[jcm93]

(Marked as draft pending some further testing and any feedback)

This PR exposes several methods in screen to the video backend in ruby, in order to resolve certain libretro shader scaling issues, and downscale source frame buffers as appropriate to improve shader performance.

Background

ares's current rendering strategy, broadly, is to render frames on the CPU one at a time before sending them to the video backend. The frames sent to the video backend can have many duplicated pixels present, as a shortcut to account for cases where frames contain scanlines with different pixel clocks. Unless we scale these buffers down to the appropriate size before sending them to librashader, we can introduce artifacts, cause incorrect shader behavior, or even crash if we cause a shader to create an overly large texture.

By allowing screen's setScale, setInterlace and setProgressive methods to call into the video backend, we can signal that ruby should scale down the framebuffers appropriately before sending them to shaders. This is (in theory of course) more efficient than performing this scaling on the CPU, and more streamlined logically in terms of making the video backend responsible for the final frame buffer creation.

Proof of concept

To test this approach, I added rudimentary implementations of these functions for the Metal backend. These methods basically do the following in a separate render pass, prior to shader passes:

• In terms of width, simply always scale the framebuffer to the largest pixel width.
• In terms of height, if we are in the progressive mode and the framebuffer is doubling identical lines, scale down.
• If we're in the interlaced mode and lines are doubled, don't scale down, because a frame larger in height than a deinterlaced frame often acts as a signal for shaders that we are outputting interlaced video.

This approach is basic and may need refinement, but is a solid improvement on existing behavior in my testing.

Future work

(Disclaimer; not trying to speak for maintainers here; just thinking out loud):

Per conversations on the ares Discord, it seems as though ares may in the future move toward a renderer that leans more heavily on the video backend, perhaps similar to the approach taken by something like CLK, where the core generates a video signal, and the display backend is entirely responsible for synthesizing that signal into frames. This could have numerous advantages in terms of accuracy, performance and latency.

While not terribly significant, this PR could be considered a small step in this direction, as it moves some of the work generating the final source frame from the CPU to the GPU/display backend.

This does, however, also mean that for other video backends to achieve parity with Metal in terms of framebuffer scaling, they will need to implement setScale, setInterlace and setProgressive themselves in some fashion.

Examples

crt/crt-maximus-royale.slangp used here, not because it is my favorite shader but because it seems to detect interlacing changes on the fly, and also creates very large textures.

SNES 240p, first with PR then without PR:

Sega 32x at combined 320/256 pixel width, first with PR then without PR:

Sonic 2 switching from interlaced to progressive during runtime (works both ways) (crashes without PR):

[LukeUsher]

I like it!

My Mac is currently out of action so I can't test it myself right now, though Any chance you could port this to the OpenGL backend so I can have a play with it on Windows?

[jcm93]

Yeah, the reason this is still in draft is that I started working on a larger PR to allow all the software rendering functions in screen to be overridden optionally by implementations in ruby (color bleed, sprites, etc.). I was planning on finishing that up and submitting it, then basing this off of that work instead.

I've also been planning on backporting some of my other work to OpenGL too, so maybe I'll look at that first.