Feature Request - Upscaling Filters

[enderandrew]

The aim is playing an old, pixelated, low-resolution game on modern hardware with higher resolution monitors.

It would be nice to have some upscaling filters such as:

• Nearest neighbor
• Bilinear
• hqx
• xbrz

[afritz1]

I'm not really accepting feature requests, sorry :/ and I don't know how upscaling filters work with SDL_Texture. This would be a very low priority issue.

[enderandrew]

Tons of other SDL projects use opensource upscaling filters in games and emulators.

Here is an example from ScummVM using SDL and nearest-neighbor to upscale 4x.

https://drive.google.com/file/d/1wk87f2r4VHwYb0PrpBj9XIFS4QlbE1QM/view

[afritz1]

Oh it just scales the texture dimensions up. I thought it was some OpenGL pixel shader thing. Well this still would be hard to implement in the engine's current state because the UI is very much in a primitive first-attempt state and doesn't do everything in vector space, so I don't see this being implemented for a long time. I still have my own big stack of features I want to do first ;)

[enderandrew]

There are lots of ways of doing upscaling, including OpenGL filters, but it can be done with SDL as well.

[Digital-Monk]

I'm not sure that I see the point of an additional nearest-neighbor upscale, given that OpenTESArena already supports arbitrary resolutions and:

• does a nearest-neighbor equivalent scaling of the UI panels to fit that resolution
• renders the 3D world using the full available resolution (or whatever percentage you've specified for aesthetic/performance reasons) using an approach that is equivalent to nearest-neighbor upscaling of the input textures

That's just for nearest-neighbor, however. I can see, and would also be interested in, quality-improving upscaling engines. They would not need to run per-frame, so there's no need for pixel shaders or anything tricky like that. They could just be wedged into the resource-loading path, so that any time the engine reads an image (texture, gui screen, whatever), it then passes that image through an upscaler before handing it off to the rest of the engine.

At least, that's the theory for my very limited understanding of how the engine actually works... But I can imagine that the raycaster/renderer may have some very specific optimizations for texture images being fixed sizes, and those optimizations may not appreciate larger textures. If that's the case, then yeah, this would be a big headache. Still very nice, but low priority.

But, if it is actually as easy as just wedging an image processor into the image loading path, then I would vote for xbrz (if it's easy to apply) or hq2x (easier to apply, not quite as nice). I believe those upscaling engines are designed to just take an image and spit out a larger image, so they should be fairly easy to wedge into place... Just, y'know, if you get tired of working on actual gameplay concerns ;)

Just gonna drop these here: xBRZ HQx-vs-'improved' approach -- I'm conflicted. Soft balloony things tend to look much better, but anything with sharp corners gets destroyed (well, softened into child-safe balloon version of itself), which probably isn't great for upscaling wall/door textures...

[afritz1]

They could just be wedged into the resource-loading path, so that any time the engine reads an image (texture, gui screen, whatever), it then passes that image through an upscaler before handing it off to the rest of the engine.

That's what I was thinking. Off the top of my head I don't know if it would make sense to have categories of UI that do or don't get upscaled.

I realized this could also be applied to game world textures. It will bring more attention to how out-of-memory situations should be handled if i.e. the player wants the 64x64 voxel textures to be like 1024x1024 instead, and if they accidentally mod in 1024x1024 textures and then want THOSE to be scaled up, accidentally asking the engine for tens of gigabytes of RAM. I'm not really handling textures as 64-bit things because I didn't expect the sizes to be that huge.

This feature probably still won't be done for a while still.

[Digital-Monk]

I would assume an all-or-nothing approach for the UI elements. If something shows up as a problem, it can be dealt with then, but I suspect YAGNY... And, personally, I don't really care if the UI elements get upscaled or not, I was just trying to follow the thread.

(I'm not sure how UI elements are put together -- if they're full dialogs or if they're various buttons and pictures and such -- but I'd generally recommend running any upscaling at a whole-dialog level to avoid weird edge effects between pieces)

Game world textures is really what I'd like to see -- well, those and billboard flats for people and lightposts and statues and such. I was thinking a wedge in the texture loading path because I assumed textures were coming straight out of Arena and weren't particularly moddable. If we can replace textures with our own imagery, well, we can do the upscaling outside the engine and provide upscaled texture packs (like DFU's Plain Vanilla Extract mod does).

If modding textures is a long way off, though, it would be cool to have a texture upscaling wedge for the vanilla textures. Basically, you shouldn't ever need to upscale modded textures, and by limiting upscaling to vanilla textures you give yourself some control over maximum resource loads. It also would be cool to keep around even after moddable textures were available, because then I could have piecemeal texture mods of specialized quality, with a fallback to xBRZ or HQx upscaled vanilla textures for anything that hasn't been modded yet. Or, possibly more likely, if I'm generally happy with the upscaled vanilla textures and sticking with the classic art style, I would only need to mod particular textures that didn't upscale nicely.

Just thoughts...

[Digital-Monk]

Yup... Never mind... Decided to try some trivial upscaling, just introducing noise or something. "Fixed" SETFile::getPixels to do the upscaling rather than patching the four (I think) places that call it. Which doesn't work because 64 is baked into the voxel/texel software renderer (I thought it might be). Bumped that value up to my new res, and it broke things in entertaining ways. Voxel textures from SET files just appeared in the upper left corner of their relevant face. Textures from other sources -- anything with a transparency, for instance -- lost its mind because there wasn't enough data for what it was trying to do.

Which is a long-winded way of saying that I now understand why this is a low priority :smile:

EDIT: I'm a doofus. I whiffed the upscaling logic, which is why the stuff rendered in the upper left corner. Fixed that, and now I have "noisy" voxel textures on most surfaces -- though my approach to "noise" doesn't work all that well with a palette. But the overall idea isn't as bad as I'd thought just a few minutes ago...

[Digital-Monk]

This post is pretty much pointless, but I can't help myself -- I just had to share some partial hackish results...

3x upscaling, center pixel remains original, edge pixels are toggled between adjacent colors using checkerboard... The first upscaling that I was remotely happy with, though it still makes me want to claw out my eyes because it's too "blurry" (sort of -- 8/9 of image is edge dithering with only 1/9 original data). But still better than everything I tried before...

5x upscaling.
I could live with this. Much more of the data is the original data, but still some "blurring" at pixel boundaries to break things up. Still not sure I'd call it an improvement over the original, but that wasn't the point for me -- I was just trying to get an upscale that I could tell changed things and didn't make them horribly worse...

Oh, and 4x upscaling is a bad idea with checkerboard, because the adjacent upscaled regions tile oddly and you get structured patterns instead of checkerboards.

One last pic, from the Imperial City. I've got the sky hacked out so it's just black (leftover of stars experiments). A couple of wall textures. Can also see what happens to any texture that didn't come from a SET file, because the renderer has a fixed texture size for all sources but I've only upscaled SET textures so far...

[afritz1]

I know this is an experiment but I don't want anyone putting in work on this when the renderer is subject to so much change still. Everything needs to be done with 32-bit floats, the shaders need to be reworked, the ray cast loop needs to be vectorized, etc...