[Feature Request] FMV Internal Upscaling

[Papermanzero]

With build b0d5562 the videos are not upscaled and stay on the native resolution. However I would expect if the internal resolution is increased a nearest neighbour or bilinear upscaling for the videos will be applied as well. The user can recognize the wrong resolution if he uses shaders. For example an aa shader.

[ggrtk]

@Papermanzero Is this an issue that is newly introduced? If so, examples would be appreciated so that the issue can be diagnosed.

[Papermanzero]

It is a n issue which exists since the introduction of increased resolution.

I will make some screenshots and explain the issue in more details later

[ggrtk]

Okay so if I am understanding correctly, this is a feature request for some sort of upscaling filter provided by the core for FMVs.

FMVs are outputted at native resolution and unaffected by the internal resolution setting because they are pre-rendered content stored in the game itself rather than rendered in real time.

[Papermanzero]

Indeed. This is the request. Otherwise some shaders are not working correctly.

[Papermanzero]

Here is an example:

with OpenGL renderer native resolution:

With Vulkan renderer 4x resolution:

With Vulkan renderer 4x resolution and shaders:

Shader Vulkan preset:

---

shaders = 3

shader0 = shaders_slang/denoisers/shaders/fast-bilateral-3d.slang filter_linear0 = false scale_type0 = source

shader1 = shaders_slang/anti-aliasing/shaders/aa-shader-4.0-level2/aa-shader-4.0-level2-pass1.slang filter_linear1 = false scale_type1 = source scale1 = 2.0

shader2 = shaders_slang/anti-aliasing/shaders/aa-shader-4.0-level2/aa-shader-4.0-level2-pass2.slang filter_linear2 = false scale_type2 = source

parameters = "SIGMA_R;AAOFFSET;AAOFFSET2"

AAOFFSET = "1.20" AAOFFSET2 = "1.60"

---

The expactation would be that the video is also scaled to 4x resolution so that the shader can be applied correctly. So a feature to increase the video resolution would be helpful.

[ggrtk]

I am not familiar with how shaders are applied by the frontend so this is something we will need hizzle's input on.

@hizzlekizzle Currently for FMVs, the Vulkan renderer hands the frontend a native resolution 240-height or 480-height framebuffer regardless of the internal upscale. If we wanted these shaders to be applied properly would it be sufficient to have the Vulkan renderer give the frontend a framebuffer at the upscaled resolution with the FMV scaled to those dimensions?

[hizzlekizzle]

Yes, I think so. Just scale it up to size via NN. As long as it's the right number of pixels per texel, the detection algos should work.

[ggrtk]

@Papermanzero Does this issue also show up with the GL and software renderers as well?

Just trying to understand the scope of this problem.

[Papermanzero]

Yes it does. Software renderer has no upscaling so the issue is not relevant there. GL renderer has the same issue because FMVs are not upscaled. The question is also how to upscale them. E.g bilinear or nearest.

[ggrtk]

The software renderer should have an internal gpu resolution option, is that not showing up for you?

[Papermanzero]

Ah ok. It shows up. I thought the software renderer is equal to mednafen standalone. :) So I believed that the software renderer ignores the resolution. But after a small test it is the same result.

[nezumisama]

I currently see three options to have smooth, higher-quality graphics, either:

• You let shaders handle the 240p output form the emulated GPU. Or just bilinear filtering instead of shaders.
• You use a combination of options in the core, like increased internal resolution, MSAA, texture filtering and other, but with nearest-neighbour interpolation (no shaders).
• Or you combine the two aforementioned approaches.

The first option always gives filtered image but there's a problem that the shaders are either designed to:

• Filter general images, but then they can't at the same time give an image, that doesn't have aliasing, preserves finer details and is free from artifacts from ringing.
• Upscale pixel-art images, which work great for low-color 2d graphics, but not so for general images. The second option is great, but has one major flaw - the MDEC videos are pixelated. The third option has the same problem with filtering as the first and also in the process of filtering, the crisper 3d graphics provided by increased internal resolution are blurred by the filter.

In general, the problem is that PSX games produce a combination of different types of graphics: textured 3d models, sprites, detailed backgrounds and videos (which are shown directly or sometimes treated as a background for 3d models like in Final Fantasy 8 for example). Each of these types of output needs special treatment. For 3d graphics we already have increased resolution, MSAA and texture filtering. If I'm not mistaken, Backgrounds are also affected by the texture filter from what I've observed. Which leaved MDEC videos.

Given the above, in my opinion, it would make sense to add MDEC Video filtering within the core. I don't know the code but perhaps the filters for textures could be re-used. Videos are a similar type of image to textures enough to support that idea.

[ggrtk]

Issue is duplicate of #292. I won't close this one though since there's good information in here.

[Papermanzero]

So I did a test with shaders. I dumped the native resolution and upscaled them with nearest neighbor to HD. After that I applied following shader preset:

---

shaders = 3

shader0 = shaders_slang/denoisers/shaders/fast-bilateral-3d.slang filter_linear0 = false scale_type0 = source

shader1 = shaders_slang/anti-aliasing/shaders/aa-shader-4.0-level2/aa-shader-4.0-level2-pass1.slang filter_linear1 = false scale_type1 = source scale1 = 2.0

shader2 = shaders_slang/anti-aliasing/shaders/aa-shader-4.0-level2/aa-shader-4.0-level2-pass2.slang filter_linear2 = false scale_type2 = source

parameters = "SIGMA_R;AAOFFSET;AAOFFSET2"

AAOFFSET = "1.20" AAOFFSET2 = "1.60"

---

HD Nearest Upscale:

HD Nearest Upscale + Shader:

HD Nearest Upscale:

HD Nearest Upscale + Shader:

HD Nearest Upscale:

HD Nearest Upscale + Shader:

It becomes a little bit blurry. But as a conclusion. I would only implement a nearest upscaling for videos which is according to the resolution factor. The users can apply shaders on top of that. Alternativly you implement a bilinear video filter option which is applied for MDEC videos. But I would do that in a later point in time. First, just upscale the video in accordance with the resolution factor via nearest neighbor

[ggrtk]

@Papermanzero If you're able to build cores, I've updated the Vulkan renderer on commit 66ff383 so that FMVs will now upscale to your selected Internal Resolution multiplier.

[Papermanzero]

I tried to compile but I have an issue with mednafen/cdrom/CDAccess.cpp: if(strlen(path) >= 4 && !strcasecmp(path + strlen(path) - 4, ".ccd"))

strcasecmp is not declared. I tried to add string.h but the issue is the same. Can u build a version?

[ggrtk]

What platform are you building the core for?

[Papermanzero]

Windows 10

[ggrtk]

@Papermanzero Here's a Windows 10 x86_64 build of 957ee8c: mednafen_psx_hw_libretro.zip

You could also acquire it via the Online Updater if you prefer now that the update has been merged into master branch, but the buildbot does not have the update yet as I'm writing this.

This build has the FMV upscaling update to Vulkan included. Just a fair warning that the GL renderer does not have the FMV upscaling changes.

[Papermanzero]

Working perfectly. Thanks a lot. Tested Silent Hill, Soul Blade etc. all games with FMV looks pretty good.

[hizzlekizzle]

Awesome. Thanks for confirming. I'll close this now.

[ggrtk]

@hizzlekizzle I'm going to reopen this for the time being since the FMV internal upscaling is currently only implemented in the Vulkan renderer. The OpenGL and software renderers haven't received this feature yet. Sorry for any confusion.

[Papermanzero]

Related to this one: https://github.com/libretro/beetle-psx-libretro/issues/245

[ggrtk]

@Papermanzero Can you provide an example of the GL renderer FMV upscaling differing from the Vulkan renderer? Looking at the GL renderer code and doing GL frame captures I'm fairly certain the FMVs are already upscaled before being handed over to the frontend.

[Papermanzero]

I will try it. Will do some screenies tomorrow

[Papermanzero]

I dont even have to make screenshots. OpenGL behaves exactly the same as vulkan. So upscaling works for both now.

[ggrtk]

Ok, thanks for confirming. So based on the previous discussion in this thread I think you said that the software renderer lacks this feature. Are your shaders not working properly at upscaled internal resolutions with the software renderer?

[Papermanzero]

The vulkan renderer missed the feature. Tested the software renderer, which works as well. So i guess we can close the issue. But if u want to implement a bilinear filter beside the nearest neighbor.

[ggrtk]

So prior to this issue thread, both the GL renderer and software renderers had internal upscaling on FMVs while the Vulkan renderer did not have it. That clears things up, thanks.

Giving a choice of upscaling filter is a possible improvement for the future.

[joppuyo]

What settings are you using?

I'm unable to use any filters/shaders on FMVs when my internal resolution is over 1x. I tried using both Vulkan and OpenGL.

Running stable version of RetroArch (1.9.13) and Beetle PSX HW (0.9.44.1).

[Papermanzero]

Standard Settings. Filters are working great.

[MishaTrent]

Ok, thanks for confirming. So based on the previous discussion in this thread I think you said that the software renderer lacks this feature. Are your shaders not working properly at upscaled internal resolutions with the software renderer?

Not to create a separate topic, I'll ask here. Do you plan to add anti-aliasing for FMV videos with more than 1x internal resolution in the future?