Testing for ReShade Fx shaders

[DadSchoorse]

I'm working on supporting reshade shaders (see #41) in wip-reshade-fx. Some shaders should now work, so testing would be quite good. Here is a build: vkBasalt.tar.gz Edit: use a newer build from below

Please install spriv-cross and glslang before testing. Edit: You don't need spriv-cross and glslang anymore

To use Reshade Shaders, download https://github.com/crosire/reshade-shaders Then set in the config:

effects = reshade

disableSpirvWorkaround = 0
reshade = *path to a shader from the repo*
reshadeTexturePath = *path to the texture dircetory from the repo*
reshadeIncludePath = *path to the shader dircetory from the repo*

Shaders that should work: LUT.fx, SMAA.fx, LumaSharpen.fx

please also test with Edit: the workaround has been removed and isn't useful anymore

disableSpirvWorkaround = 1

in games and in vkcube.

[pchome]

I did some testing with disableSpirvWorkaround = 1, simple effects alike CRT/ASCII/Cartoon seems works.

Some have no texture file, so crashing w/ segmentation fault (fx contains pooled = true). I temporary patched crashing with the following patch, but such textures likely should be updated at runtime (runtime.cpp#L491):

--- a/src/effect_reshade.cpp
+++ b/src/effect_reshade.cpp
@@ -73,7 +73,7 @@ namespace vkBasalt
             }
             VkExtent3D textureExtent = {module.textures[i].width, module.textures[i].height, 1};
             //TODO handel mip map levels correctly
-            if(module.textures[i].annotations.size() == 0)
+            if(module.textures[i].annotations.size() == 0 || module.textures[i].annotations[0].name == "pooled")
             {
                 textureMemory.push_back(VK_NULL_HANDLE);
                 std::vector<VkImage> images = createImages(instanceDispatchTable,

Other effects just freezes with GPU driver crash messages in dmesg, such ones contains e.g. source = "timer". Looks like runtime stuff too: runtime_objects.hpp#L12 and runtime.cpp#L922.

Note: ASCII.fx contains source = "timer", but looks like it never used, so effect works.

Edit: Some effects are rely on Windows case insensitive FS, e.g. MultiLUT.fx: wants MultiLut_Atlas1.png, has MultiLut_atlas1.png.

[DadSchoorse]

@pchome thanks for testing!

• Did disableSpirvWorkaround = 1 work with proton games? What gpu do you have?
• I will not support pooled textures for now, that's something for a later release
• All of the < source = .. > are not easy to implement with the current code, so that's also going to be fixed later.
• Case insensitive stuff should be fixed in the shader repo, I am trying to collect all of them and make one big pull request for it. e.g. some shaders include Reshade.fxh instead of ReShade.fxh

[pchome]

1) I'm playing with vkcube for now, because it's quick and easy to launch and/or kill if something will go wrong. Not sure about real games for now, as well as about effects correctness. Nvidia GTX 750Ti 2GB 4) The easiest way is to do git mv old_name.png new_name.png in shaders repo. But it depends.

[DadSchoorse]

@pchome Please test at least one real game. disableSpirvWorkaround = 1 causes a hang on pipeline creation in all proton games I tested and I don't know why. One Amd user on discord does not have this problem and I want to know if it's a nvidia specific issue, a steam issue or just my setup.

[pchome]

I tested a few effects, but unlike vscube only CRT.fx loaded w/o problems. Cartoon.fx, ASCII.fx, Nostalgia.fx, ... all just shows black screen in Proton.

[DadSchoorse]

CRT.fx doesn't work for me. The issue is so weird, I can't debug it and the workaround doesn't work for many effects because glslang is broken. If more non nvidia user can't reproduce it, i wouldn't even know how to report it to nvidia.

[DadSchoorse]

8310fd6b70a51812df8fa7350f702609eee51044 fixes the nvidia issue. Here is an updated build: vkBasalt.tar.gz

[DadSchoorse]

Jet another new build which removes the need for disableSpirvWorkaround:vkBasalt.tar.gz

[DadSchoorse]

An up to date build: vkBasalt.tar.gz No shader should cause a crash now. All shaders except shader which use access to the depth buffer should work correctly.

[Ahmed-E-86]

Please install spriv-cross and glslang before testing.

What are the exact name of these packages on Linux Mint/Ubuntu?

The old vkBasalt.conf didn't be replaced during installation, I had to replace it manually.

[DadSchoorse]

Please install spriv-cross and glslang before testing.

What are the exact name of these packages on Linux Mint/Ubuntu?

You do not need them anymore, the workaround has been removed.

The old vkBasalt.conf didn't be replaced during installation, I had to replace it manually.

That's intentional .

[Ahmed-E-86]

I tried LumaSharpen.fx with Darksiders Genesis, and I don't know if it is working or not, because the game is a little cartoony with dark textures, so it is hard to notice the difference, but the game didn't crash with these settings:

effects = reshade

disableSpirvWorkaround = 0
reshade = ~/Downloads/reshade-shaders-master/Shaders/LumaSharpen.fx
reshadeTexturePath = ~/Downloads/reshade-shaders-master/Textures
reshadeIncludePath = ~/Downloads/reshade-shaders-master/Shaders

What other ReShade shaders that I can use? And how can I use multiple ReShade shaders?

[DadSchoorse]

@ahmed-elsayed2017 try increasing the sharpness directly in the shader file:

uniform float sharp_strength < __UNIFORM_SLIDER_FLOAT1
    ui_min = 0.1; ui_max = 3.0;
    ui_label = "Shapening strength";
    ui_tooltip = "Strength of the sharpening";

> = 0.65;

Change the 0.65 to something higher.

You can use multiple reshade shaders: e.g.:

effects = effect1:effect2

disableSpirvWorkaround = 0
effect1 = ~/Downloads/reshade-shaders-master/Shaders/LumaSharpen.fx
effect2 = ~/Downloads/reshade-shaders-master/Shaders/CRT.fx
reshadeTexturePath = ~/Downloads/reshade-shaders-master/Textures
reshadeIncludePath = ~/Downloads/reshade-shaders-master/Shaders

If you search for "depth" in a shader file and don't find anything, the shader should work

[Ahmed-E-86]

Thanks. Is there any documentation that describes what each shader does?

[leucome]

I'm not sure why but I can not use ~/ as home in the path so I need to the use the full path.

So far these worked for me crt, fakehdr, hq4x, dof, crt, vignette, cartoon, filmgrain, clarity

These caused a crash magicbloom, cinematicdof, bloom, ppfx_bloom, mxao Ambientlight start but the effect is messed up. Depth3d completely hang my GPU,

Tested with some wine/proton game and serious sam fusion. using a RX 480 with mesa 20 ACO and TKG-PDS-fsync kernel. I used vkBasalt.tar.gz suggested in this discussion 4 day ago. So I'm not 100% it is the latest one.

Maybe there could be a list of the effect that are likely to work added to the original post.

[DadSchoorse]

@leucome I you're using this build https://github.com/DadSchoorse/vkBasalt/issues/45#issuecomment-573792909 , right? If so, I can not reproduce these problems. Make sure that your texture path is set correctly.

Please give me the terminal output of ENABLE_VKBASALT=1 vkcube --validate with one of the shaders that crash.

So far these worked for me crt, fakehdr, hq4x, dof, crt, vignette, cartoon, filmgrain, clarity

dof can not work correctly because depth access isn't implemented yet. The same is actually true for all the effects that crash for you. But it's still a bug that they crash.

Edit: One thing that would be nice to know: Does it work with amdvlk?

[leucome]

Yeah dof was in my list of working one because it did not crashes, but the effect was messed up.

An example of a crash with ~/ used in the path. https://drive.google.com/open?id=1jDTZqUCpKr-_8DR5gCTB8xwDwvnvmukl

Also a crash with cinematicdof. https://drive.google.com/open?id=1VNH_SXbqD_tgNvhjv3siBqL17QSJ4zlB

I'm installing amdvlk and I'll do some tests.

[leucome]

I tried with amdvlk. There is no shader related crash. I still have the same trouble with ~/ home path.

Maybe I'll also try mesa 19 instead of mesa 20.

[DadSchoorse]

@leucome I think I know what causes the crash. It's an unsupported VkFormat. I will try to solve that. As for the ~, it's normal that it doesn't work currently.

[DadSchoorse]

@leucome please test again with cinematicdof and mxao. The issue should be fixed, otherwise give me the terminal output again. Here's a new build: vkBasalt.tar.gz

[leucome]

There is some change in the output but it end up crashing again.

Mxao : https://drive.google.com/open?id=1_h-f3OQ1zEe_ASCAAj9VB7WuFmIxeF4n

Cinematicdof : https://drive.google.com/open?id=1OA4FtkkKJz2bGlPYIpPtEGFGRTA20d5x

Magicbloom https://drive.google.com/open?id=1dDYhrMDtBDoxZMTammWaF6rhpVeAazZg

I'm going to try with radv from mesa 19, just in case.

[DadSchoorse]

@leucome I know what's causing that. radv does not support VK_KHR_swapchain_mutable_format yet. (And anv only got it one day ago). I made a request for it: https://gitlab.freedesktop.org/mesa/mesa/issues/2354 and there is already a merge request for radv. So hopefully it will be merged before mesa 20. In most games it should work btw, but vkcube creates a sRGB swapchain.

Anyway thanks for testing!

[DadSchoorse]

@leucome please try again, I added a workaround for the missing extension: vkBasalt.tar.gz

[zany130]

i get

ENABLE_VKBASALT=1 vkcube --validate
ERROR : GENERAL - Message Id Number: 0 | Message Id Name: Loader Message
        /usr/lib32/libvkbasalt.so: wrong ELF class: ELFCLASS32

        Objects - 1
                Object[0] - VK_OBJECT_TYPE_INSTANCE, Handle 0x55b5a6c09920

interrupted create instance
/home/zany130/.local/share/vkBasalt/vkBasalt.conf
set option effects equal to effect1:effect2:effect3:effect4:effect5:effect6:effect7:effect8
set option disableSpirvWorkaround equal to 0
set option effect1 equal to ~/git/reshade-shaders-master/Shaders/3DFX.fx
set option effect2 equal to ~/git/reshade-shaders/Shaders/GaussianBlur.fx
set option effect3 equal to ~/git/reshade-shaders/Shaders/FilmicAnamorphSharpen.fx
set option effect4 equal to ~/git/reshade-shaders/Shaders/AdaptiveSharpen.fx
set option effect5 equal to ~/git/reshade-shaders/Shaders/HighPassSharp.fx
set option effect6 equal to ~/git/reshade-shaders/Shaders/Clarity.fx
set option effect7 equal to ~/git/reshade-shaders/Shaders/HQ4X.fx
set option effect8 equal to ~/git/reshade-shaders/Shaders/Colourfulness.fx
set option reshadeTexturePath equal to /home/zany130/git/reshade-shaders/Textures
set option reshadeIncludePath equal to /home/zany130/git/reshade-shaders/Shaders
ERROR : GENERAL - Message Id Number: 0 | Message Id Name: Loader Message
        /usr/lib32/libvkbasalt.so: wrong ELF class: ELFCLASS32

        Objects - 1
                Object[0] - VK_OBJECT_TYPE_INSTANCE, Handle 0x55b5a6c09920

found graphic capable queue
queue 0x55b5a6b25340
format 44
device 0x55b5a6f81480
swapchain 0xb000000000b
Interrupted create swapchain
Interrupted get swapchain images 0
Interrupted get swapchain images 3
fake image size: 1048576
fake image alignment: 1024
after createFakeSwapchainImages 
3swapchain images
current effectString effect1
3 images in firstImages
not using swapchain images as second images
3 images in secondImages
terminate called after throwing an instance of 'std::runtime_error'
  what():  unknown effecteffect1
Aborted (core dumped)

when i run the version from the aur ( i think thats the latest)

downloading it maunualy and running make install and then trying vkcube again gives this

[zany130@andres-pc vkBasalt]$ ENABLE_VKBASALT=1 vkcube --validate
ERROR : GENERAL - Message Id Number: 0 | Message Id Name: Loader Message
        /home/zany130/.local/share/vkBasalt/libvkbasalt32.so: wrong ELF class: ELFCLASS32

        Objects - 1
                Object[0] - VK_OBJECT_TYPE_INSTANCE, Handle 0x55ccc30d9b70

interrupted create instance
/home/zany130/.local/share/vkBasalt/vkBasalt.conf
set option effects equal to effect1:effect2
set option disableSpirvWorkaround equal to 0
set option effect1 equal to /home/zany130/git/reshade-shaders-master/Shaders/3DFX.fx
set option effect2 equal to /home/zany130/git/reshade-shaders/Shaders/GaussianBlur.fx
set option effect3 equal to /home/zany130/git/reshade-shaders/Shaders/FilmicAnamorphSharpen.fx
set option effect4 equal to /home/zany130/git/reshade-shaders/Shaders/AdaptiveSharpen.fx
set option effect5 equal to /home/zany130/git/reshade-shaders/Shaders/HighPassSharp.fx
set option effect6 equal to /home/zany130/git/reshade-shaders/Shaders/Clarity.fx
set option effect7 equal to /home/zany130/git/reshade-shaders/Shaders/HQ4X.fx
set option effect8 equal to /home/zany130/git/reshade-shaders/Shaders/Colourfulness.fx
set option reshadeTexturePath equal to /home/zany130/git/reshade-shaders/Textures
set option reshadeIncludePath equal to /home/zany130/git/reshade-shaders/Shaders
ERROR : GENERAL - Message Id Number: 0 | Message Id Name: Loader Message
        /home/zany130/.local/share/vkBasalt/libvkbasalt32.so: wrong ELF class: ELFCLASS32

        Objects - 1
                Object[0] - VK_OBJECT_TYPE_INSTANCE, Handle 0x55ccc30d9b70

device supports VK_KHR_swapchain_mutable_format
activating mutable_format
found graphic capable queue
50 44
format 44
Interrupted create swapchain
Interrupted get swapchain images 0
Interrupted get swapchain images 3
fake image size: 1048576
fake image alignment: 1024
after createFakeSwapchainImages 
3swapchain images
current effectString effect1
3 images in firstImages
not using swapchain images as second images
3 images in secondImages
in creating ReshadeEffect 
after creating input ImageViews
after creating ImageViews
ERROR : VALIDATION - Message Id Number: 0 | Message Id Name: UNASSIGNED-CoreValidation-Shader-InconsistentSpirv
        SPIR-V module not valid: No OpEntryPoint instruction was found. This is only allowed if the Linkage capability is being used.

        Objects - 1
                Object[0] - VK_OBJECT_TYPE_UNKNOWN, Handle (nil)

after creating shaderModule
Stencil Format: 129
after creating descriptorSetLayouts
ERROR : VALIDATION - Message Id Number: 0 | Message Id Name: VUID-VkDescriptorPoolSize-descriptorCount-00302
        vkCreateDescriptorPool(): pCreateInfo->pPoolSizes[0].descriptorCount is not greater than 0. The Vulkan spec states: descriptorCount must be greater than 0 (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-VkDescriptorPoolSize-descriptorCount-00302)

        Objects - 1
                Object[0] - VK_OBJECT_TYPE_DESCRIPTOR_POOL, Handle (nil)

after creating descriptorPool
after creating Pipeline layout
0
Segmentation fault (core dumped

[DadSchoorse]

when i run the version from the aur ( i think thats the latest)

it's the latest release, but the testing here is about the WIP version.

The second error is caused by the shader file not existing.

set option effect1 equal to /home/zany130/git/reshade-shaders-master/Shaders/3DFX.fx

This seems wrong, doesn't it?

[zany130]

fixed it works with effect1 = /home/zany130/git/reshade-shaders/Shaders/CRT.fx effect2 = /home/zany130/git/reshade-shaders/Shaders/GaussianBlur.fx effect3 = /home/zany130/git/reshade-shaders/Shaders/FilmicAnamorphSharpen.fx effect4 = /home/zany130/git/reshade-shaders/Shaders/HighPassSharpen.fx effect5 = /home/zany130/git/reshade-shaders/Shaders/Clarity.fx effect6 = /home/zany130/git/reshade-shaders/Shaders/HQ4X.fx effect7 = /home/zany130/git/reshade-shaders/Shaders/Colourfulness.fx effect8 = /home/zany130/git/reshade-shaders/Shaders/DPX.fx all effects load and work

[Sporif]

I'm using the latest commit and on my GTX 1080 most of the shaders seem to least work with vkcube, but there's a few that crash or blackscreen.

Both Colourfulness.fx and Emphasize.fx segfault with something like this (paths truncated):

.../reshade-shaders/Shaders/Emphasize.fx(55, 1): preprocessor error: could not open included file '/mnt/storage2/Mods/Reshade/reshade-shaders/Shaders/Reshade.fxh'

.../reshade-shaders/Shaders/Emphasize.fx(87, 34): error X3004: undeclared identifier 'ReShade::BackBuffer'
.../reshade-shaders/Shaders/Emphasize.fx(98, 16): error X3501: undeclared identifier 'PostProcessVS', expected function name

Splitscreen.fx and LightDoF.fx both blackscreen.

AmbientLight.fx seems to work but the effect disappears when you go fullscreen.

[leucome]

@leucome please try again, I added a workaround for the missing extension: vkBasalt.tar.gz

I get the same result with this build. Cinematicdof : https://drive.google.com/open?id=1iHuYKtMwKH5Un2NVQMbZMHkQab72PfNW

At least it's possible to use amdvlk until the missing extension get implemented.

[DadSchoorse]

@Sporif

Both Colourfulness.fx and Emphasize.fx segfault with something like this (paths truncated):

that's caused by linux having case sensitive filesystems. The file is called ReShade.fxh, not Reshade.fxh. Both shaders have been fixed by my pull request: https://github.com/crosire/reshade-shaders/pull/200

Splitscreen.fx and LightDoF.fx both blackscreen.

yeah multi technique shaders are broken

AmbientLight.fx seems to work but the effect disappears when you go fullscreen.

I think it is actually the other way around. On the small 500x500 cube it's broken.

[DadSchoorse]

@leucome We have identified the issue, it's a bug in reshade's shader compiler.

[DadSchoorse]

@leucome crosire fixed the bug, please try this build: vkBasalt.tar.gz

[leucome]

@leucome crosire fixed the bug, please try this build: vkBasalt.tar.gz

Yes this one works with RADV.

[pchome]

I created Gentoo overlay where vkBasalt, reshade-fx, reshade-fxc, reshade-shaders, etc. are present. If anyone want to give it a try, then better use it as template for your local overlay. https://github.com/pchome/vulkan-gentoo-overlay

# emerge -1 media-libs/vkbasalt -q currently points on wip-reshade-fx branch and acts as live ebuild, will change after release

Note: installing as explicit layer, name VK_LAYER_CUSTOM_vkBasalt (?, idk, CUSTOM is invalid author ID too). Also there is vkconfig in overlay, where one can temporary force-enable layers.

[leucome]

I tested the depth buffer version a little. It seem to work pretty good.

There a test with multiple effects... hq4x:cinematicdof:clarity:cas:fxaa:magicbloom:vignette It's Tony hawk 3 and it's a directX 8 game. I used d3d8to9 to make it work with DXVK 1.5.2 to finally add the VBbasalt/Reshade effects. Interesting that it work with so many API translation.

[leucome]

There seem to be some issue with mxao. Every part of the picture that is not affected by a sample stay black or is filled with something else from some random frame buffer. The same glitch happens with the vkcube too.

But good news... There is some AO shader that worked. I got some from GSshade https://github.com/Mortalitas/GShade-Shaders

From this pack MXAO3 is working and also the SSAO BTAO rayAO and quint_MXAO.

Example of the SSAO debug.

Ant then everything put together with rayao:clarity:cinematicdof:cas:fxaa:magicbloom:vignette

[leucome]

I tested with some more Proton/Wine game. Using depth with MSAA hang the GPU. If I remember correctly MSAA + depth usually cause issue with reshade too.

Unreal Engine 3 games hang the GPU or crash when depth capture is turned on.

I tried with AMDVLK and RADV and I get about the same result.

Is there is any of these issues that are unknown or unexpected ? If so, can test these some more.

Here I included on log of Alice Madness Return until it freeze with cinematicdof. Note that even before freezing the dof effect is fixed and do not seem to read an depth value. https://drive.google.com/open?id=19DLNnj-bjZ51_qHw4ttSbSYkzFn_Fb6d

[DadSchoorse]

@leucome

Is there is any of these issues that are unknown or unexpected ?

There is a reason why depth capture isn't enabled by default currently, namely that I can't tell what image layout the depth buffer is in. Nvidia doesn't have problem with that since they completely ignore layouts anyway but AMD does need them.

Using depth with MSAA hang the GPU.

yeah I probably can't fix that.

[DadSchoorse]

Here is a final build before I merge the branch: vkBasalt.tar.gz @leucome please retest MXAO in a game where depth access works.

[pchome]

@DadSchoorse Consider to add following changes before upcoming release:

• release mode, where all debug messages are silenced
• reshadeEffectsPath or even effectsPath config option

I have no patches for this changes, but I do source code hacks right before build as following:

# Ignore Debug messages
for sfile in ${S}/src/*.cpp; do
        sed -E \
                -e 's/(^ *std::cout.*$)/#ifdef LAYER_DEBUG\n\1\n#endif/g' \
                -i "${sfile}"
done

This just produces code like this:

#ifdef LAYER_DEBUG
    std::cout << ....
#endif

# TMP Hack: support reshadeEffectsPath= option
for sfile in ${S}/src/*; do
        sed -E \
                -e 's#\(pConfig->getOption\(effectName\)\)#(std::filesystem::path(pConfig->getOption("reshadeEffectsPath", "")) /= pConfig->getOption(effectName))#g' \
                -i "${sfile}"
done

Allows to do this:

reshadeEffectsPath = /usr/share/reshade-shaders/Shaders
effects = cas:effect1:effect2
effect1 = LevelsPlus.fx
effect2 = NightVision.fx

Ref: https://github.com/pchome/vulkan-gentoo-overlay/tree/db10c82bbd2e0eb208de72232ed44ead681cf12f/media-libs/vkbasalt

Multiple paths and fallback like done for the config file searching would be better.

The actual solutions may vary.

[leucome]

Mxao is a way better than it was before. There still some black spot that appear with a small sample radius. But if used with two layer of large and small sample by example 10 and 2 there is almost no black spot.

There is not that many black spot so they are noticeable in debug mode but hard to see while playing.

Adding any other effect in the chain resolve the issue and make the effect work as intended. Looks like the other effect can fill the holes, maybe. So the effect is functional when not used alone.

[leucome]

There are some shaders that use multiple techniques. I was wondering if there is a way to chose an other thechnique.

[DadSchoorse]

@pchome Disabling the debug messages would make it harder to get information on issues that people have. A better way would be to write/use a logger that can be disabled, output to stdout, or write to a file.

And the other change would require the user to have all shader files in one directory. That would be annoying when dealing with multiple shader repos.

As for whether I'll be introducing any of this for the next release, probably not as my exam season starts soon and I would like to get a release out before that.

[DadSchoorse]

@leucome currently only the first technique in the shader file gets applied, so shaders like Splitscreen.fx won't work. As for choosing a technique, you could comment out the other techniques so that the one you want is the first one.

[pchome]

Disabling the debug messages would make it harder to get information on issues that people have.

Oh, dat good 'ol "But we need that data from you to improve our product." -- developers probably. :wink: Consider to create separate debug and release builds. But I understood your point.

The second change could be reshadeEffectsPath = /shaders/path1:/shaders/path2:..., but okay, maybe I'll create an actual PR for this after release, I likely will want to use several different path for effects, so will do this part anyway.

[DadSchoorse]

Please open a separate issue for future problems. :frog: