RPCS3 Fatal Error when Stop [NPUB30320] X-Men with ReShade installed, Vulkan renderer

smbhax commented 2 weeks ago

Quick summary

RPCS3 crashes with a Fatal Error if I click the master window's Stop button while running [NPUB30320] X-Men in Vulkan with ReShade's "Legacy effects - AdaptiveSharpen.fx" installed and active.

Details

Run ReShade_Setup_6.1.1_Addon.exe, Browse to rpcs3.exe, choose Vulkan API, select Legacy effects - AdaptiveSharpen.fx
Start RPCS3
Start [NPUB30320] X-Men, having set its GPU Renderer to Vulkan
Press Home to open ReShade menu
Check the box for AdaptiveSharpen
Press Home to close the ReShade menu
Click the Stop button on the RPCS3 master window
If RPCS3 doesn't crash within a second or so, run X-Men again, and click the Stop button while the initial Konami logo screen is visible--I've had the crash very rarely not happen if Stop is used later, but it always seems to happen when Stop is activated while the Konami logo screen is visible

If instead of using the Stop button, I press the PS button on my DualSense pad to bring up RPCS3's "Home Menu" and choose "Exit Game," the crash does not occur.

The crash also does not occur if running X-Men and ReShade under the OpenGL renderer rather than Vulkan.

Sample crash window below; the address and Module name vary--I've seen module "d3d11.dll" listed instead of "avutil-58.dll":

RPCS3: Fatal Error

Unhandled Win32 exception 0xC0000005
Segfault reading location 00007ff8b0d981f8 at 00007ff8b69513b7.
Thread id = 12232.
Instruction address: 00007ff8b69513b7.
Function address: 00007ff8b6951200 (base+0x41200).
Module name: 'avutil-58.dll'
Module base: 00007ff8b691000.
RPCS3 image base: 0000000000010000.

Title: X-Men??? [NPUB30320] (emulation is stopping)
Build: 0.0.32-16471-0a956906 Alpha | master
Date: 2024-05-15T12:04:15

xmen_reshade_crash

(ReShade is an external program so I'm not sure this will be supported, but I see ReShade mentioned in numerous issues here as a means of achieving desired effects that RPCS3 doesn't supply itself. In this case I wanted to use ReShade's AdaptiveSharpen.fx for X-Men because the PS3 port of X-Men is very blurry, which AdaptiveSharpen.fx fixes.)

Attach a log file

RPCS3.log

Attach capture files for visual issues

No response

System configuration

Operating System: Windows 11

CPU: i7-10750H

GPU: GTX 1660 Ti Driver version 535.98

Other: 16 GB RAM S271HL NVIDIA High Definition Audio

Other details

No response

kd-11 commented 2 weeks ago

I'll have to reject this one. It's not crashing in our code. You can see windows points to the faulting module as avutil-58.dll but we don't have avutil integration, reshade does.

smbhax commented 2 weeks ago

Fair. I'm now getting it more often pointing to module d3d11.dll, but I'm on RPCS3's Vulkan renderer so that again is likely not directly from RPCS3.

Crash also occurs when using "SweetFX by CeeJay.dl - LumaSharpen.fx" to achieve a similar sharpening goal.

Besides using "Home Menu - Exit Game" instead of Stop to leave the game, another workaround is to uncheck the .fx in ReShade's Home menu before Stopping the game, since you likely won't want it active in whatever game you play next--and this avoids the crash.

smbhax commented 2 weeks ago

The crash only occurs when using the Vulkan renderer; setting X-Men on a custom config to launch with the OpenGL renderer--and correspondingly setting ReShade to run under OpenGL--avoids the crash.

This also has the possibly convenient side-effect of keeping ReShade active only on PS3 games in RPCS3 that are set to the OpenGL renderer; if all your other PS3 games in RPCS3 are using Vulkan, for instance, then ReShade, when it has been pointed to OpenGL in its own Setup, will only be enabled in PS3 game that is using the OpenGL renderer--so you won't have to bother toggling the .fx you're using there off/on when switching back and forth to other games.

RPCS3 / rpcs3