rsx: Enable multisampling

[kd-11]

• Adds MSAA support (vulkan only for now since almost all testers will run vulkan)
• Completely refactors out D3D12 as it was holding back progress with shared code.
• Minor improvement to fp decompiler input modifiers.

[Xcedf]

Resistance 2 AA Auto E {RSX [0x6c0e4b4]} RSX: Resolve shader:

version 420

extension GL_ARB_separate_shader_objects : enable

extension GL_ARB_shader_stencil_export : enable

layout(std140, set=0, binding=0) uniform static_data{ ivec4 regs[8]; }; layout(set=0, binding=1) uniform sampler2DMS fs0; layout(set=0, binding=2) uniform usampler2DMS fs1; layout(pixel_center_integer) in vec4 gl_FragCoord;

void main() { ivec2 out_coord = ivec2(gl_FragCoord.xy); ivec2 in_coord = (out_coord / regs[0].xy); ivec2 sample_loc = out_coord % ivec2(regs[0].xy); int sample_index = sample_loc.x + (sample_loc.y * regs[0].y); float frag_depth = texelFetch(fs0, in_coord, sample_index).x; uint frag_stencil = texelFetch(fs1, in_coord, sample_index).x; gl_FragDepth = frag_depth; gl_FragStencilRefARB = int(frag_stencil); }

F {RSX [0x7075ab4]} RSX: class std::runtime_error thrown: Verification failed (e=0x7f): (in file c:\projects\rpcs3\rpcs3\emu\rsx\vk\vkrendertargets.h:334)

AA Disabled just

F {RSX [0x7075ab4]} RSX: class std::runtime_error thrown: Verification failed (e=0x7f): (in file c:\projects\rpcs3\rpcs3\emu\rsx\vk\vkrendertargets.h:334)

Same for Killzone 2

[kd-11]

Resolve shader message is unrelated. I just left it as log_error in case of compilation errors.

[Xcedf]

Well it's just verification fail then, and even if i disable it the game still ends with some error

[kd-11]

Yea, looks like some files on my end are corrupted; I'm reinstalling some demos and re-running more tests. The verification crash at least is reproducible.

[arabek]

Killzone 2 fails for me too, even after nuking all the caches.

[kd-11]

Fixed the VkRenderTargets.h crash caused by only testing MSAA x2 during development. MSAA x4 was completely broken which is what killzone 2 is using.

[kd-11]

Several other issues identified by testers and under investigation (avoid repeating these):

• Performance slowdown in ni no kuni (even compared to early MSAA-enabled builds) Fixed.
• Graphics corruption in ni no kuni when MSAA is disabled. Fixed.

[Xcedf]

Now that's what i always expected Resistance 2 (Native res) master

[MarioSonic2987]

Comparisons:

God of War III (E3 2009 Demo) - Minor improvement

 

Midnight Club: Los Angeles (200% scaling)

 

Racedriver: Grid (200% scaling)

 

Red Dead Redemption (200% scaling)

 

Tekken Tag Tournament 2

 

Uncharted 2: Among Thieves

 

[Kravickas]

Crysis Approve 👍

[stride21]

Master: PR: Master: PR:

[MsDarkLow]

MSAA seems to work as intended in 4. However, I noticed a difference in colors in Yakuza 3, 4 and Ishin, they seem to show up incorrectly.

Yakuza 3

(Doesn't seem to support MSAA) Master  PR 

Yakuza 4

Master  PR MSAA  PR 

Yakuza Ishin

(Doesn't seem to support MSAA???) Master  PR MSAA  PR 

[e1489]

;=; how do I try this?

[Artins90]

;=; how do I try this?

Click show all checks below. Select details near appveryor Open artifacts and dowload the 17MB file The antialising option is in the GPU options section, select auto to activate it.

[raven02]

D3D12 renderer selection box will be removed as well ?

[AniLeo]

d3d12 is not yet fully removed

[kd-11]

@MsDarkLow

However, I noticed a difference in colors in Yakuza 3, 4 and Ishin, they seem to show up incorrectly.

Maybe show the differences clearly? I cannot spot any differences since I'm not familiar with those games. There are some tweaks to the fragment decompiler based on hardware testing on a PS3, but I still have doubts as to the accuracy; if the commit is wrong compared to PS3 I can revert it from the PR and submit it separately at a later date.

[anseesaw]

What does this mean for d3d12，Improve or delete？

[RainbowCookie32]

DX12 isn't being improved. I'd say that we are one step closer to removal with this PR (I think)

[MarioSonic2987]

TTT2 still crashes in Fallen Garden with MSAA enabled: Log TTT2 Broken 2

Killzone 2 has flickering shadows and blurred graphics.

Master:

This PR:

[MsDarkLow]

@kd-11

Maybe show the differences clearly? I cannot spot any differences since I'm not familiar with those games. There are some tweaks to the fragment decompiler based on hardware testing on a PS3, but I still have doubts as to the accuracy; if the commit is wrong compared to PS3 I can revert it from the PR and submit it separately at a later date.

I got a better scenario with Ishin which is way more clear. Took screenshots in the main menu including how it looks on a PS3.

Yakuza Ishin

PS3  Master  PR (both OpenGL & Vulkan look like this) 

[kd-11]

@MsDarkLow I changed the decompiler again about an hour ago; need new comparison with the updated code.

[MsDarkLow]

Looks relatively the same.

[kd-11]

I'll have to check it in a regression tracker; its highly unlikely to get resolved in this PR.

[Artins90]

Looks relatively the same.

Maybe it's hard to notice without a screen with good color reproduction. In the PS3 and master images there is a yellow glow on the character and the blue of the sky is less dark. Maybe it has something to do with post processing.

In this other image the top part of the wall on the left side of the screen has a faint blue tint: https://user-images.githubusercontent.com/9060436/59066374-d3bf7780-88a6-11e9-9270-fad21f8c8e6b.png

[Margen67]

Water isn't rendered when anti-aliasing is enabled in Bigduck Sample: On Off

[Sandvichero]

Nier has incorrect colors in this pr. LOG

Master:

PR:

[raveskirza]

Regression in SC5

Characters are messed up with AA on/off

In Master, Characters look fine

[MarioSonic2987]

RDR has missing lamps with MSAA enabled.

Images

**Master**:  **PR (MSAA disabled)**:  **PR (MSAA enabled)**: 

[AizerMortenort]

Getting Exception errors RPCS3.log

[AizerMortenort]

Here is the error of the crash i get

[kd-11]

I have dropped the shader decompiler fixes for now. I'll submit in a separate PR as the issue seems to require more testing. Should fix the graphics glitches in Yakuza and Soul Calibur back to matching master.

[darkdragonz12]

Soulcalibur V Fix lastcommit

[GreyWing3]

Eternal sonata over-blooming with anti-aliasing on. No issues with it off

[MarioSonic2987]

RDR now renders fine without MSAA, but lamps are still missing with MSAA.

RDR regression

 

Killzone 2 regression is still present (is caused by MSAA on), but renders fine (like master) without MSAA.

Killzone 2

 

[kd-11]

Pushed a few more fixes. 3D and cubemaps (reflections) are still broken in some corner cases; I'm working on that.

[MarioSonic2987]

Lamps are now fixed in RDR with MSAA.

Killzone 2 regression is now fixed.

But Racedriver: Grid still has this "repeated" image with MSAA and no longer reaches title screen without MSAA:

Log Grid Broken (w/o MSAA)

[Aurez]

Thanks! Most of my games are working well and better than before :) But TTT2 is still crashing

[kd-11]

@MarioSonic2987 I don't have the repeating image.

[GreyWing3]

Eternal sonata fixed

[MarioSonic2987]

RenderDoc Grid (Broken)

[kd-11]

@MarioSonic2987 Retest please.

[MarioSonic2987]

Now looks correct with MSAA and can reach title screen without MSAA.

[kd-11]

@Aurez Confirm if TTT2 crash is fixed.

[legend800]

Crowd is missing again on Backbreaker Vengeance. I actually prefer it this way since they're only half rendered on Master, but just letting you know since it's an accuracy thing.

PR (AA on/off):

Master:

[Margen67]

Bigduck Sample is now rendering correctly with MSAA enabled.

[Aurez]

Just tested again and TTT2 is still crashing RPCS3.zip

[Parthorisian]

msaa Disabled and Auto SH: Downpour [BLUS30565]: master - goes ingame(sometimes)/menu PR - nothing SH_Downpour_RPCS3.ZIP WET [BLUS30403]: master - goes ingame(sometimes)/menu PR - nothing WET_RPCS3.zip Ninja Gaiden Sigma 2 Ninja_Gaiden_Sigma2.zip

[kd-11]

Those issues seem unrelated. The last one is a crash in main_thread for example. Also, SH Downpour works just fine.

[kd-11]

@Aurez Install the vulkan SDK from here: https://vulkan.lunarg.com/sdk/home#windows After that, go to debug settings for the game and enable "debug output" option and run the game. You'll know if it works because your fps should be terrible. Play the game as usual, when it crashes, attach the logfile here. It's the only way to actually debug a DEVICE_LOST error.