Performance regression on some CPU's

[Asinin3]

While Yahfz and other users with newer Intel CPU's reported large performance improvements, I lost small amount of performance from #11102 in a number of games. Some users on Skylake reported similar performance regressions. Note that Yahfz noticed worse performance in mandelbrot spu test

Before: After:

[Nekotekina]

Need to bisect LLVM repo in hope to find offending commits, but it's a tremendous amount of work. If it's found, however, I will try to workaround the issue.

[Asinin3]

If it's too hard don't worry that much. In my initial tests the fps difference was larger. But I've since re-tested and uploaded new images where the difference is smaller. I think we should keep this open for awhile to see if other users experienced the same thing. And I will try and test a few more games later.

[Nekotekina]

It's rather time consuming - someone with affected CPU must do the bisection, recompiling everything in the process. But it should be worth trying.

[Augusto7743]

Testing in a "old" FX-6300 have good results to figure any issue. Since when LLVM was updated to version 13 I see a good speed improvement in PES2018. That game use 346 PPUs. In several previous version PES2018 run in less of 17 FPS. After LLVM update run between 17 to 28 FPS almost all time above of 21 FPS. After 0.0.19-13041 that same game the performance is less 20 % framerate below 17 FPS and others games even running in 100 % speed are using more CPU. I not understand exaclty if is related with SPUs, but is happen graphics issues in few games being Motorstorm Pacific Rift with wrong graphics in vehicles and a huge slowdown framerate. In PES2018 in first and second load after cleaning PPUs and SPUs when in begin gameplay doing kick off the player kick the ball, but the ball not move and showing that need to do again another kick off also not moving the ball even the player kicking the ball how if anything is breaking the game code. Both bugs in Motorstorm Pacific Rift and PES2018 had happened exactly in same RPCS3 release. I see also a good speed improvement when compiling PPUs. Not demading improvement. Only reporting.

[Linear524]

Confirming. I have and Intel Xeon CPUs E5 2690 V4 and Xeon E5 2680 V3 Both models shows speed regression ~3 to ~5 fps loss in many titles (TLoU, GoW, GoWA, MGS collection, SotC, ICO, Killzone 2,3)

[Asinin3]

@Linear524 Screenshots with avg fps counter and log please.

[Augusto7743]

Some games are having a long time creating SPU cache in gameplay thus having less game performance. Perhaps the performance regression not is because SPU compiling ?

[Asinin3]

No, nothing to do with modules compiling. Also don't comment unless you can confirm that the SPU Cache is worse in the build directly before and after the linked PR merged.

[Augusto7743]

I have compared current version with some previous versions and see clearly more SPUs modules being compiled in current version thus using CPU resources. Not saying if is good or bad details, but only reporting that detail. Teenage Mutant Ninja Turtles Out of the Shadows in first startup in few minutes all SPU modules work are done and in current version all work was done in several minutes even doing the same act (start game and not move for a long time waiting SPU compiling stop). Now if in log details not are listed SPU compiling the CPU is free for others tasks.

@Asinin3 When having slowdown is being done SPU compiling ?

[Asinin3]

I already said no, it's not caused by SPU modules compiling. This issue is about performance loss. Not longer startup times from more modules compiling. I tested around twenty games, most performed the same, some lost 1-3 fps (for me, other people got more performance as mentioned in OP).

[Augusto7743]

Yes have games running much more slow now. You have noted if the games running with performance loss have high number of PPUs threads ? PES2018 is much more slow now than a version before LLVM was updated to version 13.

[Xcedf]

Made a try, removed 5000 commits, and speed I've lost is back, but damn, recompiling LLVM every time is torture, will need lots of time to find offending commit, let's hope it's a single one.

[Nekotekina]

It's good to know you are managing to get some results, and 5000 seems like relatively recent stuff.

[Augusto7743]

Games using high number of PPUs linked are more slow since when LLVM was updated to 13. Not saying LLVM is the problem , but was exactly before LLVM was updated. Very much optimization is done about SPU, but I not understand if is possible optimize games using high number of pre compiled PPUs since all PPUs not are exactly running in same time or even optmize the "PPU linking" feature.

[Xcedf]

@Nekotekina found the commit that stole my performance https://github.com/RPCS3/llvm-mirror/commit/2211738092a404d64e16edd90ac6ca4b815fea45 reverting it helps

[Nekotekina]

Should be in master if I didn't push too early. Don't close since there are possibly more problems.

[Augusto7743]

Tested 0.0.19-13133 some game look how the "performance regression" was partialy fixed because after all SPU are compiled the performance have returned to normal , but SPUs are having much more time to compile or now RPCS3 compile more new SPUs or yet LLVM 13 is more slow to compile

[Linear524]

Wow ! Latest revisions have fixed all speed regressions for my hardware ! 👍 I'm using Intel Xeon CPUs E5 2690 V4 and Xeon E5 2680 V3. Both with 32GB 4channel memory and GTX1080. All my titles are fullspeed back again :) And even XMB/VSH now boots again ! 🥇 Thanks Nekotekina, Xcedf, KD-11, Whatcookie and all other RPCS3 Devs !

(I think this issue might be closed now, because other people also noted about recently fixed speed regression on their hardware)

[Augusto7743]

Here tested with few games and the result is performance was fixed.

[Yahfz]

The llvm regression was fixed. Asinine needs to confirm with his cpu and close it.

[AniLeo]

@Asinin3 for retesting

[Asinin3]

Performance regression still there on 13133 for me.

12975 = 43-45fps 12976 = 38-40fps 13133 = 38-40fps Realm of Hades save spot. I tried re-doing this test a few times on each build but it was always a consistent loss of ~5fps.

[Nekotekina]

Could be another issue with upstream llvm.

[Xcedf]

I'll search again, if it here on 12900k it should be obvious

[Xcedf]

@Asinin3 Tested Realm of Hades stage, LLVM target cpu - skylake, both 12975 and 13133 same framerates, perhaps only yours CPU is affected now.

[Asinin3]

Not sure whats going on then, might need some Ryzen users to test.

[Ordinary205]

I'm a Ryzen user, and I can confirm that my performance received about 2 or 4 FPS loss after 12975 build. So I tested it on InFamous.

~~12975 build: (around 25-27 FPS) ~~

~~12976 build: (around 23-26 FPS) ~~

~~13133 build: (around 23-26 FPS) ~~

Logs: RPCS3.log.gz RPCS3.log.gz RPCS3.log.gz

Btw my CPU model is Ryzen 9 5900X, incase if anyone wants to know.

[Outdated]

[Yahfz]

@Ordinary205 What do you mean? Your AVG fps is the same on all 3 screenshots and your 1% lows are actually higher on the latest master build.

[Ordinary205]

@Yahfz I understand that the average FPS said its almost the same thing on my screenshots, but to my experience I sometimes lose 2 FPS on those two builds. The latest build does allows me to go higher than 30 FPS if I dont move, and I'm pretty sure its because when I turned the camera around I lose/gain a bit of performance. I might have to test InFamous again without moving, and I need to test another game with 12975, 12976, and 13133 builds so I could tell if the performance ACTUALLY decreases, increases or has same performance on my CPU.

[Ordinary205]

Here are some weird performance results on two games that I tested by having a Ryzen CPU:

[Tested on InFamous] Build 12975: Average 22.4 FPS

Build 12976: Average 24.2 FPS

Build 13133: Average 25 FPS

Recent build: Average 32.4 FPS

[Tested on Demon's Souls] Build 12975: Average 185.2 FPS

Build 12976: Average 178.9 FPS

Build 13133: Average 199.6 FPS

Recent build: Average 217.2 FPS

Logs: (InFamous) RPCS3.log.gz RPCS3.log.gz RPCS3.log.gz RPCS3.log.gz

Logs: (Demon's Souls) RPCS3.log.gz RPCS3.log.gz RPCS3.log.gz RPCS3.log.gz

[kd-11]

@Ordinary205 The build you say is 21fps shows 25fps in the screenshot. How did you get the 21fps value? This is what is causing the confusion.

[Ordinary205]

I accidently wrote 21 fps, my bad.

[kd-11]

Infamous is RSX benchmark anyway, so it's not affected much. I'll check gow3 on my system to verify if this is still a problem.