Performance regression: long periods of extremely low performance

[vrubleg]

I have a significant performance regression in the v1.12 (fna-unstable). Sometimes (frequently enough) the game starts to work extremely slowly. These periods of low performance usually begin when I'm going to a new location, but it can also begin after pause menu, viewing inventory or some other actions. Usually, these periods of low performance end itself after some time, but in other cases it lasts very long time. I've tried to enable/disable single threaded mode and steamworks setting, but it doesn't help.

v1.11 doesn't have such problem on my machine.

I've recorded demo videos using my smartphone, so the performance isn't affected by recording software and you can easily see the difference between different versions. v1.12: https://youtu.be/ugWmL_LzGMM?t=43s v1.11: https://youtu.be/IlHDXjgUrSs?t=2m2s

My hardware: — Intel Core i5 4460 — Intel HD Graphics 4600 (driver v10.18.14.4414, 2016-03-23) — 16GB RAM — SSD Crucial MX100 512GB — Windows 7 x64

[flibitijibibo]

What happens when you turn off VSync?

[renaudbedard]

Other things I'm wondering :

• Are you using Hardware Instancing in the options? What if you toggle it off?
• What resolution and scaling mode are you playing at?

[vrubleg]

As the result of a quick test I can say that disabling VSync helps. I'll test it for 10 minutes and say if there will be some problems.

[vrubleg]

I'm using Full Aspect, 1920×1200, Hardware Instancing = On (default value). My deeper test shows that VSync = Off helps with solving this problem. Thanks.

Maybe it is better to disable VSync by default?

[renaudbedard]

I'd still not write off that bug as invalid, if VSync causes a problem I'd like to fix it if we can! If you have FRAPS or something of the like, what kind of framerates are you getting with VSync off?

[vrubleg]

Ok, I've installed FRAPS and I have gained some interesting info. Usually FPS is 60, but after switching location it falls down to ≈30, and even in pause menu it is 30 in this case. But when I go to video settings and change resolution, it returns to 60 FPS again. Sometimes it returns to 60 FPS by itself during gameplay, as you can see in my demo videos.

[flibitijibibo]

Weird! Are you familiar with apitrace by chance?

http://apitrace.github.io/

Basically what we'd need is some GL profiling data. It would work something like this, in a terminal/cmd.exe:

./bin/x86/apitrace.exe trace --api=gl FEZ.exe ./bin/x86/qapitrace.exe FEZ.trace

In the tools section there should be a "Profile" option, all we'd need is GPU/CPU times and the rest can be skipped. It'll generate a bunch of graphs, and if you trace the process you outlined it should have spikes in the 30Hz regions... that should tell us what's up, if it's GPU-bound. Otherwise it might be some CPU oddity (GC maybe?).

[vrubleg]

Does glretrace have to work in realtime for profiling purposes? On my machine it works slowly, approximately 15 FPS. Or does it extract profiling data about working of the fez.exe from the trace file using such strange method with replaying of all calls?

In any case, it shows such results (30-35FPS was at the end of trace for more than 10 seconds during recording a trace file, I had closed the game by Alt+F4): http://veg.by/z/2016-07-13-17-27-46-7097eb3d.png http://veg.by/z/2016-07-13-17-28-03-0f5a236f.png

Is there anything interesting?

[flibitijibibo]

The GL actually looks okay... there's nothing jumping out, at least. (All of that CPU time though... I really hate the Intel Windows drivers, Mesa is so much nicer!)

Either there's something in the engine that's burning cycles or the timestep is acting up and causing us to fall behind... this is where we'd have to do some .NET profiling on our end.

For now VSync is our best bet, but knowing that resetting the GraphicsDevice fixes it is also something for us to look at.

[vrubleg]

Without VSync FPS is ≈75 in rich scenes with many objects, ≈125 inside houses, ≈160 in the map mode.

What is interesting, FPS of pause menu depends on which scene is paused. It seems, that the game renders full scene under the pause menu despite it actually is invisible.

With VSync FPS is 60 usually, but in periods of described low performance it jumps near 30-35 (it is not stable). Changing resolution in settings during gameplay sometimes restores 60 FPS, sometimes not. As I can see, the probability of the problem depends on FPS of the scene without the VSync. If FPS without VSync is near 60-75, with VSync it may get stuck at the 30-35. The closer to 60 FPS without VSync, the higher the probability of this problem.

[flibitijibibo]

The dips are probably the result of VSync missing a frame and the driver jumping down to 30 as a result. This is why we aim for EXT_swap_control_tear when available, but for some reason Intel's the only vendor that doesn't provide this. The best we can do is figure out why it dips, but we're measuring dips that could be as little as a sub-16-ms jump in a single frame.

The game does indeed render out the game frame in the pause menu - I think this is for the transition graphics, and since the menus are simple enough it doesn't add too much time to the frame so it can just keep rendering the level.

[vrubleg]

The worst thing with this problem that it isn't stable 30 FPS, it permanently jumps between 30-35 FPS. The result looks awful.

Does Fez v1.11 have VSync setting? It has --force-60hz setting. It does the same thing?

[flibitijibibo]

Force 60Hz makes some changes to the timestep, but it technically breaks a few features since FEZ natively runs at 17ms. At the same time, it doesn't support VSync toggling because the game operates at a fixed timestep with a handful of modifications to the MonoGame Tick() method to get the game to cooperate more easily with the uneven update interval. For 1.12 and the FNA version we've fixed it to run at a variable drawing timestep with 17ms updates, interpolating between update skips as needed. So, when VSync is off, the updates are still at 17ms, but we draw as soon as we can with interpolation to keep things looking smooth. VSync just limits the frames so we don't burn out the CPU/GPU, but if the game hovers below the refresh rate then disabling VSync just allows the game to tear instead of hopping to 30 as soon as something goes wrong, and the resource usage will basically stay the same.

[vrubleg]

I have tested Fez v1.11 with FRAPS. It seems that this version of the game always uses VSync :) There is almost always solid 60 FPS in all scenes, sometimes it dances near 55-60 FPS (during changing of scenes).

[renaudbedard]

@flibitijibibo : Here's an idea... on platforms that don't support EXT_swap_control_tear, can we do it "manually" where if we detect framerates lower than 60, we toggle VSync off manually?

[flibitijibibo]

It's possible, but I dunno what the rules would have to be for turning it on/off intelligently. I suppose this is why id got NVIDIA/AMD to do it instead. :P

[renaudbedard]

Quick question : are you playing in fullscreen or windowed mode? With 2 monitors, I get the occasional VSync issues like you describe in windowed mode.

[vrubleg]

I'm using fullscreen mode. Also it seems that I don't have such problem in windowed mode with vsync = on.

[flibitijibibo]

Are you using fullscreen at the native resolution, or are you using a smaller res?

[vrubleg]

I'm using native resolution (1920×1200).

[flibitijibibo]

Can you try this test DLL for me?

http://www.flibitijibibo.com/FNA.dll.zip

This uses a different approach to fullscreen, see if it helps. (Also, do NOT pick a fullscreen res that is not native, or bad things will happen with this DLL.)

EDIT: Did a quick update of the FNA zip, will set the correct window size if the window wasn't the native res before moving to fullscreen.

[vrubleg]

Ok, it will be ready in 5 minutes.

[vrubleg]

No, it doesn't help. FPS still falls down to 30-35 FPS after changing locations. (Fullscreen Mode + native resolution + Vsync = On).

[flibitijibibo]

Hm, and that's with exclusive fullscreen! You may just have to run in a borderless window then; at max res it should fit in the display...

[vrubleg]

Vsync = Off is enough for me =) It works nice with Vsync = Off. Tearing isn't so annoying, it's barely visible (or even I can't notice it).

BTW, borderless window with native resolution sometimes looks like this: http://veg.by/z/2016-07-20-16-42-40-ee331be2.png It happens when I'm changing fullscreen mode to the borderless windowed mode in the settings just after starting of the fez.exe. Next changing back to fullscreen and to borderless window again works as intended.

An idea. Windowed mode could hide border if the window uses native resolution. Maybe in this case borderless windowed mode will not be needed as a separate mode =) Just a little bit of magic.

[flibitijibibo]

That offset's probably the result of the border getting removed after we've set the window position. If you switch to another res in borderless, then set the native res while still in borderless, it should center properly. The borderless on windowed fullscreen is a good thought though - I'll have to sit and think on that.

All this said, I just checked something out: On a MacBook of similar specs, but an even worse GPU, it still runs at full 60 even at resolutions higher than 1920x1200! I dunno if there's something running in the background, but my many years of experience as a Mac developer has taught me one thing: If a MacBook is performing even slightly better than another platform with similar hardware, something is VERY VERY VERY WRONG with the other machine.

[vrubleg]

On a MacBook of similar specs, but an even worse GPU, it still runs at full 60 even at resolutions higher than 1920x1200!

When Vsync is disabled, FPS is higher than 60 in all cases. (Sometimes) FPS drops to 30 only when Vsync is enabled, and it drops in cases when FPS is near 65-75 without Vsync. In the cases when FPS is higher than 75 (approximately) without Vsync, FPS is 60 with Vsync.

When FPS is dropped to 30, it may long for some time, for example, 30 seconds. And after it FPS jumps to solid 60. FPS drops to 30 usually after changing locations. For example, when you're going outside of some house.

For example: Vsync = Off. FPS inside house is 125. I'm going out. FPS drops to 75. Everything is ok. Vsync = On. FPS inside house is 60. I'm going out. FPS drops to 30-35, but after some time or after some actions FPS jumps to 60.

Maybe something wrong with drivers, I don't know :) The problem is solved for me using Vsync = Off.

[flibitijibibo]

The frame dip that comes from loading a new scene is apparently enough to scare the driver into falling back to 30, and only after a while is it convinced that it can run at 60 again. I feel like singlethreaded mode might help since it won't be doing a bunch of Wait()s, but ultimately our best goal would be to make loading times faster, or secretly disable vsync between rooms...

[renaudbedard]

Disabling VSync during loadings is something I'd be willing to try... it'll introduce tearing and might feel bizarre, but hey, if it fixes this problem...

[vrubleg]

Maybe the problem isn't so popular and it doesn't worth to add such hacks =)

[flibitijibibo]

@renaudbedard: Pushed a commit that stores when to do the vsync drop. We can just call SDL_GL_SetSwapInterval by hand with no trouble, I just don't know where we should be checking...

@vrubleg: The good news is that it's not necessarily a hack! The swap interval's something we can poll on device resets, and it's easy to jump between 0 and 1 when -1 isn't supported. We'll only be doing this when swap_control_tear isn't supported, so it's effectively a one-liner for Intel and OSX setups.

[flibitijibibo]

Pushed a commit that I think does the SwapInterval at all the right times... will let Renaud verify and push when he's home.

[vrubleg]

You could upload updated files to your website, as you did with the fna.dll.

[flibitijibibo]

Here's a build for now, but note that this is NOT the same as the Win32 build - Windows is currently built with VS2013, whereas Linux/Mac are built with MCS 4.0.5, so this could be better/worse for different reasons:

http://www.flibitijibibo.com/F12.zip

[vrubleg]

Unfortunately, it doesn't help.

I have uploaded a video where you can see FPS: https://www.youtube.com/watch?v=nnfXxKcOd00

As you can see, when FPS is 30, you can see heavy twitches while moving. It means that time between frames is different every time, and this difference is high enough, because you can easily see that these 30 FPS isn't as smooth as it could be.

I have tried to run the game (v1.11) on my old 7-year notebook with poor old GPU. I see that FPS is low, but it is visible that time between frames is stable, it is playable. For some reason Steam doesn't accept the password for the fna-unstable on my notebook. Maybe the password is changed?

[flibitijibibo]

The password's still the same.

It almost looks like something entirely different from normal VSync fallbacks is happening - do you happen to have a spare hard drive lying around? I'd really like to see if Linux/Mesa exhibits this issue on the same hardware.

[vrubleg]

Unfortunately, no. I'll try earlier versions of the GPU driver.

[vrubleg]

OMG.

I've installed driver v10.18.10.3907 (2014-08-05). Previous version is v10.18.14.4414 (2016-03-23). Vsync = Off. Now I have 200 FPS in house (it was 125), 115 FPS outside house (it was 75), 90 FPS when outside house + pause menu. Vsync = On. Solid 60 FPS inside house and outside house. But when I open the pause menu (and I outside house), FPS drops to these shitty 30-35 :)

Conclusions:

1. Fez has dramatically better performance on the older Intel driver.
2. FPS drops to 30-35 with Vsync = On in the same situations, where FPS is lower than 100 (approximately) with Vsync = Off.

Maybe it is just a driver problem also.

[flibitijibibo]

Hm, so it probably is the driver overreacting to microstutter then. Not much we can do about that :/

[renaudbedard]

I think I'll just mention in the release notes that Intel drivers have an issue with VSync in the game that we can't do anything about, and that it's preferred to play with VSync off.

Actually I think I'll do a vendor string detection and default VSync to off on Intel,... does that make sense?

[vrubleg]

Maybe other Intel GPUs work better. It is possible that this problem affects only my PC and even other computers with Intel HD 4600 will not have such problem, who knows.

It would be nice to have some stats for doing such decision. Maybe it's time to do some public beta version? :)

[renaudbedard]

We've been in private beta for so long... the "public beta" will be the release of the update. If we hit problems we'll do another, smaller hotfix update.

[vrubleg]

As for me, I think that it will be enough just to mention it in readme =)

[vrubleg]

The borderless on windowed fullscreen is a good thought though - I'll have to sit and think on that.

Just a reminder =)

[renaudbedard]

Added a note the changelog in a "known issues" section, which will be part of the Steam update changelog as well, to inform Intel iGPU users that disabling VSync could result in a better experience.

@flibitijibibo I'll let you deal with the other part (which should be another issue really)

[vrubleg]

@renaudbedard, ready: https://github.com/renaudbedard/fez-1.12-issues/issues/122

[renaudbedard]

Thanks!