Performance issues 3.2.3 rc4

[Reapetitive]

Godot version: 3.2.3 rc4

PC, API: Windows 10 64bit, ryzen 1700x, nvidia geforce gtx 1070, gles3

Issue description: Performance issues in 3.2.3 rc4 compaired to 3.2.2 stable.

I am making a big 2D, physics and graphics heavy game, that uses pretty much everything Godot has to offer and some more.

After some testing i see a huge performance diffirence compaired to 3.2.2.

The heaviest rooms went from 60-100+ fps to 30-60.

3.2.2:

3.2.3 rc4:

[Reapetitive]

@lawnjelly RC2 - 200 fps, RC3 - 30 fps, exactly

[akien-mga]

The best would be if you could bisect between RC2 and RC3 to identify the commit that triggers that regression. There's only 10 days worth of commits and not a lot that should impact rendering at all, so it's a bit surprising, but it might be a side effect of something else.

You can use this tutorial to see how to bisect regressions: https://docs.godotengine.org/en/latest/community/contributing/bisecting_regressions.html

RC2 is commit 9918fd722e3e555cb174f9806cdb38b6e8b0c2b7. RC3 is commit 4f891b706027dc800f6949bec413f448defdd20d.

I would also recommend start with custom builds of RC2 and RC3 to verify what the expected "good" and "bad" states are in custom builds, as the build options impact the performance too (so if you're bisecting with a debug build, it will be slower in any case). I'd suggest building with target=release_debug at least to ensure that you have optimizations (default is target=debug).

[lawnjelly]

If you go to this page: https://github.com/godotengine/godot/commits/3.2

You can click on the green ticks and download artifacts (builds) for most of these commits, and work out which caused your performance regression.

In terms of 2d rendering I've only done one PR #46163, to do with light masks. If you are using light masks (Light 2d set to mask mode) that could possibly slow you down as it disables batching those masked items. My only other one I think was merged before RC2 (#45843).

That's assuming you are using 2d rendering, it could have been a 3d rendering change if you are using that, or any of the other non-rendering commits.

Ah - Akien slightly beat me, but gist is the same.

[Reapetitive]

@akien-mga @lawnjelly

220f24c

The performance issues start at this commit.

@lawnjelly btw, thanks, it was very helpful to know, that i can download the builds directly. :)

[akien-mga]

CC @RandomShaper @hpvb

[hpvb]

@Reapetitive That commit overhauled the way that multithreading works in Godot. Most importantly it corrects some behavior that previously was incorrect and could lead to data corruption. However it appears that it also has a pretty big impact on whatever it is that you are doing in your game.

If you would be willing to confidentially share your game project with me in its entirety I'd be happy to try to figure out why this regressed your game by so much. Our usual testcases showed a marked improvement in performance in other cases we tried.

You can email me at hp@tmm.cx or just upload the project (please leave out the .import directory :)) here; https://cloud.tmm.cx/s/Q7YpkCfTRr3cF5a (This is a file drop only location. Nobody but me will be able to download your project)

[RandomShaper]

I think it'd be good to try on different compilers to tell if the atomics are per se expected to harm performance that bad (I don't expect that to be the case, though) or if the compiler/library used to build is doing something stupid.

@Reapetitive, would you be up to running your game on an editor build I upload here to see how the performance compares?

Aside, @hpvb, I'm happy you are looking into this. Anyway, you know I'm here to help if you want me to check or discuss something.

[Reapetitive]

@RandomShaper Sure! @hpvb i will look into it, thank you! If i find out what is goind wrong, i will let you guys know, if not, i will give you a go on trying it with my project.

[Reapetitive]

@RandomShaper @hpvb ok, i found the issue. I am using a custom algorithm to animate cables and chains and some sort of soft bodies on base of line2d. Every single one of this objects generates a thread to calculate the motion. I put the method out of the thread and it fixed the performance issues. There is literally only "thread.start(self,"do_stuff", delta,1) thread.wait_to_finish()" happening every second physics frame, while "do_stuff" only does the math.

[RandomShaper]

Thanks for the update. It's good to see that the problem is not due to subtle stuff in the new atomics or something like that. However, it's puzzling that Thread can worsen performance like that from RC 2 to RC 3.

Just to be sure, did your game work better in RC 2 even with the threaded approach you described?

[Reapetitive]

@RandomShaper I'd say it runs equally well - RC2 with threads = RC3 without them. From some quick tests. There is a lot going on in this game, i wouldn't want to lie and it is hard to tell, but it seems to me, that the performance of RC3 is generally higher. For example, if only the player and hud is on screen, i get around 500 fps in RC2 and 700 fps in RC3, not much of a diffirence, but still something. I tested a specific spot on RC2 with and without the threads, and RC3. RC2 threaded = 106 fps, RC2 nonthreaded = 90 fps, RC3 nonthreaded = 106 fps.

[Reapetitive]

Another things i experience, but kinda let it slide:

The performance of my game sometimes gets randomly halfed on the start and stays like that until i restart the game.

Random but steady stuttering. It appears sometimes, seemingly random and keeps occuring for, again, seemingly random time with a set speed, like every second it lags once.

[hpvb]

@Reapetitive It's difficult to control many threads, I can't tell you exactly why your game has this behavior obviously but I would suggest that if you can keep the number of threads stable, and if you can keep your threads running rather than starting and stopping them constantly you will see more consistent results. This isn't really a Godot thing, this is more or less true on any program. For instance on Windows creating and destroying threads is way, way more expensive than it is on Linux.

I'd recommend trying to eliminate all threads you don't need, and to make worker threads for things that need to stay on threads, and feeding them work rather than starting a new one for each piece of work.

[Reapetitive]

@hpvb thank you! I have never worked with threads before this project, and to be honest, i didn't really get it to work anywhere, except for this one method. It seems, i don't really need it now anyways. Might be i ll need to look into threads and get a hang of it some day, but for now i guess i ll be fine, performance wize. Especially when godot 4 and vulkan do it's magic. :o

[Reapetitive]

Random but steady stuttering. It appears sometimes, seemingly random and keeps occuring for, again, seemingly random time with a set speed, like every second it lags once.

This one was my bad i guess. I iterated through very big arrays, didn't even realize they were getting so huge, 2000 entries, and made operations with the entries.

This iteration happens exactly once a second.

Optimized it, the amount of entries before:after is 20:1. Doesn't stutter anymore.

[akien-mga]

So in the end it seems like the issue might have been overuse of threads which are more costly on Windows since Godot 3.2.3? But in the end the issue was mitigated by not using so many threads which we not actually helping performance? If so, should this be closed?

[Reapetitive]

The game seems to run pretty fine, except for the costly shaders + light2D, but it has nothing to do with this topic. So yeah, thank you guys!