load/load_fast too many times degrades performance

[gurka]

Video showcase: https://www.youtube.com/watch?v=Oov3vFclJdc

Basically, if you load saves too many times without restarting the (whole) game, eventually performance will become very poor. This is very noticeable in some areas/maps like on metro. You can see in the video linked above that I get around 150fps when having just started the game and loaded the initial save. Then after a couple of loads of my quick save it goes down to 85fps, and after a few more down to 50fps. Note that a full restart of the game is required to get back to normal.

This can also happens when playing the game normally (i.e. not spamming load or load_fast) as it only requires 5-10 loads before performance starts to degrade. Playing the game from start up until metro in one go will definitely trigger this.

I've tried to investigate this but haven't been able to pinpoint the exact root cause yet. One thing that I found is that a snd_restart (e.g. restarting the sound subsystem) helps to restore performance in some areas (but not all). But only when I reproduced it by playing from start to metro. It doesn't help when I spam loads to trigger the issue.

I also profiled using the built-in vprof tool, but there is nothing really that stands out when comparing normal performance vs. degraded performance. All the same functions is at the top of spent time, but they just take more time to run for some reason.

There might be some more info in the discord server, channel #speedrunning from 2021-04-22 and 2021-04-23, as I discussed this issue there as well.

[gurka]

Did some additional testing. The number of loads needed until the degraded/poor performance happens seem to depend on what map you are on, but it is consistent. I did a quick test on this by:

1. Starting INFRA
2. Starting a new game on chapter n
3. Wait until I have control of Mark then make a quick save
4. Keep loading the save until you notice that it takes more time to load than usual *
5. Exit the game completely and repeat

* With normal performance, e.g. after just starting the game, my loads load almost immediately (<0.2s). When the degraded/poor performance hits a load start to take around 1-4 seconds to load. So it's very easy to tell when the bug has been reproduced.

Results:

• Chapter 1: 7 loads
• Chapter 2: 19 loads
• Chapter 3: 28 loads
• Chapter 4: 6 loads
• Chapter 5: 8 loads
• Chapter 6: 89 loads
• Chapter 7: 12 loads
• Chapter 8: 28 loads
• Chapter 9: 27 loads
• Chapter 10: 11 loads

I think it's pretty safe to say that the number of loads required to reproduce this bug directly depends on the size of the map, or maybe not the size but the number of entities/etc in the map?

I also did some FPS tests to see how much the performance actually degraded when this bug occurs. When doing this I also noticed that Multicore Rendering plays a role in this, so I tested a few combinations of different locations and Multicore Rendering settings. Normal performance is after just starting the game and Poor performance is the degraded performance, after loading a quick save too many times.

The locations I tested were all in the metro map. Stairs is the stairs just after the hot plate that you need to turn off. Archive H is looking into the archive room, specifically looking directly at the letter H. Protest is looking at the people protesting outside the metro.

Multicore rendering, normal performance:

• Stairs: 250 fps
• Archive H: 165 fps
• Protest: 90 fps

No multicore rendering, normal performance:

• Stairs: 190 fps
• Archive H: 105 fps
• Protest: 35 fps

Multicore rendering, poor performance:

• Stairs: 200 fps
• Archive H: 40 fps
• Protest: 33 fps

No multicore rendering, poor performance:

• Stairs: 180 fps
• Archive H: 90 fps
• Protest: 20 fps

Degradation, multicore rendering, normal -> poor:

• Stairs: 250 -> 200 = 50 fps decrease
• Archive H: 165 -> 40 = 25 fps decrease
• Protest: 90 -> 33 = 67 fps decrease

Degradation, no multicore rendering, normal -> poor:

• Stairs: 190 -> 180 = 10 fps decrease
• Archive H: 105 -> 90 = 15 fps decrease
• Protest: 35 -> 20 = 15 fps decrease

I'm not sure exactly how to summarize this, but it at least seems that multicore rendering gets much worse when the bug occurs as the fps decrease much more compared to when the bug occurs and you don't have multicore rendering enabled. Another interesting thing (but unrelated to this bug) is that I get 190 and 105 fps with multicore rendering disabled with normal performance but only 33 fps when looking at the protesters. I guess the huge number of NPCs is one reason for this.

[gurka]

Another thing I did not clarify earlier. With "loading a quick save" I mean using the load or load_fast commands, and both of them can be used to trigger this bug. Using reload to reload the map and the last created save does not trigger this bug.

I can't find any information at all on load_fast online, so I assume this is implemented in INFRA and not in the Source engine/SDK. Maybe it's also the case that you override/alias load to simply do a load_fast, as they seem to do the exact same thing. load in other Source game seem to reload the whole map, which INFRA doesn't do.

If this is the case then it should be related to how you implemented load_fast in INFRA, and that the bug is somewhere in your implementation and not in the Source engine/SDK. This also makes sense as I haven't been able to reproduce this bug or found any reports of similar bugs in other Source games. Unfortunately this also means that I won't be able to troubleshoot this further myself...