[Bug/Performance Issues] GeneralImprovements causes significant ship lag

[ajsglist]

Hey there,

I've been working on narrowing down the culprits behind ship lag, and unfortunately it seems that GeneralImprovements is a major contributor.

This was most obvious/reproducible on March. A few feet from the ship, the ship starts updating and causes major performance issues. I have benchmarked this in a few simple cases, which I can only really illustrate via the apparent FPS of a few gifs I've collected.

In the vanilla case, there are no performance issues at the edge of the ship.

https://i.imgur.com/1zAZY5s.gif

In the modded case, with GI enabled, beyond the edge of the ship (no ship updates occurring --> no performance hit)

https://i.imgur.com/T9hstzp.gif

In the modded case, with GI enabled, at the edge of the ship, revealing major performance issues

https://i.imgur.com/6O9vIcN.gif

In the modded case, with GI disabled, at the edge of the ship, confirming GI is a major culprit in this performance issue

https://i.imgur.com/gLh1G1z.gif

GI enabled also appears to cause a significant performance hit on the ship. I suspect the update rate of the monitors (or individual monitors being set up to poll too frequently or in a resource-intensive way) contributes to this. I would love if you could do a performance pass on the monitor system, because at present they are causing appreciable performance issues.

Thanks

[Shaosil]

I finally got around to hooking up a profiler and worked on deep profiling it in several ways for about an hour yesterday. Basically, I tried unmodded, then with and without GI, and in both cases tested in orbit, while landed, inside the facility, outside the ship, inside the ship, etc. I even used a couple other people's profile codes that had 200-300 mods, with GI enabled and disabled.

Unfortunately, the end result was that I could not only see no FPS differences in any of the above cases, but also checking through countless frames in the profiling timeline, I didn't see a single instance of GI taking more than 0.04ms to process anything.

That being said, I'm not an expert at profiling and I could be missing something. But again, I didn't notice a difference in frames when toggling GI and doing the same tests, even with the 300 mod profile active. So I'm going to have to close for now, until someone can provide me with a screenshot or other proof of a specific stack trace from GI being the cause. Sorry!

[ajsglist]

The best I can think to provide is a profile code where GI-enabled manifests extreme performance issues on the ship but doesn't exhibit this behavior with GI disabled. It's probably worth noting that the performance issues will be worse after playing for a few days and grabbing some scrap, so I'd recommend considering that aspect as well. All of this has been reproduced in single player, so there's no trick or anything required with multiplayer lobbies in order to reproduce.

Beyond that, all I can say is the monitors as they're currently set up in that profile seem to contribute to the major performance issues, and the performance issues have been observed by multiple people with an apparent range of hardware (no common CPU/GPU/etc as far as we can tell)

I love GI (and have been reporting bugs/issues with it all year), but in its current state it simply has too high a performance cost to be usable. Regardless, thanks for taking an initial look at the issues, even if you didn't manage to track anything down.

If there's anything else you'd like in order to investigate, just let me know

019106ed-a6ce-ccee-d21c-427d61240d4d

[Shaosil]

Alright I have some news on this. I did find that in many cases, especially on march as you said, there was a HUGE performance drop around the perimeter of the ship. I profiled this exact scenario, and as you can see below, it has nothing to do with ANY mods. It's caused by pure vanilla code when multiple AudioReverbTrigger game instances each search for a gameobject every single frame.

Now, since I was in there, I went ahead and experimented with optimizing that code and caching the audio preset object it looks for so it doesn't need to do a lookup - and that results in a 10x performance boost near the ship when I tested it using your modpack.

There's a couple other things to note though. One, ReverbTriggerFix may also give you further performance boosts. Two, DiFFoZ is working on the Lethal Performance mod, and is planning on doing a global optimization overhaul for things like that. Something to keep an eye on and perhaps add it to your modpack.

Anyway, when v1.3.7 is out, I'd recommend enabling GI if you want to see performance improvements around that specific area. If you're concerned about monitor performance, simply ensure UseBetterMonitors = false. Again, I haven't seen GI appear on any of the stack frames as a culprit for poor performance in my few hours of testing.

[ajsglist]

I definitely agree there's trigger-related lag (I reported that possibility to Batby a month or two ago), which was corroborated by a mod someone else released removing these triggers (https://thunderstore.io/c/lethal-company/p/EugeneWWolf/VanillaMoonsLagFix/). The "ReverbTriggerFix" you linked has just come out today, so I still need to test it for any potential improvements.

But the triggers are only one piece of the puzzle. Even with the triggers ripped out on March, GI still yields performance issues like those documented in the OP. Which is to say: performance is improved in the absence of these laggy triggers, but the GI-enabled modpack still manifests considerable ship lag (not just on the boundary of the trigger zone) with the monitor setup in the profile code sent to you. With the monitors setup linked + a small amount of loot + only one player at the front of the ship looking around, performance drops from a clean 60fps to sub-30fps with absolutely jarring frame pacing.

Have you looked at the performance of those specific monitor setups on a ship with a nominal level of loot?

Regardless, thanks for your continued work. I'm excited to try out any potential improvements from either GI or third party mods. When I get some time to take a deeper dive, I'll try to update you here.

Cheers.

edit: and just to be clear, I'm not solely "blaming GI" for these issues. I don't want you to feel like I'm exclusively blaming your mod for what appears to be a rather-complicated performance issue. There's definitely a vanilla component to this with triggers. LLL appears to exacerbate this further (reduced performance in these zones with only LLL and no GI). LLL + GI exacerbates it even further which is why I reported it here as well. Apologies if you thought I was exclusively blaming your mod, that wasn't my intent!

[Shaosil]

I'm not solely "blaming GI" for these issues.

No no, I didn't really take it that way. If you're sensing any frustration it's just because after all this time of hearing people say GI causes them performance problems (and it very well may), I haven't been able to track down a single piece of it after spending so much time profiling lol.

I welcome extra info wherever possible, no worries!

[ajsglist]

I welcome extra info wherever possible, no worries!

In a (hopefully) nice change of pace, I come bearing good news.

I just finished running 3 quotas, keeping as much on the ship as I could. I ran the last quota exclusively on March. A mod pack with GI <1.2.7 and VanillaMoonsLagFix had performance issues from simply existing on the ship. Upgrading to GI 1.2.7 and swapping VanillaMoonsLagFix to ReverbTriggerFix has definitely improved performance.

It's still so bizarre to me that LLL generating a facility interior manifests performance issues compared to vanilla -- which has nothing to do with GI, I'm just thinking out loud. Thanks again for the effort you've put into this. Excited to have better monitors back in the mix :)

[Shaosil]

Glad to hear it! I'll try to remember to keep you posted if I find more things to optimize.

[Shaosil]

@ajsglist so I just pushed update v1.4.0 which has a lot of research and profiling, and me and Zaggy found an issue that allows both of us to optimize the extra cameras we use and remove a ton of overhead that used to exist. I'd bet the better cameras will be much more performant in this version, though I can't guarantee it's not still a little worse than not using them. Hope it helps!

[ajsglist]

Thanks for the update, I'll check that out soon.

While I'm here I have a quick question since I'm not sure there's a better place to ask. OpenBodyCams recently updated to fix FPS limiting so that it's actually a potential performance booster. I remember for the longest time it was actually a performance loss to limit the OBC. For whatever reason, I assumed this was also true of GI's camera limiter. Do you know offhand whether the GI setting to limit the inside/outside cams helps or hurts performance? And is there a reason why the limiter caps out at 30fps instead of say 60fps?

Thanks

edit: just gonna make a new post for this since I'm interested in some info/followup :)