Suppression drains performance

[Iridar]

Problem: Firing cosmetic shots during Suppression (perhaps, firing any kind of projectile volley) reduces game's performance (frames per second).

Reproduction: Set up approx. 6 XCOM soldiers vs 6 enemies and have them all suppressing each other. Witness the FPS gradually reducing over time.

Iridar's Test Report

I've done my own test with the Suppression. I cleared config and launched the game only with Highlander, and set up 6 cannon grenadiers to suppress 6 advent troopers. RAM consumption started at 2 GB, and FPS at 90. Over the course of the next 10 minutes, RAM consumption grew up to 3.6 GB, and FPS dropped to 15. Note: I still had about 8 GB of free RAM to go. Then I entered "SkipAI" and ended turn. When the next turn began, RAM consumption dropped to 3.0 GB, but FPS stayed at 15.

RAM consumption was growing only when there was some shooting happening on screen. It was stable before and after the suppression turn.

FPS didn't recover even after killing all enemies and skipping a few turns and waiting a few minutes.

After a few more minutes RAM consumption dropped to 2.3 GB and kept falling, but FPS still didn't recover.

Temporary Fix Loading a saved game seems to instantly recover any lost FPS. Restarting the game is not necessary.

Ramifications This seems like a bug that is endemic to XCOM 2 WOTC (perhaps vanilla XCOM 2 as well).

When playing WOTC without increasing enemy count and squad size, this bug is not much of an issue. Missions are short, shots are few and there is barely any suppression going in. The opposite example of that is Beagle's Operators vs Aliens mod collection, which relies on increasing squad size, enemy count, and almost every unit gains Suppression, and missions quite often become trench warfare with multiple turns of multiple units using Suppression non stop, which drains performance quickly. Missions themselves are longer, further exacerbating the problem.

For the good of all people who play modded XCOM, this issue must be resolved, and Highlander seems the perfect place for it, assuming the problem can be resolved without overly invasive measures and nasty side effects.

Potential Causes It has been speculated that there is a some sort of memory leak that is at fault. Indeed, during most of my tests, the XCOM's memory consumption would gradually grow over time, following the reduction in performance.

However, if memory loss was the cause of the performance loss, it would be a sharp loss of performance that would happen only as the system would run out of RAM space. But in my tests the FPS loss was gradual, and would reach unplayable levels long before my system would run out of RAM.

Additionally, in some of my tests, the consumed memory would eventually get freed by the game, but the FPS would not recover.

Therefore I'm compelled to conclude that memory leak is a side effect of the problem rather than the direct cause of performance loss.

I speculate that each time the unit fires a projectile volley, the game creates some sort of actor or object, and then "forgets" to remove it. If there are any functions that iterate over all instances of actors or objects of that type, they would take more and more time to do their job as the number of these actors or objects would grow, which would explain the very gradual loss in performance.

The fact that FPS seems to recover the moment you load a save game (which loads all the state objects, but destroys any currently active visualization objects), I'm guessing the problematic objects or actors are a part of the game's visualization system rather than the gamestate system.

Thoughts of other prominent modders on the topic:

robojumper:

You should still try to find what precisely is leaking. Tip: Write some sort of function that loops over Actor types and prints out interesting actors, do the same with impact components (need to find what they're attached to though)

Xymanek:

isn't there some sort of console command to get classes by instance count?

robojumper:

Oh yeah OBJ LIST or something

Musashi:

there is ListDynamicActors - Outputs a list of all dynamic actors (actors which were created after the start of the level) to the log. https://docs.unrealengine.com/udk/Three/ExecFunctions.html

E3245:

https://docs.unrealengine.com/udk/Three/ConsoleCommands.html#Debugging%20Commands Also this. The memory commands are also useful.

Next Step As robojumper has indicated, the next step seems to be confirming the problem is indeed caused by the game creating and forgetting some sort of actor or object, narrowing down said actor or object, and figuring out if we modders can do something about it, within the confines of Highlander or otherwise.

[Iridar]

Inspired by E3245's post on Discord, I investigated if this performance drain is caused by the game not cleaning up some sort of an Actor. Spoiler alert: doesn't seem so.

Here's my test case:

struct ActorClassStruct
{
    var name ClassName;
    var int iCount;
};
var config array<ActorClassStruct> OldArray;

exec function PrintActors(optional bool bRemember)
{
    local Actor locActor;
    local ActorClassStruct ActorClass;
    local ActorClassStruct EmptyActorClass;
    local array<ActorClassStruct> ActorClasses;
    local int Index;
    local WorldInfo WInfo;

    WInfo = class'WorldInfo'.static.GetWorldInfo();
    foreach WInfo.AllActors(class'Actor', locActor)
    {
        Index = ActorClasses.Find('ClassName', locActor.Class.Name);
        if (Index != INDEX_NONE)
        {
            ActorClasses[Index].iCount++;
        }
        else
        {
            ActorClass = EmptyActorClass;
            ActorClass.ClassName = locActor.Class.Name;
            ActorClass.iCount++;
            ActorClasses.AddItem(ActorClass);
        }
    }

    class'Helpers'.static.OutputMsg("=== BEGIN PRINT ===");
    `LOG("=== BEGIN PRINT ===",, 'IRIPROJ');
    foreach ActorClasses(ActorClass)
    {
        Index = default.OldArray.Find('ClassName', ActorClass.ClassName);
        if (Index == INDEX_NONE || default.OldArray[Index].iCount < ActorClass.iCount)
        {
            class'Helpers'.static.OutputMsg(ActorClass.ClassName @ ":" @ ActorClass.iCount);
            `LOG(ActorClass.ClassName @ ":" @ ActorClass.iCount,, 'IRIPROJ');
        }
    }
    class'Helpers'.static.OutputMsg("=== END PRINT ===");
    `LOG("=== BEGIN PRINT ===",, 'IRIPROJ');

    if (bRemember)
    {
        default.OldArray = ActorClasses;
    }
}

The test:

I spawned six Grenadiers with Suppression in Tactical Quick Launch and a few advent trooper targets for them to suppress. Before taking any actions with the soldiers, I entered the console command with the "true" argument.

The called function did a head count of all existing Actors and put them into an array to be checked against later.

Then I put soldiers on Suppression and let them stew for a few minutes. I was monitoring RAM and FPS during the wait, and - expectedly - saw a gradual rise in RAM usage and a reduction in FPS.

Then I entered the console command again, this time without the "true" parameter.

Then the console command did the head count all Actors again, but printed information only about those that either did not exist in the "remembered" array when the console command was used for the first time, or those that have increased in count.

Here's the output log for the curious: https://pastebin.com/RcdyFrdc

While I did witness increased number of instances of certain Actor classes, that increase was mostly static, i.e. the game created new actors and properly cleaned up old actors as necessary.

Whatever this bug is, it doesn't seem to be caused by the game not cleaning up unnecessary Actors.

[Iridar]

I redid the test, this time using the game's own OBJ LIST console command.

Unfortunately it appears to be native to the game's engine, since I didn't find its implementation in the unreal script code, though there is some "help" for it in XComCheatManager::Help(). Fortunately, OBJ LIST does print into Launch.log.

As before, I entered the command once at the start of the test, though this time I derped and hit it after one soldier was already using Suppression.

And then entered the command the second time after a few minutes of Suppression.

Then grabbed the output and analyzed it using an MS Excel macro.

It appears the game creates over 10k objects of each of the following classes: ParticleSystemComponent, StaticMeshComponent and MaterialInstanceConstant.

@robojumper or @Xymanek, any idea how to proceed from this point?

[Iridar]

A report on further developments.

Xymanek advised me to trace the projectile code that creates objects of these classes.

Since Particle Systems routinely use Static Meshes and Materials, an educated guess can be made that the issue is most likely with PSCs, and SMCs and MICs are there just along for the ride.

It appears that - for whatever reason - the particle systems that are spawned when projectiles are fired does not get cleaned up together with the projectile.

robojumper noticed there is a global EmitterPool class that has a MaxActiveEffects property, and he suggested that using it to set a global limit to the number of concurrently existing particle effect systems would resolve the problem.

I have ran a few tests and can confirm that is indeed the case.

MaxActiveEffects is not set to anything by default, so the system operates without a global limit, causing the eventual performance drain.

The EmitterPool has a few properties that seem to be intended to manage the overall memory use:

IdealStaticMeshComponents=250
IdealMaterialInstanceConstants=250
SMC_MIC_ReductionTime=2.5 // The amount of time to allow the SMC and MIC arrays to be beyond their ideals.

However, they are obviously not working, probably because Particle Systems that spawned them remain active.

I have tested several MaxActiveEffects values. All of them have successfully resolved the uncontrollable RAM growth and performance drain.

• 100 - with 6 soldiers using a conventional cannon to suppress, I noticed some of the projectile trails disappearing earlier than they should.
• 250 - no longer noticed that.
• 1000 - same, obviously.

I think we should go for the value of 1000, which would mean there will be up to 3k PSCs + SMCs + MICs in the memory at the same time, which is still a huge step from 30k I have seen during previous tests.

The reason I think the limit should be high is so that the bug can be resolved without any visual downgrades even if the player uses a huge squad on a large map with a game set up to allow most units to use Suppression, or when using Idle Suppression mod, which is all about the visuals.

Projectiles disappearing is not too big of a deal for automatic weapons, but it may look pretty poor with single shot weapons, and I'm not sure if the limit applies to environmental effects, like fire or smoke on tiles, which you obviously want to be able to see at all times.

[Iridar]

I have tested, and the particle effect limit definitely applies to world effects, like smoke, fire and acid created by grenades.

The tiles still have these effects, of course, but they cannot be seen. As such, we definitely want to set the limit higher rather than lower.

[Iridar]

Correction: with any limit value, continued projectile fire is guaranteed to start eating up particle systems used by world effects sooner or later, so setting a limit is not a valid solution.

[Iridar]

Further report: Xymanek and I have uncovered the following facts:

When a Particle System Component (PSC) is spawned into the Emitter Pool, the PSC.OnSystemFinished delegate is automatically assigned to EmitterPool::OnParticleSystemFinished(), a native function that - according to the dev comment - handles freeing the PSC's spot in the pool so that another PSC can take it.

X2UnifiedProjectile sets its own OnSystemFinished delegate to PSCs it spawns into the Emitter Pool, which means the Emitter Pool will never be notified that this particular PSC has done its work and can now be let go.

As such, these PSCs, and their MICs and SMCs accumulate indefinitely until the world is destroyed by restarting the game or loading a save.

We have successfully solved this problem by calling EmitterPool::OnParticleSystemFinished() on the PSCs in X2UnifiedProjectile manually when it's time for them to die.

Tests with the Obj List command show the steady increase in the number of objects is now gone, and the console command shows the PSCs are no longer accumulating in the pool:

exec function PrintEmitterPools()
{
    local ActorClassStruct ActorClass;
    local ActorClassStruct EmptyActorClass;
    local array<ActorClassStruct> ActorClasses;
    local int Index;
    local WorldInfo WInfo;
    local EmitterPool EPool;
    local int NumPools;

    local ParticleSystemComponent Comp;

    Cout("=== BEGIN PRINT ===");
    WInfo = class'WorldInfo'.static.GetWorldInfo();
    foreach WInfo.AllActors(class'EmitterPool', EPool)
    {
        NumPools++;
        Cout("--- NEW POOL --- ");
        Cout("PoolComponents:" @ EPool.PoolComponents.Length);
        Cout("ActiveComponents:" @ EPool.ActiveComponents.Length);
        Cout("RelativePSCs:" @ EPool.RelativePSCs.Length);
        Cout("FreeSMComponents:" @ EPool.FreeSMComponents.Length);
        Cout("FreeMatInstConsts:" @ EPool.FreeMatInstConsts.Length);

        foreach EPool.ActiveComponents(Comp)
        {
            Index = ActorClasses.Find('PathName', PathName(Comp.Template));
            if (Index != INDEX_NONE)
            {
                ActorClasses[Index].iCount++;
            }
            else
            {
                ActorClass = EmptyActorClass;
                ActorClass.PathName = PathName(Comp.Template);
                ActorClass.iCount++;
                ActorClasses.AddItem(ActorClass);
            }
        }
    }
    Cout("-------------------------------------");
    Cout("Number of Emitter Pools:" @ NumPools);
    Cout("--- Active Components ---");
    foreach ActorClasses(ActorClass)
    {
        Cout(ActorClass.PathName @ ":" @ ActorClass.iCount);
    }
    Cout("=== END PRINT ===");
}

static final function Cout(const string Msg)
{
    class'Helpers'.static.OutputMsg(Msg);
}

[Iridar]

I've run some checks on other Particle System Components and how their OnSystemFinished delegate is handled, but it looks like there are no other instances of PSCs being spawned into the Emitter Pool and their OnSystemFinished being touched, so we should be good, there should be no other instances of this bug in the game.