Particle emitter overflow breaks entity rendering

[thegrb93]

Once a particle emitter overflows, for some reason all entities will no longer render except for the map and you need to reload the map in order to fix it.

There is console spam with

CUtlLinkedList overflow! (exhausted index range) (65535)

[thegrb93]

I've found that citadels can also trigger this effect.

lua_run for I=1,200 do e=ents.Create("prop_physics") e:SetModel("models/props_combine/combine_citadel001b.mdl") e:SetPos(Entity(1):GetEyeTrace().HitPos+Vector(0,0,3000)) e:Spawn() end

CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CUtlLinkedList overflow! (exhausted index range) (65535) CShadowMgr::AddShadowDecalToSurface - overflowed m_ShadowDecals linked list! CUtlLinkedList overflow! (exhausted index range) (65535) CShadowMgr::AddShadowDecalToSurface - overflowed m_ShadowDecals linked list! CUtlLinkedList overflow! (exhausted index range) (65535) CShadowMgr::AddShadowDecalToSurface - overflowed m_ShadowDecals linked list! CUtlLinkedList overflow! (exhausted index range) (65535) CShadowMgr::AddShadowDecalToSurface - overflowed m_ShadowDecals linked list! CUtlLinkedList overflow! (exhausted index range) (65535) CShadowMgr::AddShadowDecalToSurface - overflowed m_ShadowDecals linked list! CUtlLinkedList overflow! (exhausted index range) (65535) CShadowMgr::AddShadowDecalToSurface - overflowed m_ShadowDecals linked list! CUtlLinkedList overflow! (exhausted index range) (65535) CShadowMgr::AddShadowDecalToSurface - overflowed m_ShadowDecals linked list! CUtlLinkedList overflow! (exhausted index range) (65535) CShadowMgr::AddShadowDecalToSurface - overflowed m_ShadowDecals linked list! CUtlLinkedList overflow! (exhausted index range) (65535) CShadowMgr::AddShadowDecalToSurface - overflowed m_ShadowDecals linked list! CUtlLinkedList overflow! (exhausted index range) (65535) CShadowMgr::AddShadowDecalToSurface - overflowed m_ShadowDecals linked list! CUtlLinkedList overflow! (exhausted memory allocator) CUtlLinkedList overflow! (exhausted memory allocator) CUtlLinkedList overflow! (exhausted memory allocator) CUtlLinkedList overflow! (exhausted memory allocator) CUtlLinkedList overflow! (exhausted memory allocator) CUtlLinkedList overflow! (exhausted memory allocator) CUtlLinkedList overflow! (exhausted memory allocator) CUtlLinkedList overflow! (exhausted memory allocator) CUtlLinkedList overflow! (exhausted memory allocator) CUtlLinkedList overflow! (exhausted memory allocator)

[thegrb93]

After removing them, things go back to normal at least. In the particle case, it doesn't recover.

[X-Coder]

I have the same issue on my servers since the release of garrysmod 13, didn't have the issue before garrysmod 13. It happens on my windows and linux dedicated server.

Whenever people are using guns which spawns bullets (bullets using the ParticleEmitter in LUA) every prop spawned in the map stops rendering and goes invisible. The invisible props will still have collision. The map stays visible. But moving map parts, like tracks or doors become invisible too. Cleanup the map doesn't fix the issue, only map reload helps.

I have seen this issue for example on spacebuild weapons, and lots of the stargate weapons of carter's addon pack. Basically every weapon which makes use of the ParticleEmitter for creating bullets.

Here is one implementation of ParticleEmitter triggering this bug: https://github.com/X-Coder/SBEP-Weapons/blob/master/lua/effects/tinypulsesplode/init.lua

Can you please check how the ParticleEmitter cleanup is currently working in garrysmod/engine? Maybe the Finish-function is not cleaning up / freeing up the memory (in time)?

[X-Coder]

Is it related to this? https://facepunch.com/showthread.php?t=1309609#2

The bug is due to Garry's Mod not removing the previous emitter. Calling Finish will not remove the particle emitter. The particle emitter appears to be left dangling in memory thus causing the overflow.

Is there a universal fix for the base lua code?

[robotboy655]

Are you not calling Emitter:Finish() in your code?

[thegrb93]

The generic_smoke effect is the main culprit for my server I think

[robotboy655]

It's not a Lua effect, it's an Orange Box .pcf effect.

[thegrb93]

I know. If it gets spammed, it breaks everything though.

[X-Coder]

@robotboy655, I linked a lua code above which triggers the bug. Emitter:Finish() is used there already in line 45, but it makes no difference.

Edit: In the link above some wrote "Finish doesn't work as they stay in memory." If thats the case, then no wonder it get overflown and breaks the rendering.

Just tried to recreate the bug by calling the linked effect 70000 times in a row. Bug didn't show up. Great. I love bugs that only show up under some unknown circumstances.

[thegrb93]

Revisiting this issue. https://github.com/garrynewman/garrysmod/blob/master/garrysmod/gamemodes/sandbox/entities/entities/gmod_emitter.lua#L125 Is it because there's no Emitter:Finish() for this ParticleEffect?

[robotboy655]

That's an orange box particle, it has nothing to do with Lua Particles

[thegrb93]

According to https://facepunch.com/showthread.php?t=1350845, you need to emitter:Finish()

[Bo98]

ParticleEffect ≠ ParticleEmitter

[thegrb93]

Ah sorry, bad reading.

[thegrb93]

A friend of mine made a video showing how smoke emitters can affect the number of active particle systems. He uses cl_show_num_particle_systems 1 to show the number of active particle systems. It shows that the number of particle systems can continue increasing to infinity, causing broken rendering and crashes.

https://www.youtube.com/watch?v=mPq2nNn28xU&feature=youtu.be The number of active particle systems is displayed in the top right. 2:30 is when the leaking happens.

[X-Coder]

Thank you for making this video. This is exactly what I have seen too on my and other servers. So the rendering bug happens when there are ~850 active particle systems as you can see in the video at 2:40.

Is there a convar to limit active particle systems? Or can this be fixed lua side?

[Python1320]

Has anyone figured which linked list it even is that is overflowing? Two guesses:

• https://github.com/ValveSoftware/source-sdk-2013/blob/0d8dceea4310fde5706b3ce1c70609d72a38efdf/sp/src/game/client/clientleafsystem.cpp#L299
• https://github.com/ValveSoftware/source-sdk-2013/blob/0d8dceea4310fde5706b3ce1c70609d72a38efdf/mp/src/game/client/particlemgr.h#L746

Maybe it is possible to recover from it.

[robotboy655]

Pretty sure it's the renderables one. The more props/entities are on the map, the easier it is to do.

[X-Coder]

Don't know but can the integer data type of this CUtlLinkedList be changed to something else?

[Kefta]

@X-Coder It's not an integer limit. It's an unsigned short

[X-Coder]

@Kefta Thank you, on closer look I have seen it now too.

There is this comment for the CUtlLinkedList class:

// Class S is the storage type; the type you can use to save off indices in 
// persistent memory. Class I is the iterator type, which is what should be used
// in local scopes. I defaults to be S, but be aware that on the 360, 16-bit 
// arithmetic is catastrophically slow. Therefore you should try to save shorts
// in memory, but always operate on 32's or 64's in local scope.
// The ideal parameter order would be TSMI (you are more likely to override M than I)
// but since M depends on I we can't have the defaults in that order, alas.
template <class T, class S = unsigned short, bool ML = false, class I = S, class M = CUtlMemory< UtlLinkedListElem_t<T, S>, I > > 
class CUtlLinkedList{...}

The slowness warning in the comment is only about the 360, which doesn't matter for garrysmod anyway.

Could you try to change the iterator type to unsigned int or long to get around the exhausted index range 65535 error in the lines Python1320 posted above? This would at least delay it, before it errors out again. The other question is what is causing this to overflow in the first place.

[thegrb93]

I imagine you can't just change the default index type of a container used widely throughout source engine without consequences. The root of the problem needs to be fixed, otherwise it'll just cause the game to run out of memory rather than overflow the container.

[X-Coder]

You are right, the root of the problem is what it needs to be fixed, or its gonna overflow sooner or later again. Increasing the data type also increases memory usage, which is not really what we want.

But somewhere we need a starting point. This error shows up almost daily on my server since the release of garrysmod 13 in 2013, before that I can't remember to show up, at least not in the way of broken rendering.

If you look through the sources, the lists are used only in some simple for each loops. I think there is only one for loop that is using int data type.

As Python1320 said, we need first to find out which one it is that is overflowing. How we can do this? Can you maybe add some debug code, which get printed together with the "CUtlLinkedList overflow!" error message so we can locate the one causing this error?

Here is another use of CUtlLinkedList which has to do with the particle's materials: https://github.com/ValveSoftware/source-sdk-2013/blob/0d8dceea4310fde5706b3ce1c70609d72a38efdf/mp/src/game/client/particlemgr.h#L593

My hope is you can find a way to fix it. With all the brains here there has to be a way :)

[robotboy655]

So the problem with the "Generic Smoke" effect is that it creates a new particle system for each particle. This is obviously wrong, I think I will add support for continuous OB particle systems for the Emitter Tool and make a continuous smoke particle system, that should fix this exploit. Doesn't fix the underlying issue, but it can't really be properly fixed, there will always be a limit of concurrent renderables.

[thegrb93]

Yeah we know it creates a new particle system each time, but why it doesn't free the particle system once the smoke disappears (after there are too many) is a mystery.

[robotboy655]

It does, but you can spam enough of them before they despawn.

[X-Coder]

Why didn't it break in this way before garrysmod 13? -at least I can't remember any map breaks like this one. Is it related to a garrysmod 13 update or a change in Valve's engine code?

If you have access to old garrysmod sources you could check how garrysmod used the particle emitter before 13?

[Kefta]

It did -- maybe not from particle emitters but I remember random linked list errors; it was just way less common. No idea why

[robotboy655]

Because there were no "Generic Smoke" effect that could spam in insane amounts of particle systems/emitters.