search

anegostudios / VintageStory-Issues

Vintage Story's public issue tracker for reporting bugs, crashes and the like
43 stars 16 forks source link

1.19.7 Fatal Server Freeze From Endless Error #3858

Open RogueRaiden opened 4 months ago

RogueRaiden commented 4 months ago

Game Version

1.19.7

Platform

Linux

Modded

Modded

SP/MP

Multiplayer

Description

The issue I am having is every so often my server will encounter an error and endlessly spam this error causing almost the entire server to become unresponsive.

I have been plagued by this crash issue for over a year now. I have no idea how it happens, what causes it, how to cause it, how to prevent it, nothing. All I know is that it sometimes goes a while without it happening, sometimes it happens several times a day. Could be 10 players on, could be 60 players on. It seemed to be caused on player death, however, now it seems to happen without a death sometimes.

We do run with a number of mods. Only one of them has anything to do with player death. I've reported this error to the author of PlayerCorpse, but he says its not his issue.

Please let me know of any debugging or extra logging commands I could have running that may shine more light on this issue. Really, I just want it to stop locking up my server constantly.

How to reproduce

No response

Screenshots

No response

Logs

The second error is the one that is repeated endlessly, 60-70 of them per second. Audit and debug logs end at the moment the error spams.

30.4.2024 19:38:13 [Error] Exception: Object reference not set to an instance of an object.
   at Vintagestory.GameContent.EntityProjectile.IsColliding(EntityPos pos, Double impactSpeed) in VSSurvivalMod\Entities\EntityProjectile.cs:line 172
   at Vintagestory.GameContent.EntityProjectile.OnCollided() in VSSurvivalMod\Entities\EntityProjectile.cs:line 133
   at Vintagestory.API.Common.EntityBehaviorPassivePhysics.DoPhysics(Single dt, EntityPos pos) in VintagestoryApi\Common\EntityBehavior\BehaviorPassivePhysics.cs:line 194
   at Vintagestory.API.Common.EntityBehaviorPassivePhysics.onPhysicsTick(Single deltaTime, Boolean usePhysicsDormancyState) in VintagestoryApi\Common\EntityBehavior\BehaviorPassivePhysics.cs:line 183
   at Vintagestory.API.Common.Entities.Entity.OnGameTick(Single dt) in VintagestoryApi\Common\Entity\Entity.cs:line 874
   at Vintagestory.GameContent.EntityProjectile.OnGameTick(Single dt) in VSSurvivalMod\Entities\EntityProjectile.cs:line 101
   at Vintagestory.Server.ServerSystemEntitySimulation.TickEntities(Single dt) in VintagestoryLib\Server\Systems\World\EntitySimulation.cs:line 330
   at Vintagestory.Server.ServerSystemEntitySimulation.OnServerTick(Single dt) in VintagestoryLib\Server\Systems\World\EntitySimulation.cs:line 169
   at Vintagestory.Server.ServerMain.Process() in VintagestoryLib\Server\ServerMain.cs:line 886
30.4.2024 19:38:13 [Error] Exception: Object reference not set to an instance of an object.
   at Vintagestory.GameContent.EntityProjectile.ToBytes(BinaryWriter writer, Boolean forClient) in VSSurvivalMod\Entities\EntityProjectile.cs:line 327
   at Vintagestory.Server.ServerPackets.getEntityDataForClient(Entity entity, MemoryStream ms, BinaryWriter writer) in VintagestoryLib\Server\Network\ServerPackets.cs:line 484
   at Vintagestory.Server.ServerPackets.GetEntityPacket(Entity entity, MemoryStream ms, BinaryWriter writer) in VintagestoryLib\Server\Network\ServerPackets.cs:line 452
   at Vintagestory.Server.ServerPackets.<>c__DisplayClass16_0.<GetEntitySpawnPacket>b__1(Entity item) in VintagestoryLib\Server\Network\ServerPackets.cs:line 419
   at System.Linq.Enumerable.WhereSelectListIterator`2.ToArray()
   at Vintagestory.Server.ServerPackets.GetEntitySpawnPacket(List`1 spawns) in VintagestoryLib\Server\Network\ServerPackets.cs:line 432
   at Vintagestory.Server.ServerSystemEntitySimulation.SendEntitySpawns() in VintagestoryLib\Server\Systems\World\EntitySimulation.cs:line 830
   at Vintagestory.Server.ServerSystemEntitySimulation.OnServerTick(Single dt) in VintagestoryLib\Server\Systems\World\EntitySimulation.cs:line 169
   at Vintagestory.Server.ServerMain.Process() in VintagestoryLib\Server\ServerMain.cs:line 886
ATree42 commented 4 months ago

One of the most annoying errors. Anego please fix!

Lantalia commented 4 months ago

Partially reproed the repeated calls via throwing a spear into the sky and having it land on me, haven't managed to repro the null ProjectileStack. The issue would likely be made less impactful by allowing for projectiles with null ProjectileStack at the point of ToBytes and FromBytes, which may be nescessary if this is a race condition that can occur in vanilla

Separately managed to repro FiredBy going null on save and reload of a world where a spear landed on top of the throwing player

Lantalia commented 3 months ago

The server hanging part of this turned out to be a race condition with spears that lose their itemstack due to breaking but don't die. I can't tell if this is a race condition in vanilla, mod related, or something more complicated, but the vanilla code will break if the itemstack is null when it tries to persist it (either when saving the chunk, or over the wire in multiplayer), and vanilla code does have paths where the itemstack is nulled (though, I think, most of those should result in the projectile dieing).

I have a server only mod to prevent this and a few other similar issues from breaking a server, as well as emitting some debug information as to where and what caused the issue that may be useful in tracking down the actual root cause

https://github.com/Lantalia/RHDebug/blob/master/RHDebug/RHDebugModSystem.cs#L125

In this particular case, I found that, by the time of the ToBytes call, you had to actually write out the projectile, and that doing so required creation of a dummy itemstack, but, that the server could be protected by triggering die on the projectile after it is written

Effect on the client side is occasionally picking up spear entities that don't yield any item

Craluminum2413 commented 3 months ago

@Lantalia does it happen in 1.19.8?

Lantalia commented 3 months ago

@Craluminum2413

@Lantalia does it happen in 1.19.8?

I don't know, RogueHouse hasn't moved to 1.19.8 yet, and it is rare enough that I've only seen it in large multiplayer servers, that said, the most immediate code involved hasn't changed between .7 and .8

https://github.com/anegostudios/vssurvivalmod/blame/master/Entities/EntityProjectile.cs L327, in ToBytes: ProjectileStack.ToBytes(writer); So ProjectileStack being null is going to blow things up, if we make it into ToBytes

and

ProjectileStack.Collectible.DamageItem is apparently expected to result in PorjectileStack becoming null in some instances (or at least, the places it is called are coded that way, first calling it and then checking to see if it has become null). Now, in theory, those should result in Die getting called, but given the overrides and what not, along with any race conditions, it is possible that one of the persists is happening before Die cleans everything up. It is also possible that some mod (such as mod aim), will result in damage being done without calling die, or that there are other ways for damage to be done outside of the actual impact, resulting in the stack going null but the projectile entity surviving (well, till it breaks everything). In any case, while I was unable to identify the specific bug leading to this, it is certainly a bug that persisting the entity can't handle a state, even a transitory state, that the code itself passes the entity through, and my mod patches around that to protect the server (similar things for the basket trap and the microblock material references)