search

FTBTeam / FTB-Modpack-Issues

Any and all FTB Modpack issues are here! Please create a new issue, add to an existing one or maybe even suggest a feature in the discussions
https://feed-the-beast.com/modpack
40 stars 12 forks source link

[Bug]: Server crashes randomly from tick taking too long. Seems to be mob AI problem. #87

Closed legitswitch closed 3 years ago

legitswitch commented 3 years ago

Modpack

FTB OceanBlock

Modpack version

1.2

Log Files

https://hastebin.com/ibamuficim.properties

Describe the bug

I've had this same crash several times. Started happening in 1.2. The server watchdog will stop the server due to a tick taking over 60 seconds. It seems to have something to do with mobs and their AI. Looks like Roadrunner causes a crash when a zombie attacks something underwater?

The crash log mentions various mods' mixins, including: Roadrunner Aquafied BetterDiving ImmersiveEngineering WYML BetterEndForge FTBJanitor KubeJS Botania

Steps to reproduce

Not sure how to replicate it easily yet. I'll test spawning zombies and other mobs underwater and then post an update.

Expected behaviour

Server shouldn't crash

Screenshots

No response

Additional information

No response

Raidobw2 commented 3 years ago

If you set max-tick-time=-1 in server.properties that should fix it by disabling the watchdog. Is your server dedicated or rented on an online host, using a vCPU?

legitswitch commented 3 years ago

Dedicated server on my home PC, no vCPU

Crash happened again: https://hastebin.com/ucafoqagur.rb

Raidobw2 commented 3 years ago

Have you set max-tick-time=-1 in the server.properties file?

Sunekaer it's the second watchdog crash i see about Zombie Ai Goals, could be a coincidence or not.

Raidobw2 commented 3 years ago

From user gdharron in support in the Discord, made a Spark report and had the same error! https://pastebin.com/1ej9eGiA https://spark.lucko.me/1zD1MDY89K image Something seems to be messed up about how the mobs spawn. I don't think that's actually spawner blocks, but rather, how the game does the mob checks to make them spawn. The Drowned/Zombies could have ZombieAi goals if Mojang wrote it like so, so it could make sense in a way.

I'll see a bit later if i can grab a snippet of the JVM with Spark, in peaceful this time. Maybe the spawn checks are not done in peaceful, and then we would see the difference.

Sunekaer commented 3 years ago

How many players are there on your server when the issue is there?

legitswitch commented 3 years ago

Out of the dozen crashes of this I've had in the past two days, it was anywhere from 1 to 6 players. There is no common player in the crashes.

Something I noticed in the two crash reports I posted if it helps: Crash 1: Has net.minecraft.entity.ai.goal.MeleeAttackGoal Crash 2: Has net.minecraft.entity.ai.goal.RangedAttackGoal

I just removed RoadRunner instead of disabling the watchdog and making do with a possible two minute lag spike from the tick. I haven't tested downgrading RoadRunner yet. Haven't had this crash in 18 hours so far after removing it.

gdharron commented 3 years ago

My server would have anywhere from 5-15 people online when crashes occur. Server tps would be 20 and ticks taking around 30-40 ms, all running good then suddenly it would freeze and after disabling watchdog it would have to skip 1800 ticks+ if it recovered from it.

Sunekaer commented 3 years ago

Could you try and go to wyml.json in config folder and change ALLOW_PAUSE_CLAIMED and ALLOW_PAUSE_FORCED to true and see if that reduces the tick lag when there is that many people

gdharron commented 3 years ago

Ok I'll do that now. But there is no lag at all solid 20 tps even with 15 people and then suddenly it crashes or has a huge lag spike

gdharron commented 3 years ago

Ok changed those configs, does that mean force loading is now disabled due to that change?

Tick duration down to about 15ms now but if that because chunk loading is disabled then that increase in performance is expected.

gdharron commented 3 years ago

the hang happened as soon as a certain player joined which caused this chain of input to the console looks like the sever is changed the uuid of many items at the same time resulting in a lag spike

https://pastebin.com/cMzpS2pB

Sunekaer commented 3 years ago

It doesnt disable force loading, so in this map, we have issues with servers hanging due to MC not being able to spawn mobs, this alters the code to prevent that issue. These 2 are normally just not enable by default as it makes SP spawn mobs very slow

Sunekaer commented 3 years ago

Maybe try and kill the entities around that player

gdharron commented 3 years ago

It doesnt disable force loading, so in this map, we have issues with servers hanging due to MC not being able to spawn mobs, this alters the code to prevent that issue. These 2 are normally just not enable by default as it makes SP spawn mobs very slow

Right I saw this last night a new guy couldnt get mobs to spawn on a platform he made. I'll run with these settings for a bit and have completely turned off watch dog now so will see what happens

Sunekaer commented 3 years ago

remember mob farms also can fill the mob cap and cause new players to not get mobs But let me know how it goes

gdharron commented 3 years ago

remember mob farms also can fill the mob cap and cause new players to not get mobs But let me know how it goes

Hostile mobs are cleared every 30 mins in case they are stuck in a mob farm. Most mob farms kill pretty quick and even when I would clear hostile mobs they still wouldn't spawn on his platform. Had to get some zombie eggs so he could continue his first few quests lol

legitswitch commented 3 years ago

Setting max-tick to -1 didn't fix it for me. One tick was at 5 minutes before I stopped it manually. Everyone couldn't join due to a timeout error.

I got crashes relating to entity AI even after removing roadrunner: https://hastebin.com/nikovufopo.sql Crash after removing only wyml mod (why you make lag): https://hastebin.com/ugabuxihiy.rb

The last lag spike from this was about 37 seconds at a player's underwater base immediately after they logged in. Didn't see anything unusual aside from a drowned zombie with a fishing hook. It had crashed twice right before this when the player joined.

Sunekaer commented 3 years ago

Don't remove wyml... that only gonna make the issue worse..

legitswitch commented 3 years ago

I was just throwing darts at the wall to see if one of the mods was causing it directly.

Server still crashes after the wyml config change you mentioned: https://hastebin.com/iruqezarez.properties

legitswitch commented 3 years ago

Last crash was a horse, this one is a rabbit.... https://hastebin.com/ciyuqunebe.rb

I can send a backup of the world if you'd like to test it out. It seems to be at a player's base. If you do, I might have to trim the world down since it's 9 GB currently.

Raidobw2 commented 3 years ago

I read the Minecraft wiki a bit, tried to get info about ticks, so here it is:

In Java Edition, in each game loop the following actions are processed in order:

  Each dimension is ticked in order of overworld, the nether, and the end. For each dimension:

      The time is sent to the client
      The world border is updated
      Weather logic
      Player sleeping logic
      If this dimension is overworld:
            The game-time and day-time increase
            Scheduled functions are executed
      For each loaded chunk:
            Chunk info is sent to the client
            Chunk tick logic
      Phantoms, pillagers, cats, and zombie sieges will try to spawn
      Entity changes are sent to the client
      Chunks try to unload
      Scheduled ticks are processed
            Scheduled block ticks
            Scheduled fluid ticks
      Raid logic
      Wandering traders try to spawn
      Block events are processed
      Entities are processed
      Block entities are processed

No idea is this is pertinent, maybe Sunekaer will know a thing or two that i don't. Here goes:

There's a lot of water in OceanBlock. I guess that shouldn't cause a problem, but notice "Scheduled fluid ticks" is placed before the processing of "Entities are processed" according to that list (is this info trustable from the wiki?).

Everytime the server crashes for legitswitch, it's a different mob. So probably the watchdog crashes at the end of that processing, and something in-between adds a metric ton of lag for some reason.

"Block entities are processed" is at the end, and i think that would be minecraft campfires? Anyways, this might link to the following info from the 4 crashes from legitswitch:

1 ----- Level stats: ResourceKey[minecraft:dimension / minecraft:overworld]: players: 1, entities: 1291 [minecraft:wither_skull:914,minecraft:item:74,minecraft:rabbit:39,botania:spark:32,minecraft:pig:29], block_entities: 4549 [minecraft:campfire:498,mekanism:basic_universal_cable:327,minecraft:chest:261,pipez:item_pipe:214,cb_multipart:saved_multipart:213], block_ticks: 3749, fluid_ticks: 154030, chunk_source: ServerChunkCache: 32052

2 ----- Level stats: ResourceKey[minecraft:dimension / minecraft:overworld]: players: 2, entities: 1492 [minecraft:wither_skull:916,minecraft:item:235,minecraft:tropical_fish:42,minecraft:rabbit:41,botania:spark:32], block_entities: 4306 [minecraft:campfire:1036,cb_multipart:saved_multipart:389,minecraft:chest:258,mekanism:basic_universal_cable:147,botanypots:botany_pot:130], block_ticks: 22421, fluid_ticks: 897178, chunk_source: ServerChunkCache: 26942

3 ----- Level stats: ResourceKey[minecraft:dimension / minecraft:overworld]: players: 3, entities: 180 [minecraft:tropical_fish:61,botania:spark:22,minecraft:squid:18,minecraft:rabbit:11,minecraft:chicken:10], block_entities: 2503 [minecraft:campfire:909,cb_multipart:saved_multipart:223,mekanism:basic_universal_cable:140,minecraft:chest:130,minecraft:mob_spawner:70], block_ticks: 7863, fluid_ticks: 312879, chunk_source: ServerChunkCache: 14739

4 ----- Level stats: ResourceKey[minecraft:dimension / minecraft:overworld]: players: 2, entities: 105 [minecraft:tropical_fish:43,minecraft:squid:11,minecraft:rabbit:10,minecraft:pig:8,minecraft:chicken:6], block_entities: 1701 [minecraft:campfire:611,cb_multipart:saved_multipart:146,mekanism:basic_universal_cable:140,minecraft:chest:116,mekanism:basic_mechanical_pipe:61], block_ticks: 6316, fluid_ticks: 259754, chunk_source: ServerChunkCache: 12332

The fluid ticks are massive and especially at the second one, where it almost hit a million. Apparently these ticks are for the flow of water, but i'm honestly not sure -- speculation from seeing this online and guessing about it too.

There's a guy who casually dropped about one thousands wither skulls from the 1st and 2nd crash but i don't think that'd be a problem, as we see from the 3rd and 4th crash. Mek cables should not cause an issue unless they're all congested, we can compare with the Spark profile from gdharron and they don't seem to cause a lot* (edited) of lag.

The number of campfires is also high but that's normal from all the rafts, plus, from earlier i mentioned that's technically processed after the Entities, so i don't think they got processed yet in those crashes (if this info was trustable).

Anyway that's just an observation i wanted to bring to Sunekaer's eye, he'll know if it's pertinent.

Might dig again later when i get more time.

P.S.: Understood from the Spark report that WYML regulates mob spawn checks, so that should probably stay within servers, it technically reduces lag from spawn checks. If i understood correctly!

gdharron commented 3 years ago

Sorry this is getting abit long but just wanted to throw this up here for you guys to glance at, it's got the last 4 hours of my server on it looking at ticks over 100 ms

https://spark.lucko.me/oH54O9HoAO

gdharron commented 3 years ago

so crashed the server at this player's base and then went a second time and was able to avoid a crash and grab a spark profile from it.

https://spark.lucko.me/UPjbTL52Qg

gdharron commented 3 years ago

Could you try and go to wyml.json in config folder and change ALLOW_PAUSE_CLAIMED and ALLOW_PAUSE_FORCED to true and see if that reduces the tick lag when there is that many people

had this setting enabled since you posted it and server is still crashing at a certain players island. see spark report above.

ronmaster9000 commented 3 years ago

looks like we are having the exact same problem over here https://mcpaste.io/2e509a3bbee053d1 we disabled roadrunner mod and still no help. it looks like the crashing is happening when a certain player tries to access his island and if anyone teleports to it the server insta freezes until the watchdog kills it. we have about 12 users on

ronmaster9000 commented 3 years ago

so i happened to find the exact source i think... i set gamemode to peaceful which should despawn all hostile mobs but i had noticed one drowned zombie in the server with crashustilities, tried to clear the zombie and it would not clear. i teleported to the drowned and there was an obsidian boat at the exact location i teleported to... so it could be the fact that the drowned might be inside the obsidian boat on the bottom of the ocean causing the crash...

Sunekaer commented 3 years ago

Are you guys still having issues in 1.4.1? As i have lowered the spawn caps a bit in that one

gdharron commented 3 years ago

I did a clean install of 1.4 and did a world reset and so far had no issues since update. So you did something and it worked lol 👍 Thank you

Sunekaer commented 3 years ago

Awesome thanks, open a new issue if you find something new