[Bug]: Entity function seems to cause lag and crash. mixins: BetterDiving, ObdisianBoat, RoadRunner, Aquafied.

[Diversion98]

Modpack

FTB OceanBlock

Modpack version

1.4.1

Log Files

https://paste.feed-the-beast.com/view/49da57ad

Describe the bug

The server lags sometimes and not just some small lag. look next line.

[net.minecraft.server.MinecraftServer/]: Can't keep up! Is the server overloaded? Running 527333ms or 10546 ticks behind

So what I did is said the max tick time to 60000 so I could get a crash report. (look paste)

Steps to reproduce

unknown, sorry

Expected behaviour

no big lag :p

Screenshots

No response

Additional information

No response

[Raidobw2]

Reconstructed stacktrace for visibility:

java.lang.Error: ServerHangWatchdog detected that a single server tick took 60.00 seconds (should be max 0.05)
at sun.misc.Unsafe.park(Native Method) ~[?:1.8.0_292] {}
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) ~[?:1.8.0_292] {}
at net.minecraft.util.concurrent.ThreadTaskExecutor.func_223705_bi(SourceFile:558) ~[?:?]
at net.minecraft.util.concurrent.ThreadTaskExecutor.func_213161_c(SourceFile:129) ~[?:?]
at net.minecraft.world.server.ServerChunkProvider.getChunkBlocking(ServerChunkProvider.java:1249) ~[?:?]
at net.minecraft.world.server.ServerChunkProvider.func_212849_a_(ServerChunkProvider.java:1145) ~[?:?]
at net.minecraft.world.World.getChunkLithium(World.java:4205) ~[?:?]
at net.minecraft.world.World.func_212866_a_(World.java:4191) ~[?:?]
at net.minecraft.world.World.func_180495_p(World.java:379) ~[?:?]
at net.minecraft.util.math.shapes.VoxelShapes.func_216386_a(SourceFile:259) ~[?:?]
at net.minecraft.util.math.shapes.VoxelShapes.func_216383_a(SourceFile:207) ~[?:?]
at net.minecraft.entity.Entity.func_213313_a(Entity.java:782) ~[?:?]
at net.minecraft.entity.Entity.func_223307_a(Entity.java:740) ~[?:?]
at net.minecraft.entity.Entity.func_213306_e(Entity.java:712) ~[?:?]
at net.minecraft.entity.Entity.func_213315_a(Entity.java:534) ~[?:?]
at net.minecraft.entity.LivingEntity.func_233633_a_(LivingEntity.java:2053) ~[?:?]
at net.minecraft.entity.LivingEntity.func_213352_e(LivingEntity.java:2014) ~[?:?]
at net.minecraft.entity.passive.horse.AbstractHorseEntity.func_213352_e(AbstractHorseEntity.java:724) ~[?:?]
at net.minecraft.entity.LivingEntity.func_70636_d(LivingEntity.java:2449) ~[?:?]
at net.minecraft.entity.MobEntity.func_70636_d(MobEntity.java:488) ~[?:?]
at net.minecraft.entity.AgeableEntity.func_70636_d(SourceFile:127) ~[?:?] {re:mixin,re:classloading}
at net.minecraft.entity.passive.AnimalEntity.func_70636_d(AnimalEntity.java:51) ~[?:?]
at net.minecraft.entity.passive.horse.AbstractHorseEntity.func_70636_d(AbstractHorseEntity.java:511) ~[?:?]
at net.minecraft.entity.LivingEntity.func_70071_h_(LivingEntity.java:2158) ~[?:?]
at net.minecraft.entity.MobEntity.func_70071_h_(MobEntity.java:300) ~[?:?]
at net.minecraft.entity.passive.horse.AbstractHorseEntity.func_70071_h_(AbstractHorseEntity.java:547) ~[?:?]
at net.minecraft.world.server.ServerWorld.func_217479_a(ServerWorld.java:611) ~[?:?]
at net.minecraft.world.server.ServerWorld$$Lambda$20804/182913825.accept(Unknown Source) ~[?:?]
at net.minecraft.world.World.func_217390_a(World.java:554) ~[?:?]
...

From the mixins, not in order: xf:fml:better_diving:LivingEntity canBreatheUnderwater Transformer xf:fml:better_diving:LivingEntity decreaseAirSupply Transformer xf:fml:better_diving:LivingEntity increaseAirSupply Transformer xf:fml:better_diving:LivingEntity Transformer pl:mixin:APP:roadrunner.mixins.json:ai.nearby_entity_tracking.LivingEntityMixin pl:mixin:APP:roadrunner.mixins.json:alloc.enum_values.LivingEntityMixin pl:mixin:APP:roadrunner.mixins.json:entity.stream_entity_collisions_lazily.EntityMixin pl:mixin:APP:roadrunner.mixins.json:entity.fast_suffocation_check.EntityMixin pl:mixin:APP:roadrunner.mixins.json:entity.fast_fluid_check.EntityMixin pl:mixin:APP:roadrunner.mixins.json:entity.skip_fire_check.EntityMixin pl:mixin:APP:aquafied.mixins.json:AquafiedLevelMixin pl:mixin:APP:obsidianboat.mixins.json https://github.com/FTBTeam/FTB-Modpack-Issues/issues/87#issuecomment-939315609

This happened in the overworld: entities: 216 [thermal:blizz_projectile:82,thermal:blitz_projectile:31,thermal:basalz_projectile:18,minecraft:tropical_fish:12,minecraft:squid:12]

There's two islands loaded (2 players online) and some chunks in the Nether for a lava pump and that's it. CompactMachines has crop farms and a wither farm for the special fluid, looks alright.

To author, please answer back when possible:

1. Do your users have a cursed earth mob farm as well as a blessed earth mob farm? Are they always on?
2. Can you check around the position of your 2 players if there are obsidian boats on the top of the ocean or at the bottom, on the seafloor. Check according to your server's render distance, the positions to start looking from are: (x=260, y=128, z=9470) and (x=280, y=160, z=4870). Either at the surface of the water or at the bottom. Describe if they have mobs inside too, if you see some.

To dev: Sunekaer this was reported with lag in the other issue 87 linked above in the mixins part. Their mob farms look ok-ish, that's some mobs but a server should be able to handle that. Does better diving scripts/mod apply to other entities than the player? The mixins look odd, I honestly don't really know what it does, but it seems applied to another entity, after the call to some horse AI. Otherwise, the same obsidian boat appears in that log too. Search with Ctrl + F in the log the term "boat" and look at the current call everytime. It's always an entity. Could this entity be the target of Better Diving scripts/mod, inside an Obsidian Boat and causing tons of lag?

[Diversion98]

1. We have indeed both a cursed eath and blessed earth mob farm always turned on (we can turn off the cursed earth) -> same island
2. Can't find any obsidian boats near those coords, also tried /kill @e[type=obsidianboat:obsidianboat] to make sure I idndt miss it

[Raidobw2]

Thanks for your reply. I'll try and investigate with the info you gave us. I've tested some things already tonight with boats and cursed earth, but I'll test some different things again later when I get more time. We'll keep you updated if we find anything, or if we have other questions.

[Raidobw2]

For the record, some shots of what I've tried earlier for a few hours: 7x7 cursed earth Near the end I got curious when I saw a llama spit at a baby zombie when the trader got hit, at dusk I don't know if llamas are some entity.passive.horse.AbstractHorseEntity but I found that peculiar enough to post that here. Mostly just some observations.

[Elliona]

I have a similar problem on my Server. No Crashes, thanks to max-tick-time=-1 but performance is overall bad, with TPS ranging from 8-16. I was using version 1.5 and 1.6. Sparklog: https://spark.lucko.me/pIfgqccRJz

I have deleted all Obsidianboats, Blursed Earth deactive.

If i can help with any additional informations I gladly help out.

[Raidobw2]

Interesting, the spark report shows WYML as the lag source. This was a problem we had in earlier versions but is supposed to be fixed. Perhaps it's different a bit this time. So we can correctly report this to the mod author if it's needed, which version of OceanBlock are you currently using?

[Elliona]

The Log was created using Version 1.6. I'm running this Version aswell atm.

[Raidobw2]

To dev: Sunekaer, could this be a WYML problem?

[Sunekaer]

Im not sure, ill forward the issue

[Sunekaer]

Is this in SP or MP that it happens? If MP how many players is on when it happens? What kinda hardware are you running this on? Have you changed any configs or removed mods?

[Elliona]

This is on a MP Server, with just one player online at that time. Its running on a VPS with and AMD EPYC 7282 16 Core 32GB Ram (Oceanblock gets 16 of them) Other then Spark i have not changed anything.

[Sunekaer]

Okay then it shouldnt be WYML, causing lag. Just to check could you upload the wyml.json from config folder

[Elliona]

For some reason it wont let me upload the .json. wyml.txt

[TilBlechschmidt]

We've been experiencing the very same issue where the server occasionally runs into tick times upwards of 60sec and crashes (as per max-tick-time). To me, the crash logs contain seemingly random stack traces. Over the past couple of days, this happened both with players online but also with the server just "idling" (but chunk loaded islands). Our tech-level is medium to low with mostly sluices, smelteries, some Mekanism machines, and one player with a small ME system.

We suspected Chisel & Bits because that mod started glitching after updating to 1.7.0, however, the crashes also happens with it removed. The server crashes both on 1.6.0 and 1.7.0. I've attached a ZIP file with all the crash reports. The world save is too large for GitHub, but in case it helps, I can upload it elsewhere. If there is anything else I can do to provide more information, let me know.

(The hardware is a dedicated server with four AMD EPYC CPU cores and 16G RAM which usually runs the server at ~15ms per tick so I do not suspect hardware performance to be an issue — especially since we ran FTB Revelation on it earlier this year just fine

Crash reports

1.6-crashes.zip 1.7-crashes.zip

Debug logs

1.6-debug.log.txt 1.7-debug.log.txt

Another thought that came to mind is that some mod could just be hogging memory and thus forcing the server into more and more GC cycles until it eventually runs out of stuff to garbage collect and crashes. This would certainly explain the seemingly random stack traces. However, without detailed profiling, this is nothing more than a guess. I've now installed spark and will continue monitoring until the next crash occurs.

[Raidobw2]

Thanks for the info, we'll try and sort that out soon, the error is a bit peculiar so it's not easy to find the cause. I've checked the linked issue and just so you are aware, the current branch for Spark on github is probably for Minecraft 1.17. This MC version had a huge change in Java and exclusively uses Java 16+. Minecraft before 1.17, such as 1.16.5, uses Java 8, 11, sometimes 16. On the CurseForge page, which is were you normally get the mods, you'll find different Spark releases and even more with the Files tab at the top. (https://www.curseforge.com/minecraft/mc-mods/spark). A view from the quick lateral downloads, not all the available files. I see you got it compiled yourself which is amazing! However, I suspect the highlighted version to be compatible with Java 8, since it's for Minecraft 1.16.5 (the version OceanBlock uses) and I've used it in the past while doing tests. Just some knowledge that might save you some time for the future!

[TilBlechschmidt]

Well, thanks for the tip. Totally did not consider checking CurseForge 😅

Anyways, the self-compiled version works like a charm. I'll report back when it yields any results (aka the server crashed again). Question is whether or not Spark auto-uploads the current profile when the server crashes or I have to catch it right as it crashes to stop the profile. Either way, I'll try my best ^^

[Sunekaer]

Hey, So we honestly think this is down to the CPU being to slow to handle this modpack, do you have another system with a faster CPU that you can test on?

[TilBlechschmidt]

I have to politely disagree. The server is running with a 95-percentile tick time of ~30ms and CPU utilisation of the core used by the Minecraft server is below 40% on average. Our server runs on dedicated cores of a AMD EPYC 7702P 64-Core Processor. While this is a server-chip with a single-core performance that is only around 55% of what desktop-class chips can deliver, profiling and system load monitoring indicates that it is not fully utilised.

Additionally, the symptoms are very short-term and localised. The server runs just fine on average with a maximum recorded tick time of 120ms during a GC cycle with a majority of players online. Just before it crashes, it continuously starts lagging behind over a time period of maybe 30-60sec. From the players perspective, it goes from 20TPS to 0TPS instead of gradually slowing down.

In the report below, collected with spark, you can see that the TPS, tick duration, and CPU utilisation are all well within limits.

I am by no means an expert in modding Minecraft, but from my DevOps background I'd consider the (linux) server absolutely healthy with no single core approaching limits. Unfortunately, I do not have any other machine to run the server on. However, in typical fashion it did not crash over the past 48 hours since I've added the monitoring mod.

If there is anything else I can test for or examine, let me know. I'll make sure to keep an eye on the CPU graph next time it crashes.

[TilBlechschmidt]

However, one possibility I can not deny is that our server hoster did some weird shenanigans which resulted in the machine overall slowing down for just long enough so that the Minecraft server would crash. I do not have anything in favour or against this theory though 🤷

[Sunekaer]

7792P is still only a 2.0 GHZ CPU, which isnt great for modern minecraft.

[Raidobw2]

Hmm. I wonder if potentially all people in this post could be using online hosts, and that this could be the results of lag spikes which can happen at times.

I've especially had that problem with hosts that share CPU between instances, but I've never experienced crashes because max-tick-time is always set to -1 to prevent lag spike crashes for that matter (it has happened before I started using it on online hosts back then, hence why I'm using it now!).

We have not heard from the original user in a while and all issues seem semi-random. Some crash reports are mentioning Sheeps, other Zombies or Horses to cause lag, but all mention the watchdog (aka Lag Watcher). We cannot replicate it as well... therefore; If I can get confirmation from people on this post that haven't answered yet:

• Are you using an online host?
• Did setting max-tick-time=-1 in server.properties solve the issues?
• Did you speak with your hosts to know if the CPU had been experiencing stress when crashes occur? (yes, if you pay for it you should be able to ask for that info).

Based on the answers, I could potentially close this issue, as all seems to point at lag spikes. Thanks in advance for anyone who will be responding.

P.S.: If your players are using Blessed or Cursed earth farms, chunkloaded and on 24/7 I'd expect the CPU to try and catch a break at some point (stacktrace mentions Sheep, Zombie and Horse). I'd recommend shutting those farms off when you don't need them and observing the server's behavior after setting max-tick-time.

[Elliona]

• Yes, im using an online host
• Yes, i did set max-tick-time=-1. The server didn't crash, but the lag was still there.
• I have not, will do next thing in the morning. As for the CPU, mine runs at 2.8GHZ.

Hope the info helps.

[Elliona]

I can confirm that the CPU was under high load when the major lag spikes occurred. So for me it was most likely that.

[Raidobw2]

Thanks for the feedback!

[JeNoVaViRuS]

Good day,

I am hosting Minecraft Modpacks on private servers for about 10 years already and this is the first pack I have major issues with. I can confirm the seemingly random high ticks and all the exact behaviour mentioned above (sudden spikes that are gone again immdiately, sometimes it just works with no lags at all, sometimes it lags 2-3 seconds every 30 seconds). My server has 4v Cores of a Ryzen 9 3900X, 16GB RAM and an NVMe SSD. This is not an issue about hardware specs. I also experience crashes with almost the same crash log (except my crash log mentions a PigEntity).

This is the first time I couldn't analyse the source of lags. I just can't pinpoint it at all no matter which chunks I load/unload.

The weird thing is.....if I download the world and start it in singleplayer mode then there seem to be no bad TPS at all (analyzed with spark). Am I missing something? Are the spawn and other players chunks not loaded if I start it in singleplayer mode?

My pack is on version 1.9.0 crash-2021-11-27_01.05.32-server.txt .

[Raidobw2]

Hi @JeNoVaViRuS, could it be possible to send us the world download of that server? We'll check and load as many sources of lag as we can to replicate it. We'll have time to test that in about a week, at maximum two, so it won't be an instant fix.

We've tried to pinpoint the error but the stacktraces are so random and it's lag... Could be anything in-game. However I'll label potential culprits as WYML, RoadRunner, Minecraft Water ticks or MiniUtilities (earths that spawn mobs) -- optimization mods (I think water ticks is vanilla though) and one content mod that as been relatively laggy apparently.

You could try the Observable mod and looking onto Cursed/Blursed/Blessed earth, you'll see what I mean.

Thanks in advance if you send us the world for us to test. If you want to keep it private, you can message me on the FTB Discord (name will be "R3GEN") and send a Google drive link or something.

[JeNoVaViRuS]

Hi @Raidobw2, I have already tried disabling WYML but it didn't have any effect.

I have uploaded the world folder: https://mega.nz/folder/UpJTVKZb#DQgC8T9Ip0sTCa-KvkJoMw

Thank you very much for taking the time to analyze it!

[Gaz492]

Could you please confirm if you are still experiencing this issue? If you are, would you be able to upload new log files for us so we can take a further look?

[github-actions[bot]]

This issue has been marked as stale. Please reply to the issue or the issue will be automatically closed in 3 days.

[github-actions[bot]]

This issue has been closed as there has been no recent activity after being marked as stale.