simibubi / Above-and-Beyond

[Forge Modpack] Create's official Challenge-pack
140 stars 47 forks source link

Watchdog timeout/infinite load when loading world after server shutdown. #443

Open Starwhip opened 2 years ago

Starwhip commented 2 years ago

Server crash log file

For about a month a server I have been helping maintain has had occasional problems loading after shutting down. It did not happen every time it shut down, but around once a week it would fail to fully load, and either crash (server watchdog timeout) or just hang for hours, if the watchdog was disabled.

Restoring from backup usually would solve the issue, but all of our daily backups (5 days worth) now fail to load. If someone knows how to interpret the log or has any other information, it would be much appreciated.

We have run the world through regionfixer, and it reports the file "idcounts.dat" is unreadable. Removing idcounts.dat did not solve the problem.

Looking at similar issues on this github, such as #345 , I have attempted some of the recommended steps, like removing Darker Depths, but still no luck.

Starwhip commented 2 years ago

Update: Potentially found a solution. Apart from removing a couple custom mod additions (minecolonies and its requirement, structurize) that might have been causing problems, as well as Darker Depths, after deleting the entire "data" folder, the server loaded again. This deleted all the antique atlas data, lootr chest data, default map item data and so forth, so the issue seems to lie in that directory.

Starwhip commented 2 years ago

I will continue to test, but it's not working just yet. The server does load and responds to commands like "help", "say" or "time", which it did not do before deleting the data folder. But when I join the world, it times me out immediately after loading in, and then the server stops responding.

AbiDrew commented 2 years ago

With tick hang crashes the crash log isn't that helpful. If you would add the mod spark https://www.curseforge.com/minecraft/mc-mods/spark to the server and get a spark report just as you're trying to join from a client, it might help.

UnderscoreKilburn commented 2 years ago

Hi! Spark comes pre-installed with the modpack so you won't need to install it. Since the server responds to commands before anyone joins it, are you able to run /spark profiler --timeout 5? This should start the profiler then stop it after 5 seconds and output a link to the profiling results. If that does work, are you able to start the profiler and then immediately join? Since the server completely hangs the profiler might not be able to output anything, but it's still worth a shot.

Could you also send the debug.log file after the server stops responding? Even better, would it be possible for you to zip up the entire world folder and upload it somewhere so we can try and replicate this?

Starwhip commented 2 years ago

Hi! Spark comes pre-installed with the modpack so you won't need to install it. Since the server responds to commands before anyone joins it, are you able to run /spark profiler --timeout 5?

Ran the spark profiler command. The server is currently responding to commands, both before and after the spark command, but it doesn't look healthy (stuck on .park(), which is a built in Java wait method as far as I know)

Log: https://spark.lucko.me/TrEdjDbsqt

Working on getting a world download put together, but it's pretty big due to world pregeneration.

AbiDrew commented 2 years ago

Yeah, that is definitely weird. Though when you removed the entire data directory it probably removed things the pack depends on. Try downloading the server files again and copying a fresh data directory to your server and see what happens.

Starwhip commented 2 years ago

Yeah, that is definitely weird. Though when you removed the entire data directory it probably removed things the pack depends on. Try downloading the server files again and copying a fresh data directory to your server and see what happens.

A fresh server install has no files in the data directory, which is why we felt it was safe to delete. The Data folder should only store info about generated structures, villages and in-game maps (https://www.spigotmc.org/threads/what-does-world-data-folder-saves-and-do.194231/) so it doesn't seem like this would have any relevant info.

AbiDrew commented 2 years ago

Oh, I clearly misunderstood which data directory you meant. There are a few that we make use of for the pack, but yeah, the entire world directory is generated on world creation and populated as things get generated, so that would not cause any problems for the pack.

Starwhip commented 2 years ago

If anyone has the space/time to debug the world, here's a link: https://drive.google.com/file/d/1Zu4vVP-zWFHTFbsjpwnVzi-Zr2U0dKAe/view