[Bug]: Force loaded chunks not unloading, even when offline chunkloading=false

Raidobw2 commented 2 years ago

Mod

FTB Chunks

Mod version

ftb-chunks-forge-1605.3.2-build.71.jar

Forge / Fabric version

1.16.5 36.2.8

Modpack & version

FTB OceanBlock 1.5.0 & FTB University 1.16 1.3.0

What issue are you having?

Chunks marked as force loaded by players will not unload. This has been a consistent issue since the packs were released and I've had to unload players' chunks manually every so often. I think the issue is they fail to unload for some reason.

This is very bad as lag will pile up on a server as more and more people join and forceload chunks.

Crashlogs

No response

Steps to reproduce

Go on a server that has players with force loaded chunks.
Use Crash Utilities mod
Press CTRL+U to bring up the menu.
Click the second tab for a list.
Click the "ENTITIES" button and switch to "TICKETS".
See list of forced tickets by clicking on the text.

Anything else to note?

Expected behavior: Chunks should unload Linked by main source, OceanBlock: FTBTeam/FTB-Modpack-Issues#231 Linked to secondary source, University: FTBTeam/FTB-Modpack-Issues#236

MojangPlsFix commented 2 years ago

I can also confirm this issue on ATM 6. https://i.imgur.com/oCZEupy.png

Using the mod observable also shows the loaded chunks in the profiler

legitswitch commented 2 years ago

Still a confirmed issue up to build 79 with Create: Above and Beyond modpack. Have not tested beyond that version.

I can also confirm this issue on ATM 6. https://i.imgur.com/oCZEupy.png

Using the mod observable also shows the loaded chunks in the profiler

Are you using Dynamic View mod by any chance?

MojangPlsFix commented 2 years ago

Are you using Dynamic View mod by any chance?

I am. But this shouldn't be a issue regarding the perma loading. We also had the Issue on FTB Endeavour, where this mod is not included

ButrBwedd commented 2 years ago

I'm having the exact opposite issue. There seems to be a bug that when the player disconnects, it unloads the chunks (this is intentional and set in the config files) but when the player rejoins, the chunks arent loaded again. The menu says they are loaded but updates when you toggle force loading on a chunk, whether it used to be loaded prior or not. This is an issue for the fact that you have to manually toggle force loading in every chunk you had loaded every time you join. I cannot seem to find what is making this happen.

DeadPsych commented 2 years ago

Unfortunately, I feel the same way, I can only agree with the previous speakers.

Maybe we could find out whether the problem is being worked on there?

Thank you for the effort!

SirEndii commented 2 years ago

Updates?

Duckgrass commented 2 years ago

We're having the same issue on our E6E server. Crash utilities are saying 1800 chunks are ticking and causes tps to go to 8. Removing the /world/ftbchunks/ directory does not fix the issue at all, but removing the ftb chunks mod jar fixes the issue entirely and leads to 20 tps.

Duckgrass commented 2 years ago

With the jar installed, crashed utils shows that almost every base has its chunks loaded by a forge:entity_ticking. Removing the mod jar causes this issue to no longer happen.

Diagnul commented 2 years ago

I also ran into this issue in the FTB Plexiglass Mountain pack. Some force loaded chunks never unload even when told to unload in the GUI. Unloading the chunks with commands did not work and removing the minecraft/world/ftbchunks directory did not fix it either. The mod reports that no chunks are loaded but Crash Utilities still shows all the chunks as loaded and ticking. Removing the FTB Chunks mod resolved the issue but I wished to keep the chunk claiming feature so this was undesirable.

I was able to work around the issue by removing the chunks.dat file located in minecraft/world/data/. That file appears to store the data for which chunks should be force loaded. Removing it before starting the server prevented all of the forced chunks from being loaded without affecting claimed chunks.

jinkhya commented 2 years ago

Hello, I also noticed that in various 1.16 and 1.18 packs. I tried forcing it with ftb ranks with the permission node to false as well but no luck. I tried using limited chunkloading mod which does unload chunks but upon player relog the chunks don't re-chunkload so you have to unmark/mark chunks as chunkloading every time, which isn't ideal.

This is really problematic for servers with a lot of different players playing at different times. I hope this issue gets some love soon-ish (as much as life permits ofc) so we can all have a better experience on servers <3

Thank you! Jin

sirus628 commented 2 years ago

With the jar installed, crashed utils shows that almost every base has its chunks loaded by a forge:entity_ticking. Removing the mod jar causes this issue to no longer happen.

ATM7 4.23 Forge 40.1.60 has this issue as well. use /ftbchunks admin unload_everything then delete chunks.dat

Around 400-1000 forge:entity tickets up on chunks at any given time is even giving a ryzen9 3900x with access to all cores a hard time. the tie in with crash utilities which is suposed to unload after a given time might be the issue, for now we will have to be very careful with who can force load with ftbchunks or try to get admins to place fueled physical chunk loaders

MichaelHillcox commented 2 years ago

Please note, I am aware of this issue and I'm trying to find some time to sort it out

togreglove commented 2 years ago

Some things I've noticed. Need to do a more clean test. If you claim things as an individual player and chunkload them, then join an ftb team, you can't access the chunks anymore and have to leave the team, unload the chunks, and rejoin the team. However if you are on a team, claim and chunkload some chunks, then leave the team, it shows the chunks as unclaimed, but they are still ticking, even offline, and even if you reform the team, you can't stop them. You can reclaim them and chunkload/unchunkload, but they will tick forever. I have found no in-game commands that would affect the chunkloaded state of blocks in that condition. Unload says it's not claimed. Over time, this causes small areas of the map to become perma-chunkloaded and causes issues.

Server: TNP Limitless 5 MC Version: 1.18.2

jeremiahwinsley commented 2 years ago

We noticed on our servers that clearing the Forge chunks.dat seems to resolve the problem, at least temporarily. It looks like FTB Chunks is using the Forge chunk manager, but not providing the callback to remove expired chunk loads (since Forge and FTB Chunks both persist loaded chunks). It could likely just use the vanilla chunk manager here like it does on Fabric, since that doesn't persist across restart. This would also allow using a custom ticket instead of forge:entity.

phit commented 1 year ago

this is such a meme at this point since it hasn't worked for years, that there's other mods adding workarounds for this

https://github.com/Darkere/CrashUtilities/blob/7d57f501bd3b96853a37af8783c57246991866f4/src/main/java/com/darkere/crashutils/CrashUtils.java#L150-L168

as far as I remember this didn't even work properly most of the time when this was still called FTB Utilities

jinkhya commented 1 year ago

Yes but I believe players have to manually mark all their chunks as chunkloading every time they log back on with that option of crashutilities on. (which can be tedious if you have to go to 6 different places, sometimes in other dimensions too, to do it)

Raidobw2 commented 1 year ago

I've poked my mod development team about it, thanks all for commenting. I'll try and have the issue updated soon / add more info here

desht commented 1 year ago

I've started looking at this on 1.18.2 (will follow up on 1.16.5 later), and I am having difficulty reproducing the problem. What I've done:

Set chunk_load_offline: false in serverconfig/ftbchunks.snbt
Entered the world on a dedicated server
Set a chunk to force-load via the FTB Chunks GUI
Created a small farm with some wheat seeds
Disconnected from the server (at this point the chunk should unload, since chunk_load_offline is false
On the server console, run gamerule randomTickSpeed 3000 to force-grow the farm, and gamerule randomTickSpeed 3 a few seconds later
Reconnect to the server as the player
I observe at this point that the farm has not grown, since the chunk I force-loaded did not stay loaded when I disconnected (additionally, I verified with the IDE debugger that the necessary call to unforce the chunk was indeed executed on player disconnect)

Validated the opposite behaviour by setting chunk_load_offline: true and this time the farm did grow while I was offline.

For those people experiencing problems on 1.18, please let me know under what scenarios you're seeing this problem. Some instructions to reproduce the issue would be really useful here.

(I have observed an issue where a player's force-loaded chunks are not loaded when the player connects if chunk_load_offline is false; this is something that will be fixed in the next release)

jeremiahwinsley commented 1 year ago

I don't have a consistent method of reproducing this at the moment, I'll take a look later, but as I mentioned above we saw this happen when the FTB Chunks loads get desynced with the Forge loads (because it saves them in two places). This happens frequently on our public servers, possibly to do with joining/leaving teams and force loads not being removed completely.

desht commented 1 year ago

OK, thanks. I might be able to do something useful with Forge's validation callback (i.e. ensure any forge-loaded chunks are also still listed as loaded by FTB Chunks).

Note this won't be relevant on 1.16.5 (where others have reported problems), since the 1.16.5 version of the mod doesn't use the Forge chunkloader functionality at all. It's purely vanilla forceloading there.

togreglove commented 1 year ago

1.18 (TNP5)

The way I cheeze the bug to keep my chunks loaded is to join the server, claim and force-load chunks, create a team, realize I lost access to my old chunks because I'm now in a team, quit the team, unload the chunks, create the team again. Most of the time, those chunks that were force-loaded when I formed the team are now perma-loaded. (I may have gotten it backwards, it might be that chunks you forceload while in a team get bugged when you leave the team and leave the team empty. I am not online to test this right now)

For reference, my first exposure to this is when I was playing solo and a friend wanted to join my team. I invited him and he couldn't access my claims because I had claimed them while unteamed. I kicked him out, then left the team myself, giving myself access to the claims again. Once I unclaimed the chunks, reformed and re-invited him, the chunks were bugged and stayed loaded, even when the chunks were not even claimed by anyone.

My test was a bit lower tech than yours. I put a furnace with a hopper full of wood on it and an attached lever, in each chunk I suspected of being bugged in a grid around the area I was testing. I ran around flipping all the levers and logged off for 10 minutes. If, when I came back, it had burned less than half a stack, it most likely did that while I was flipping levers, but the ones in the center of my base that I had force loaded beforehand had burned thru multiple stacks of wood.

jinkhya commented 1 year ago

I've noticed that upon server restart, a lot of chunks are still loaded even when nobody else has joined yet. As an example, on ATM7 I played a little, made some bonsai trees outputing in really big sophisticated chests. Stopped playing for a couple weeks, and the other day when I joined back, all the chests were completely full with like more than 10 thousands stacks in them. Nobody in my team and no neighbors around to load it either. I'm not sure when players are kicked with the idle setting in server.properties that it unloads either.

desht commented 1 year ago

Thanks for the info, all. Just to let you know, I'm working through this now - spotted a few issues already (mainly around the creation and deletion of parties, and the transfer of claims between players and parties). Hopefully will have something soon(ish) but there's a lot of corner cases to test here - I want to get this as robust as possible.

Malaxis12 commented 1 year ago

Thanks for the info, all. Just to let you know, I'm working through this now - spotted a few issues already (mainly around the creation and deletion of parties, and the transfer of claims between players and parties). Hopefully will have something soon(ish) but there's a lot of corner cases to test here - I want to get this as robust as possible.

hello, how to update files to fix issue?

JustAnotherGhostName commented 1 year ago

Can confirm it still occurs in 1.19.2 in ATM8 1.0.7. The corruption is way too frequent and also halts the server from cleanly shutting down. If you do /stop (dosnt matter if its console or not), the server will never fully shut down, and does not get to the saving state for the various dimensions. It ends up resulting in minor roll backs right before since the process has to be completely killed.

togreglove commented 1 year ago

We have resorted to running a /save-all by script just before shutdown on any pack that uses FTBChunks, then issue an automated /kill 60 seconds after it's supposed to have stopped. I hate it, but it works till they have time to address this.

On Sun, Jan 15, 2023 at 7:04 PM JustAnotherGhostName < @.***> wrote:

Can confirm it still occurs in 1.19.2. The corruption is way too frequent and also halts the server from cleanly shutting down. If you do /stop (dosnt matter if its console or not), the server will never fully shut down, and does not get to the saving state for the various dimensions. It ends up resulting in minor roll backs right before since the process has to be completely killed.

— Reply to this email directly, view it on GitHub https://github.com/FTBTeam/FTB-Mods-Issues/issues/101#issuecomment-1383317502, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE6ONK3EL6C4LEHJH4PD22LWSSNDVANCNFSM5HICN3RQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

JustAnotherGhostName commented 1 year ago

utdow

That likely would work, however, combined with an issue in FTB Teams getting corrupted at nearly every other restart (https://github.com/FTBTeam/FTB-Mods-Issues/issues/449) im having to move away from allowing players to use either FTB Teams or FTB Chunks. There should not be a need for a server admin to manually intervene at nearly EVERY restart of the server. I love the mods, but these corruptions are killing it.

desht commented 1 year ago

I'm going to a need lot more info to proceed with this: "it still doesn't work" doesn't really help. For a start, how do you know that it's actually even Teams or Chunks preventing a clean server shutdown when you run /stop.

In addition, Teams data getting corrupted does not necessarily indicate a bug with Teams. If this were on every single shutdown and startup, I would expect a lot more people to be reporting problems right now.

SirEndii commented 1 year ago

This does not seem to happen with a Direwolf 20 1.1.0 server Nothing corrupts, the server starts and stops normally

JustAnotherGhostName commented 1 year ago

I'm going to a need lot more info to proceed with this: "it still doesn't work" doesn't really help. For a start, how do you know that it's actually even Teams or Chunks preventing a clean server shutdown when you run /stop.

In addition, Teams data getting corrupted does not necessarily indicate a bug with Teams. If this were on every single shutdown and startup, I would expect a lot more people to be reporting problems right now.

I can confirm its teams data for both the corruption and loaded chunks by first going into ./server/world/data/chunks.dat with NBT explore and deleting the whole ftbchunks category of it, the unclaiming all chunks with the respective command. After that, not only could you run /stop successfully with no issues and a clean shutdown, but the ghost loaded chunks that you can see with crash utilities are no longer loaded.

Ive spent at least 6-8 hours tracing it down to specifically ftb chunks slowly removing and trying diff files after I found it was something with the world file.

JustAnotherGhostName commented 1 year ago

I'm going to a need lot more info to proceed with this: "it still doesn't work" doesn't really help. For a start, how do you know that it's actually even Teams or Chunks preventing a clean server shutdown when you run /stop.

In addition, Teams data getting corrupted does not necessarily indicate a bug with Teams. If this were on every single shutdown and startup, I would expect a lot more people to be reporting problems right now.

As for the teams data, it is already a confirmed in 1.18+. I can also confirm it by rolling back the data for specifically teams. There is no other mod that uses the teams data folder (./server/world/FTBTeams/), but ftb teams. See the following bug report where the bug has been open for nearly a year.

Im willing to bet its more likely people dont report it because not all server admins will spend the time diagnosing the issue. As far as it looks at first glance, the world data is corrupted. But if you just delete the ftb teams folder or roll it back a few hours, world boots up fine with no issues. When asking on the ATM discord, no one had any idea what the cause or solution was to these issues. However, if you look through the history, it clearly has been complained about several times in various places. I wish it were not the case, however, I could not find a single piece of evidence to suggest it was not from these two major issues.

https://github.com/FTBTeam/FTB-Mods-Issues/issues/449

My apologies for going on a rant about a different bug, but just since it was mentioned I wanted to describe my experience so far. I have spent a decent bit of time tracing things down.

JustAnotherGhostName commented 1 year ago

I also want to note that I do not know how to replicate it, at all. It just seems to occur on a world after a ~2 weeks weeks of new players joining and server activity. I only know that I can source it down to ftb chunks. I am currently in the process as well of swapping my users over to a different set of claiming and force loading methods, so at the very least, that will add another layer of confirmation as to whether or not ftb chunks causes the shutdown not to be clean.

desht commented 1 year ago

I can confirm its teams data for both the corruption and loaded chunks by first going into ./server/world/data/chunks.dat with NBT explore and deleting the whole ftbchunks category of it, the unclaiming all chunks with the respective command.

The problem I have with all this right now is the conflation of Chunks and Teams here. I'm not disputing that a problem may exist, but your description of the problems seriously needs to be disentangled.

FTB Teams has exactly nothing to do with chunk loading, so I don't understand why you keep dragging it in to a discussion about chunk loading.

As for needing to clean out chunks.dat - I don't know. That's just the forced chunk info that vanilla maintains, same as if you ran /forceload for the chunk. FTB Chunks uses the Forge chunk manager to force/unforce chunks, and that just goes through the standard vanilla API for forcing/unforcing. I cannot reproduce any issue where the server won't stop while there are forced chunks present in chunks.dat and I don't know what's happened to your world to cause that.

JustAnotherGhostName commented 1 year ago

I can write up a further description of just the FTB Chunks description from my end later today, however, my intention was just to further confirm other server admins are experiencing this issue and that its rather big. I dont think its a question as to whether or not this bug exists in FTB Chunks since this issue has been open for over a year, flagged as high priority, with multiple individuals explaining the issue exists.

As for FTB Teams, I originally only mentioned FTB Teams to highlight why the /save-all command and auto killing the process is not a valid solution in my case as it does not solve the FTB Chunks issue, and even if it did, FTB Teams still has the issue. My apologies for any confusion beyond that.

To strictly define the line between the two bugs, the FTB Chunks bug prevents the server from completely shutting down and just hangs there. As for FTB Teams, it prevents the server from completely booting up, stopping at loading world and not progressing further.

desht commented 1 year ago

with multiple individuals explaining the issue exists.

Leave the anecdotes out, please. If other people are experiencing this, they need to be confirming it, ideally in this thread. If we actually knew what environment those who experience these problems are in , that might provide some useful clues as to what's causing them. Because I know for a fact that it's not everybody who gets these problems (and the fact that I simply can't reproduce either problem on a dedicated server doesn't make it easy or even possible to determine what's going on).

JustAnotherGhostName commented 1 year ago

If we actually knew what environment those who experience these problems are in , that might provide some useful clues as to what's causing them.

Certainly, I understand that. it is a pain to diagnose the code side of things when you dont even know the steps to reproduce it in game. I cant say for certain when I can spend a few hours investigating this further than the time I took to track down the existence of this issue, however, I will try to replicate it in single player this week to see if there is any insights. I am guessing though that it might have some interaction with joining and leaving a FTB Team's party / team, but I may be wrong. It is also possible that the issue may become worse combined with the FTB Teams file rollbacks which become necessary at times, so I will try to incorporate that in my tests.

SpottyTheTurtle commented 1 year ago

I'm experiencing this on a 1.19.2 forge server, I can provide the mod list as necessary, it's a public server so I'm happy to help assist in trying to find the issue if needed.

desht commented 1 year ago

I've actually just committed another fix which I'm hoping will help; expect a new release in the next day or two.

chrisfand commented 1 year ago

Not sure if this was already accounted for in the fix, but if the server crashes while players were online with force loaded chunks, I noticed the chunks would be stuck forceloaded after restarting. So maybe a check is needed in that case to see if the player is online and, if not, it gets unloaded. This is on 1.16.5 with build 220.

desht commented 1 year ago

My fixes are focused on 1.18 and 1.19; 1.16 is in critical fixes only now. However I will take a look at fixes for 1.16 when I get a chance. The server crashing is likely to leave persistence files like chunks.dat in an inconsistent state, it's true.

JacobASorensen commented 1 year ago

I can confirm that there is this issue exists in Medieval MC [FABRIC] MMC1 (version 18.5) (MC version 1.19.2) as @chrisfand said the issue is related to the server crashing.

Steps to reproduce: (not yet verified, but highly likely) 1) force-load some chunks using FTB-chunks 2) crash the server (terminate the process) 3) restart the server

On our server, the chunks that were force loaded upon a server crash are stuck as pseudo-lazy chunks, no matter how close a player is. Specifically, the chunks will not tick blocks. Interestingly, entities will tick fine in these chunks, and so will random ticks (crops grow fine). What doesn't work is furnaces, hoppers, water flowing, bamboo fully breaking, etc.

Additionally, these chunks can be force-loaded again, which will work fine for entities and random ticks, but no matter what I do I cannot get them to tick blocks again.

Also, if the server restarts normally while a new entity is put into one of these broken chunks, the entities are not saved to the chunk.

For example, the villagers in my villager breeder vanish at every restart. The chickens in my automatic chicken farm vanish on a restart. Both of these were in FTB force-loaded chunks upon a server crash, and both of these chunks don't tick blocks anymore.

I assume what's happening is that the chunk is labeled as load level 32 or 33 (i.e. some level of lazy), so it assumes that it has already saved the entities contained within. Therefore the chunk doesn't save the entities, and so they simply vanish when the server closes. Some evidence for this theory is that my iron farm (in a broken chunk) has 2 villagers in it, and when I restock it with the third villager, that villager vanishes upon a server restart, but the original 2 remain there.

Evidently, the chunks.dat file doesn't exist in the server files so I'm not sure how to go about fixing the issue.

I wanted to make this post to help clarify the issue. I hope this helps!

desht commented 1 year ago

Really, if the server crashes and any important data files are corrupted, your only option is to restore from backup. You do have regular backups, right?

FTB Chunks can't save you if the files it needs are broken...

quartzar commented 1 year ago

Encountering the same issue on Fabric 0.14.17, Minecraft version 1.19.2.

Deleting /server/world/data/chunks.dat does not fix the issue upon server restart.

This is what I see when I run /ftbchunks admin view_loaded_chunks and open up the map:

It persists after removal of FTB Chunks jar too. I'm seeing 4000-7000 chunks being loaded at any one time, irrespective of anybody being online.

JacobASorensen commented 1 year ago

It looks like this commit may address the issue. https://github.com/FTBTeam/FTB-Chunks/commit/db045c17b0ec4bdb17b4d1b37188f195542854f1

We are going to try to update the server to the most recent version of the mod pack to see if that "properly clean up stale non-ticking forge chunk tickets" change to FTB-Chunks fixes the issue.

The chunks are not corrupted. The data is fine. The problem is related to block ticking within these chunks.

It would make sense that their chunk-load ticket is stuck as non-ticking due to the server crash. "Non-ticking chunks" precisely describes the issue we are experiencing with these chunks.

If FTB-Chunks makes a custom load ticket, it may make sense that the load ticket isn't being overridden/replaced by "player" tickets if the "FTB-Chunk Non-ticking" ticket is somehow stuck to these chunks due to the server crash not clearing them.

In that commit linked above it looks like you added a garbage collection to the Non-ticking tickets, which seems like it should fix our issue.

is that "validateLoadedChunks" function something that runs upon server/mod startup?

I will keep you posted on how updating the server mods works out.

If the most recent version of FTB-Chunks fixes the issue, then this Bug ticket might be resolved

creaper33 commented 7 months ago

I'm experiencing this issue too. Deleting chunks.dat removed 1 bugged force-loaded base's force-loaded chunks, but the rest remained (odd). I would be willing to provide a server zip with most region files removed for size reasons.

FTBTeam / FTB-Mods-Issues