Crash with chunk overflow

[GedeonGrays]

Hello o/

Crucible Version: 4.1 (version with fix a written book by juanmuscaria in Discord)

I checked on Crucible my main issues from other server cores, which concern me first. The moment of checking "Duplication with chunk overflow and rollback" has come.

What went differently: After completing step 5, namely, reconnecting to the server after chunk overflow, the server crashed instead of rolling back the chunk.

Crash report: https://pastebin.com/2Pg4nT0d

P.S. After reconnecting when starting the server, I found an empty chunk, without any chests, and some of its sections were replaced by End Stone blocks. It looks like the chunk has been regenerated.

[juanmuscaria]

I had implemented the oversized chunk fix a while ago (fab221161351720e6a16a442f389dc77fec275ce) but was unable to test because books where broken, so I guess the fix is just deleting the chunk instead of actually saving it?

[GedeonGrays]

I guess the fix is just deleting the chunk instead of actually saving it?

I suppose it is. Have you implemented your own fix? Alternatively, you can try to adapt the fix made by terrainwax.

[juanmuscaria]

It's an adapted version of terrainwax's fix, but I might have messed up something because of thermos ram chunk storage and the server differences.

[GedeonGrays]

I hope that with a second attempt to fix this, there will be no special problems.

[juanmuscaria]

Do you have the full logs before the crash? I'm looking for "Provided chunk is null for " and some error

[GedeonGrays]

Oh, I'm sorry, but now I don't have them. I will try to reproduce the problem and then provide the logs.

[GedeonGrays]

This is a complete log before the attempt to reconnect to the server and the subsequent crash: https://pastebin.com/Cwnaf5sQ

[juanmuscaria]

It tried to save the oversized chunk [04:16:21] [File IO Thread/INFO]: [Crucible] Oversized Chunk at (5, 0), do you have the log before the crash, to see what went wrong when trying to load that chunk?

[GedeonGrays]

This is probably the log that you need: https://pastebin.com/HUSc3Nuy

[juanmuscaria]

Yeah, it's saving the oversized chunk but not loading it, I might have missed something in the fix.

[juanmuscaria]

Update on the issue: I did not implemented the chunk offset part, it will just break any oversized chunk when saved.

[juanmuscaria]

Can you test if the issue persist? https://juanmuscaria.beta.teamcity.com/repository/download/Crucible_Dev/105:id/Crucible-1.7.10-4.1-dev-c4db280-server.jar https://juanmuscaria.beta.teamcity.com/repository/download/Crucible_Dev/105:id/libraries.zip (login as guest to download those files)

[GedeonGrays]

Can you test if the issue persist?

I'll check and let you know.

[GedeonGrays]

Well, I can say that the crash has been fixed, but now the chunk is rolled back again when overloaded.

[juanmuscaria]

Can you send your logs from this time?

[GedeonGrays]

Can you send your logs from this time?

Sure: https://pastebin.com/yM8zRZzx

[juanmuscaria]

Can you try the exploit again with this jar and send the logs? https://juanmuscaria.beta.teamcity.com/repository/download/Crucible_Dev/106:id/Crucible-1.7.10-4.1-dev-4421309-server.jar it will print all chunk read and write, this way we can track down why it''s resetting the chunk.

[GedeonGrays]

Okay, I'll try it now.

[GedeonGrays]

In general, on the second connection to the server (after overloading the chunk), a crash occurred: https://pastebin.com/uVywNuq3

Full log from startup to server crash: https://pastebin.com/ESPAHtfD

[juanmuscaria]

I think the patch is working as intended:

[19:30:33] [File IO Thread/WARN]: Oversized Chunk at (5, 0)
[19:30:33] [File IO Thread/INFO]: No free space found, growing file
[19:30:33] [File IO Thread/INFO]: Saving chunk with size 12201117 at 3009
[19:30:51] [Chunk I/O Executor Thread-1/INFO]: Loading oversized chunk with 2979 sectors

your server crashed because you had not enough memory to uncompress and load the chunk data (as you can see it crashed with Caused by: java.lang.OutOfMemoryError: Java heap space, and probably reseted the chunk because of that), maybe if you increase the server allocated ram you can load the oversized chunk

[GedeonGrays]

How much memory do you think will be enough for the server to handle such a large chunk?

If anything, I use a startup line like this: java -noverify -jar Crucible-1.7.10-4.1-dev-4421309-server.jar

P.S. I think you can understand this from the log, but I will still clarify that my VDS has 8GB of free memory.

UPD: I should probably use something like -Xms/-Xmx ¯_(ツ)_/¯

[juanmuscaria]

When not defining on your java arguments the default max heap size is 1 gb, try using java -noverify -Xmx4096 -jar Crucible-1.7.10-4.1-dev-4421309-server.jar, 4 gb of ram might be enough.

[GedeonGrays]

Oh, you have confirmed my guesses above. Well, I'll try to run a couple more tests and if everything goes well, I'll write here and you can close the issue.

[GedeonGrays]

Well, all I can say is that the patch works much better in Mohist (I respect your time and effort, and I'm not trying to offend you). I just tested saving an overloaded chunk and connecting to a server with an overloaded chunk. Unlike your implementation, Mohist saves the chunk instantly and just as instantly lets me into the server without any hint of a crash. In addition, I start the server with -Xmx2G and have no problems, while on Crucible, even with -Xmx4G, the server lags very much and eventually crashes.

In general, I just propose to devote some more time to this problem, because this situation does not seem normal.

UPD: When connecting to the server (Crucible) after overloading the chunk, the following crash appears: https://pastebin.com/UmjaK9F7 A simple Google search reveals that the GC is running out of residual resources after the entire server has grabbed some memory. But then again, there are no such problems at all with the Mohist implementation.

[juanmuscaria]

The patch is the same as the one on mohist, so my guesses are that it may have other patches that makes chunk saving and loading more faster. About the crashes, all of them are out of memory crashes?

[GedeonGrays]

About the crashes, all of them are out of memory crashes?

As far as I understand, there are only 2 types of crashes in the context of this problem:

1. Crash when connecting to the server due to the absence of the -Xmx argument, that is, Java heap space.
2. Crash when connecting to the server due to some problems with GC, that is, GC overhead limit exceeded.

But what I know for sure is that none of these problems are present with Mohist.

I don't know if Mohist had any patches for faster saving and loading of chunks before I wrote them about this overload problem. But for my part, I am ready to test any implementation of fixing this problem.

[juanmuscaria]

Both of those crashes are crash due to not having enough ram. From the last crash you sent, it happens when uncompressing the chunk data and copying the strings. I'll try to replicate the exploit with some debugger to see what's actually happening.

[GedeonGrays]

Okay, then I'll wait for any news.

[ChefMC]

Faced this problem (spam of this line):

Teleported to the chunk, but there no something oversized (also checked this chunk in NBTExplorer inside it's .mca region, and I don't see something there).