Massive lags upon island creation

[Aevox01]

### Description

Describe the bug

I already stated this problem in the discussion section on spigotmc. Whenever someone creates a new island, there is a lag spike of around 10 seconds, where the whole server just does nothing. The distance between the islands is 1000. This worked with ASkyblock and shouldn't be a problem at all. I'm using the casual starter islands at the moment. The server is definitely powerful enough. Disabling the generation of the "end" island decreases the lag by around 5 seconds. The server is running on 1.15.2 paper (latest version, 98). Decreasing the paste speed to 16 didn't help much / not at all. Decreasing the distance between islands from 1000 to 250 only reduces the lag a bit (one second, or two). Upon island creation the whole server lags, the scoreboard stops it's animation (until island creation is done) and console says that the server is running behind. This whole issue is about the bSkyblock addon by the way.

#### Steps to reproduce the behavior

Any island creation causes this behavior. It doesn't matter if you start out for the first time doing /is or if you reset your island using /is reset. Any way of island creation causes this lag, that usually lasts around 10 seconds, sometimes even more.

#### Expected behavior

The server shouldn't lag at all. We are currently running ASkyblock on our 1.12.2 server. Island creation has never been a problem there. Even with around 30-60 players at the same time, island creation never lagged the server. A one second lag would be way better, but not enough. We can't allow lags, especially of this kind, at all.

#### Screenshots and videos (Optional)

There is not much to take screenshots of. I only get one error message in the console, but this appears due to the server running behind.

### Environment Linux

##### BentoBox Version (Mandatory)

bentobox version
[15:32:22 INFO]: Running PAPER Invalid.
[15:32:22 INFO]: BentoBox version: 1.10.0-1.13.2
[15:32:22 INFO]: Database: JSON
[15:32:22 INFO]: Loaded Game Worlds:
[15:32:22 INFO]: bskyblock_world (BSkyBlock): Overworld, Nether, End
[15:32:22 INFO]: Loaded Addons:
[15:32:22 INFO]: Border 1.0.5 (ENABLED)
[15:32:22 INFO]: BSkyBlock 1.10.0 (ENABLED)
[15:32:22 INFO]: Chat 1.0.4 (ENABLED)
[15:32:22 INFO]: ControlPanel 1.7.0 (ENABLED)
[15:32:22 INFO]: IslandFly 1.7.0 (ENABLED)
[15:32:22 INFO]: Level 1.9.3 (ENABLED)
[15:32:22 INFO]: Likes 1.7.0 (ENABLED)
[15:32:22 INFO]: Warps 1.9.0 (ENABLED)

#### Plugins (Optional)

Plugins (45): AnimatedScoreboard*, BentoBox, ChangelogBot*, Citizens, CitizensCMD, ClearLag, CMI, 
CombatLogX, CoreProtect, FakeOnlinePlayers*, FastAsyncWorldEdit, GCBanz*, GCBcast*, 
gcBoosters*, gcEssentials*, GCForceLog*, GCPrefix*, JBU*, LastLoginAPI, LuckPerms, MineResetLite*, 
Multiverse-Core, MVdWPlaceholderAPI*, Parties, PlaceholderAPI, PlayerPoints*, PlayerVaults*, 
PlugMan, ProtocolLib, Sentinel, StaffEssentials*, SuperVanish, TradeMe, UUIDChanger*, Vault, 
ViaBackwards, ViaRewind*, ViaVersion*, VoteParty*, Votifier*, WorldBorderAPI*, WorldEdit, 
WorldGuard, WorldGuardEvents, WorldGuardExtraFlags

### Additional context (Optional)

The timings you asked for: https://timings.aikar.co/?id=a3522d1349054b5b851ee36889bf58ca

[kleinesNugget]

Can confirm, tested this on a localhost and dedicated server. I assigned 4GB RAM and tested it with Paper and normal Spigot, also tested 1.15.1 and 1.15.2, both seemed to start lagging after creating 2-3 islands, and the lags started getting worse, I also tested installing FAWE and it didnt help. On 1.14 the lags didn't seem to occurre, has something to do with 1.15. I also changed paste speed to 1 and it didnt change a thing, pre-loading the world or using a database also didnt change anything. I once changed the paste-speed to 512 and it basically imediately pasted the islands, it didn't seem to have an issue with that, but as i stated earlier, after creating the 2nd or 3rd island after a server restart, the whole server started lagging again. Those lag spikes can be up to 10 seconds or even longer and drop the server tps by alot.

[BONNe]

You are using 1.13.2 version on 1.15.2 server. It could be one of the reasons.

[kleinesNugget]

Is there any other 1.15 version? Its saying that this version is supporting 1.15, I also tested the latest dev version btw, forgot to say that.

[BONNe]

bentobox version
[15:32:22 INFO]: Running PAPER Invalid.
[15:32:22 INFO]: BentoBox version: 1.10.0-1.13.2

it is 1.13.2 version ... it literaly is telling it. One thing that could be laggier... biome setting... but world pre-generation should negate that.

[kleinesNugget]

Yea i saw it was telling him there, but its also saying

Compatibility Works with server versions 1.13.2, 1.14.x and 1.15.x. This release uses BentoBox 1.11.0.

and as i said i also tried the latest dev version

[BONNe]

It is not quite true. The message is misleading, as a server can run on 1.13.2, but it may not work completely correctly. That is the reason why there exists a specific version for 1.13.2, which is not compatible with 1.14.4 and 1.15.2, even if it does not crash. But I doubt that is the reason why it is lagging.

[kleinesNugget]

As far as i know, Bentobox was the "recode" of ASkyBlock for 1.13, but they also stated that 1.15 is more stable than 1.14 on their discord, so they will start dropping support for 1.14 and use 1.15, but apparently that one doesnt seem to be stable for us, and its not like we changed much, we just used default settings and tested around to make it stop lagging.

[Poslovitch]

I just had a look at the timings @Aevox01 provided. The high resource consumption is indeed caused by our BlueprintPaster task, however it states that the "wait time" is due to a "chunkAwait" process, on which I doubt we have any control.

Please do not deviate from the main topic of this issue : performances issues when pasting islands.

EDIT: I stumbled upon this similar issue on Essential's bug tracker: https://github.com/EssentialsX/Essentials/issues/2701

the lag is caused not by Essentials, but by "chunkAwait" within the MineCraft server itself. In other words, the lag is indirectly caused by the loading and unloading of chunks in the world. You can see the same issue for any other teleportation plugin such as HorseTpWithMe which also displays in your timings as being laggy. There is probably not any way for this to be "fixed" except for disabling the tp commands on your server if it can't handle them.

Paper 1.14.4 doesn't have the same chunk loading inprovements as it did in 1.13.2 yet. Some of these improvements are being tested and refined currently, but there isn't anything we can do to improve world loading performance right now.

[kleinesNugget]

This probably wont be purely BSkyBlocks/BentoBox's fault, since it's working perfectly fine on 1.14.4 for example, but I also noticed that some other SkyBlock plugin is pasting islands fine without lags on 1.15.2

[RazzerDE]

same problem here.

[Poslovitch]

same problem here.

@R4zzerYT @kleinesNugget

Could you please provide timings? I am unable to reproduce the issue on my end - island pasting works absolutely fine.

[kleinesNugget]

@Poslovitch what Server jar are you using? My timings are pretty much the same as the ones of @Aevox01 but I'll get em again and edit a link to them in this comment

[Poslovitch]

I am running on a Spigot 1.15.2 build I compiled a week ago.

[kleinesNugget]

https://timings.aikar.co/?id=2354213ef64b4c0191a406f6a03ac85a There are the timings, and are you using the "dev" bentobox builds or the normal 1.11.0 version thats available as the stable download?

[Poslovitch]

Thanks for the timings. Same issue: the "chunkAwait" thing.

are you using the "dev" bentobox builds or the normal 1.11.0 version thats available as the stable download?

I am using a local development build. I don't recall there has been optimization made on island pasting as of 1.11.0. The last improvement we've made is the integration of PaperLib... And weirdly enough, those are seemingly Paper users that are having issues.

[RazzerDE]

Anyone here uses a Server-Booster Plugin for spigot for mobfarms, itemdrops, view-distance or something like this? because i have it. Is this a possible reason? @Poslovitch

[kleinesNugget]

@R4zzerYT im using a vanilla server without any plugins but Bentobox

[RazzerDE]

thats totally awkward

[Aevox01]

I am quite confused, tbh. BentoBox and some other premium plugin are not working. Then we tried another Skyblock plugin and that worked, why would 1 out of those 3 work, while the others just cause lag?

[Poslovitch]

Then we tried another Skyblock plugin

Which Skyblock plugin is working?

[kleinesNugget]

Okay uhm, I just ran the buildtools for spigot 1.15.2, downloaded BentoBox 1.11 and BSkyBlock 1.11, threw it onto the server with 4GB of RAM, which should be enough for just BentoBox and 1 player, and its still causing lags. There are the timings: https://timings.spigotmc.org/?url=jucuxiwaco

and also everytime I create an island, it shows this in the console, which also makes sense, considering its noticibly lagging. [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7541ms or 150 ticks behind

[Aevox01]

SuperiorSkyblock is working.

[Poslovitch]

SuperiorSkyblock is working.

Well, interestingly enough, I have heard they are using NMS... It is known for its efficiency, since its messes with the minecraft server, which is a no-go on our end. That's too much of a hassle to keep it updated. So I suppose the other Skyblock plugins do the same thing as we do, i.e. use the proper API to set blocks. Which is slow, and seemingly laggy.

[Aevox01]

No idea, the other plugin that we tried was "FabledSkyblock"

[kleinesNugget]

Actually makes sense if you look at it that way, although do you think it has to be fixed from BentoBox's side or spigot/mojang

Edit: Also why does it work for you tho?

[Poslovitch]

Actually makes sense if you look at it that way, although do you think it has to be fixed from BentoBox's side or spigot/mojang

I still consider this issue as a release blocker. We need to get it fixed, if not mitigated, as it severely impedes our ability to be a free and trustworthy alternative to existing Skyblock plugins.

It pains to admit it, but we might have to resort to NMS. That'll get rid of the issue entirely. I doubt anything else would. Implementing an island creation queue could only mitigate the issue, not fix it.

@tastybento Your thoughts about this?

Also why does it work for you tho?

Actually, the default settings (paste-speed, most notably) that are provided by default are based on tests on my end. They are optimized for my local test server, which is quite a potato. And I assumed that if it's "optimized" on my end, bigger servers would have no issues. Actually, it works fine on my end, because I made sure it did. That sort of tells me that the debug I have been doing for the last 30 minutes was pointless.

[kleinesNugget]

Actually, the default settings (paste-speed, most notably) that are provided by default are based on tests on my end. They are optimized for my local test server, which is quite a potato. And I assumed that if it's "optimized" on my end, bigger servers would have no issues. Actually, it works fine on my end, because I made sure it did. That sort of tells me that the debug I have been doing for the last 30 minutes was pointless.

This seems interesting, considering I dropped the paste-speed to 1 and upped it to 512, both lagged the same kinda, with the only difference being, pasting the island itself was way faster on 512 than on 1, but both had the weird server lag, thanks for investigating this tho.

[tastybento]

From the timings reports, I agree that it looks like the chunk loading/unloading is taking a long time. It's also being done sync and not async. As anyone who starts a server knows these days - when you start, the server will pre-generate a number of chunks around spawn - you can see it taking a while. When we paste an island, the first thing the server does the following:

1. Finds a clear spot in the world.
2. If chunks have never been generated in that area, they are generated
3. Paste the blocks over a number of ticks using a recurring timer

The first part has been optimized to try to avoid loading any chunks, but it's not async and does load chunks to check if there is actually an island there if there's a spot it doesn't know about. This usually only usually affects imported worlds and is unlikely the cause in this case.

The second part could be a contributing cause. Once a clear spot has been found, the server will generate the chunks. This is done in all 3 dimensions (overworld, nether, end). If those chunks are near spawn, they may already exist and so the server will get right to pasting. If they are not, then the chunk generation kicks in. In theory, the only dimension that has anything in it is the Nether that has a roof. However, with 1.15.x, biomes changed and became 3D. (have height). There is an API to set them still using the x and z coords only, but I suspect it runs a for loop to set the y height as well. This might add some lag to the system. There may be other overhead involved with chunk generation too, I don't know.

The 3rd part is done via a recurring Bukkit task that does a few chunks at a time. This is what the config settings change - how many blocks are pasted every go, and how long to wait before doing the task. So, you could set it to do 1 block every second (20 ticks) for example, or 512 blocks in 1 tick. We do not use NMS to paste blocks right now. By using the API, blocks are subject to the rules of block setting. We turn off physics, but lighting calculations might have an effect.

Let's do some testing. Here is a BSkyBlock build that has Biome setting and the Nether roof turned off. Please try and make some islands in this way:

1. Start with a clean world (stop server, delete the world, restart)
2. Use the default BentoBox and BSkyBlock settings so we are all on the same page
3. Log in and make sure you are Op. Have some others on the server to check if lag occurs.
4. Create an island using /island
5. Unregister yourself from your island using the /bsb unregister (name) command. Do this instead of resetting because I don't want island deletion to be done.
6. Make a new island.
7. Repeat from step 5 at least 10 times to get outside the pre-generated chunks.
8. See if there is a perceptible lag during the new island pasting.

Feel free to paste timings. Note that the paster task will appear to take a long time but that because it's a recurring task.

BSkyBlock-1.11.1-SNAPSHOT-LOCAL.jar.zip

If this seems to help, then try repeating but this time using /island reset instead and see if the lag returns. It'd be good to know if it's due to the deletion or not.

I'll be trying stuff too, but your assistance will help a lot!

Thanks!

This below part is just FYI Differences to ASkyBlock:

1. ASkyBlock used NMS to fast-paste blocks. We decided to avoid NMS as much as possible with BentoBox and instead try to do everything by the API and NMS is not API. ASkyBlock approach just pasted blocks all in one go and the server took them all and set them async. I'm really surprised that it ever worked as well as it did.
2. ASkyBlock was also able to delete islands using the RegenerateChunk method in the API. Since 1.13.x, this no longer exists in the API or the server. Therefore, deleting islands has to be done by explicitly pasting blocks one by one. This kicks in whenever islands are reset.
3. ASkyBlock never had an End Island - only the overworld and nether had to be pasted. So we're deleting/regenerating and pasting a lot more.

[kleinesNugget]

I'm not sure If I already mentioned it, but i also tested if the lag/amount of lag changes if I disable the end/neither and it didn't change anything noticable. I cant test the thing you said, since I'm not on my pc atm and won't be for a few more hours

[tastybento]

Okay, hold off on the testing. I am trying a number of things to see what might make a difference. Even with the changes I made I'm seeing things like:

[18:13:43 WARN]: Can't keep up! Is the server overloaded? Running 8820ms or 176 ticks behind

I think it's all down to chunks, but let me see.

[tastybento]

I'm narrowing in on a solution. The end world appears to be taking a huge amount of time to load chunks. Edit: it's not just the end world. If I remove it, another world can take its place as taking a long time. Okay, time to make that async.

[tastybento]

Okay, this should help. Try the latest build (after it compiles) on the CI server and see how it works for you. The issue is caused by chunk loading taking a long time. I found that the server can choke for many seconds when trying to load a chunk. This looks like a server bug to me, but there's not much I can do about it except wait. I use Paper's async chunk loading, which should work with Spigot too. I also updated the pasting time estimate. This will display the worst-case time since the last server restart, i.e., it's an estimate only. In my testing, if the chunk had never been generated then it could sometimes take up to 10 seconds to load! It appeared to be random as to where the chunk was, sometimes the overworld, sometimes the end and maybe the nether too. I'm really not sure what the server is doing. It may be doing something "clever" like prioritizing chunks that have players in them and so the unpopulated chunk gets stuck on the back burner, but it's weird. Anyway, this should fix it. Let me know.

[kleinesNugget]

https://timings.spigotmc.org/?url=gidewasoda Timings first. I downloaded the latest dev version from Jenkins (codemc.io) for both BentoBox (1.11.1-b1648) and BSkyBlock (1.11.1-b592) and the lag didnt really get better except for it behaving a little bit differently. The lag now occurres before sending the "Estimated Time" message, and all the other messages. And its not dropping the TPS as low as 14 or something, its just going down to 18, at least in my testing, although the lag that you can feel still persists at the same length. The TPS might aswell just be higher cause of not testing for that long. Also, used a Spigot-1.15.2 that I built yesterday, without any other plugins installed and default configs.

[tastybento]

I'll give you a debug version that will print timings to the console so we can see where the issue is coming from. Please try it and follow these instructions:

1. Stop your test server
2. Delete the old BSkyBlock world folders
3. Delete the plugins/BentoBox/database folder
4. Edit the BSkyBlock config.yml to set the island distance to 1000. This will make sure the new islands are placed in chunks that need generating
5. Start server
6. Log in - you'll be in the regular world.
7. Op yourself from the console if you aren't already
8. Make an island /island
9. After the initial one, do /island reset and make another.
10. Look in the log, it will be making debug.
11. Continue to make islands once or twice until you see the lag you are concerned about. A good test of lag is to try and open the chest on your island while things are happening and see if you can.
12. Stop the server
13. Grab the latest log from the server's log folder, zip it and drag and drop to this ticket.

Thanks! BentoBox-1.11.1-SNAPSHOT-LOCAL.jar.zip

[kleinesNugget]

I ran the test, apparently nether generation always took quite some time in the debug messages, although even if I technically disable the nether and end, it still lags, so its not related to certain dimensions as we all should've figured out by now. Also running commands during island creation process is also able to show you its lagging. TPS for some reason stay kinda, not stable, but high considering how hard its lagging.

Here are the logs you asked for. log.zip

[tastybento]

Hmm. I haven’t tried myself with Spigot but what I don’t see here is any ticking around the chunk loading. Can you run the same test with Paper and see how it performs. If the chunk loading is blocking on Spigot and not on Paper then it’s because Paper is doing async loading and Spigot is not. It’ll also cause me to do some Spigot specific testing.

[kleinesNugget]

Okay, using paper basically removed the lag entirely, although I still noticed very minor lags that i would just assume being caused by old islands generating and getting deleted while i created 3 new ones it feels like 😂 But that runs way better. Did you guys switch to NMS yet or are still trying to make it work with Async methods? Thanks for taking care of this issue and looking further into it!

[Poslovitch]

using paper basically removed the lag entirely, although I still noticed very minor lags that i would just assume being caused by old islands generating and getting deleted while i created 3 new ones it feels like 😂 But that runs way better. Did you guys switch to NMS yet or are still trying to make it work with Async methods? Thanks for taking care of this issue and looking further into it!

Well, that is not surprising, considering that the change tastybento made was to use Paper's asynchronous chunk loading. Yet, in case you are using Spigot, these asynchronous methods are not available and fall back to the default ones... Hence the lag...

That doesn't really please me to consider this issue as fixed if it only benefits to 56% of our user base.

[kleinesNugget]

Which makes sense, I wouldn't really consider it a fix either, but it's a big improvement. Also tastybento said it should also work for Spigot which quite confused me. But that it works at least for Paper right now is a good thing to start with I guess. Since I and I believe @Aevox01 is using Paper aswell it would be fine for us, though it makes sense that this isnt considered a "proper" fix.

[tastybento]

The root cause here is that the chunk loading on Spigot is not just blocking the main thread, but it's blocking it for a loooooong time. There is no way that it should take that long. I will try and write a test plugin that only does a block setType() on a non-generated chunk to test how long it takes on Spigot and then report it to the Spigot Jira. I'll try and find out whether this happens when using the vanilla chunk generator or if it's specific to the void generation. It might also be worth testing on a vanilla server and using the setblock command to see if it chokes as well. It might also be related to some kind of structure checking that the server is doing, e.g., should there be a village there or shipwreck or something. There is a WorldCreator generateStructures​(boolean generate) option that I'm going to experiment setting to false to see if that helps.

[BONNe]

It is not in effect for CustomWorldGenerator.

CustomWorldGenerator (used by BentoBox) does not support structures. In second february they implemented ability to generate structures in the custom chunk generator.

[tastybento]

I created a test plugin and was able to replicate this issue outside of BentoBox. I filed a bug with Spigot. Let's see what md5 says: https://hub.spigotmc.org/jira/browse/SPIGOT-5573

[BONNe]

For everyone, who want to use BentoBox now without lag on island creation/reset, i can suggest to use spigot/paper builds that was created before 2. february.

[tastybento]

Just FYI: Pretty sure this is where the bug was introduced: https://hub.spigotmc.org/stash/projects/SPIGOT/repos/craftbukkit/browse/src/main/java/org/bukkit/craftbukkit/generator/CustomChunkGenerator.java?at=8a66d4c7ac262b6c0a97e52a3caa08a93feb90aa#241

Old method:

@Override
    public BlockPosition findNearestMapFeature(World world, String type, BlockPosition position, int i, boolean flag) {
        return "Stronghold".equals(type) && this.strongholdGen != null ? this.strongholdGen.getNearestGeneratedFeature(world, this, position, i, flag) : null;

Feb 2nd commit:

@Override
    public BlockPosition findNearestMapFeature(World world, String s, BlockPosition blockposition, int i, boolean flag) {
        return delegate.findNearestMapFeature(world, s, blockposition, i, flag);
    }

It appears that the findNearestMapFeature method is going off and searching hundreds of chunks. I'm not sure what the original method would have returned but trying an older 1.15.2 build revealed it only generated 25 chunks when trying to load a new chunk.

We'll just have to wait for the Spigot guys to fix it or find a workaround.

[kleinesNugget]

Didn't actually expect to find out that there is kind of a big "bug" in Spigot, I mean, those lags are quite massive considering not much is actually happening. Didn't know what I was expecting in general tbh so well.

[Poslovitch]

For reference, here is the link to the JIRA issue. https://hub.spigotmc.org/jira/projects/SPIGOT/issues/SPIGOT-5573

[kleinesNugget]

tastybento already mentioned that link somewhere earlier

[tastybento]

This is fixed! https://hub.spigotmc.org/stash/projects/SPIGOT/repos/craftbukkit/commits Use Paper Build 105 or later or rebuild BuildTools for Spigot.