Orebfuscator causing lag

[jordanwilliams1]

Hello,

After a few hours of the server running we seem to get HUGE lag spikes. Massivelag is just spamming us with this plugin. https://hastebin.com/oramiyuciv.md

Any ideas? It seems to run find for a few hours then just starts going nuts and spamming this in massivelag.

Server stops getting huge lag spikes when it drops below 60. When its around 70+ We seem to get huge lagspikes :(

[jordanwilliams1]

Lag spike stop after plugin is disabled. "/ofc disable" back to a 20tps smooth.

[ProgrammerDan]

Wow, this is a new one. Thanks for the hastebin -- can you also show us your Orebfuscator config, as well as what other plugins you are running that deal with chunk loading / unloading, and block manipulation?

[MrPowerGamerBR]

Same here, Orebfuscator engine mode 2 causes the "doChunkMap" timings go to a whopping 68%, engine mode 1 is a bit less laggy, but still lags a lot.

Disabling Orebfuscator fixes the problem entirely. (drops the doChunkMap timings to only 0.22%)

[RoboMWM]

Isolated, controlled tests as well as linking the respective timing reports would help.

[ProgrammerDan]

Yes please, I cannot diagnose in a vacuum. @MrPowerGamerBR As requested before:

show us your Orebfuscator config, as well as what other plugins you are running that deal with chunk loading / unloading, and block manipulation?

[MrPowerGamerBR]

@ProgrammerDan I was going to share my Timings, but I'm still trying to figure out if it is Orebfuscator or not, so I didn't bother sharing it when I wrote that comment (because I thought it was Orebfuscator but I wasn't really sure if it is or not, still trying to hunt down the lag cause)

But anyway, if you really want to check it, here it is: https://timings.aikar.co/?id=59f0b1aa021146d3b3249219e8677f78

as well as what other plugins you are running that deal with chunk loading / unloading, and block manipulation?

Uh... I'm not sure, I don't think that I have any plugin that does that... but I do have TerrainControl (using BiomeBundle).

Config:

ConfigVersion: 13
Booleans:
  UseCache: true
  Enabled: true
  UpdateOnDamage: true
  NoObfuscationForMetadata: true
  NoObfuscationForOps: false
  NoObfuscationForPermission: false
  LoginNotification: true
Integers:
  MaxLoadedCacheFiles: 256
  DeleteCacheFilesAfterDays: 0
  EngineMode: 2
  InitialRadius: 1
  UpdateRadius: 6
Strings:
  CacheLocation: orebfuscator_cache
  NoObfuscationForMetadataTagName: NPC
Lists:
  TransparentBlocks: []
  NonTransparentBlocks: []
Worlds:
  Default:
    Types:
    - DEFAULT
    Enabled: true
    AntiTexturePackAndFreecam: true
    AirGeneratorMaxChance: 43
    DarknessHideBlocks: false
    DarknessBlocks:
    - MOB_SPAWNER
    - CHEST
    Mode1Block: STONE
    RandomBlocks: []
    ObfuscateBlocks: []
    ProximityHider:
      Enabled: false
      Distance: 8
      SpecialBlock: STONE
      Y: 255
      UseSpecialBlock: false
      ObfuscateAboveY: false
      ProximityHiderBlocks:
      - DISPENSER
      - MOB_SPAWNER
      - CHEST
      - DIAMOND_ORE
      - WORKBENCH
      - FURNACE
      - BURNING_FURNACE
      - ENCHANTMENT_TABLE
      - EMERALD_ORE
      - ENDER_CHEST
      - ANVIL
      - TRAPPED_CHEST
  Normal:
    Types:
    - NORMAL
    Mode1Block: STONE
    RandomBlocks:
    - STONE
    - COBBLESTONE
    - WOOD
    - GOLD_ORE
    - IRON_ORE
    - COAL_ORE
    - LAPIS_ORE
    - TNT
    - MOSSY_COBBLESTONE
    - OBSIDIAN
    - DIAMOND_ORE
    - REDSTONE_ORE
    - CLAY
    - EMERALD_ORE
    ObfuscateBlocks:
    - GOLD_ORE
    - IRON_ORE
    - COAL_ORE
    - LAPIS_ORE
    - CHEST
    - DIAMOND_ORE
    - REDSTONE_ORE
    - GLOWING_REDSTONE_ORE
    - EMERALD_ORE
    - ENDER_CHEST
  TheEnd:
    Types:
    - THE_END
    Mode1Block: ENDER_STONE
    RandomBlocks:
    - BEDROCK
    - OBSIDIAN
    - ENDER_STONE
    - PURPUR_BLOCK
    - END_BRICKS
    ObfuscateBlocks:
    - ENDER_STONE
  Nether:
    Types:
    - NETHER
    Mode1Block: NETHERRACK
    RandomBlocks:
    - GRAVEL
    - NETHERRACK
    - SOUL_SAND
    - NETHER_BRICK
    - QUARTZ_ORE
    ObfuscateBlocks:
    - NETHERRACK
    - QUARTZ_ORE

EDIT: Strange that now the onChunkMap is at only 19.87%, but that's due to the low player count at the time I was using Timings...

Here's another Timings that had a HUGE onChunkMap lag: https://timings.aikar.co/?id=837ee62ce9204a62aae8416d2fae4060

But again, I'm really not sure if it is really Orebfuscator or another plugin that is causing this lag... I will probably do some YourKit remote profiling to track down what's causing this lag.

Yeah, the TPS is at ~20, but it is an lag spike that is causing lag, not constant lag.

[MrPowerGamerBR]

Also, sorry for not sharing the timings report before.

I will try to do some controlled testings (as @RoboMWM said) tomorrow (too late here where I live to do this 😢, I also want to test with more players online so I can get a better result (because if it is nobody online it is a bit difficult to test Orebfuscator, after all, low player count = less chunk loading = less lag from Orebfuscator (if that's the problem of course))

I already asked about the "doChunkMap" lag in #paper before, some people said it was related to chunk generation, however my world is already 100% generated with WorldBorder, so that's isn't the issue.

The lag did go away when I disabled Orebfuscator in its config, however I'm not sure if that was an coincidence or it was really the cause of the lag.

[MrPowerGamerBR]

Okay, after testing Orebfuscator a bit (by disabling everything) it seems that the lag didn't go away, so... yeah, sorry for the confusion.

I think it was just a coincidence that, when I disabled Orebfuscator before, the doChunkMap lag did go away but, when turning Orebfuscator on, the lag came back.

So just forget about everything that I said, sorry!

[ProgrammerDan]

@MrPowerGamerBR The one thing I immediately noticed in your config is your update radius.

A radius of 6 means that every time someone breaks a single blocks, 13x13x13 (2,197) blocks are checked and updated if necessary. This is why we strongly recommend not setting it up 4 -- which would still be 729 per single break. I personally use 3 -- which is 343 per break -- an order of magnitude less then the impact of 6.

I'd recommend trying that right off -- drop update to 3 (or 4 if eff5 users complain) and see if that helps when you've got player counts in the 60s - 70s.

As for other plugins -- that is a hard one to diagnose what else might be keeping chunks open, but feel free to paste a list here or I'll see if anything pops out at me on your timings.

[RoboMWM]

I already asked about the "doChunkMap" lag in #paper before, some people said it was related to chunk generation, however my world is already 100% generated with WorldBorder, so that's isn't the issue.

I've heard that world generators may fail to populate the chunks (adding trees, grass, flowers, etc.). Not sure if Worldborder goes too fast for this or not - but someone else who claimed they had their world pre-generated did not have these generated.

A radius of 6 means that every time someone breaks a single blocks, 13x13x13 (2,197) blocks are checked and updated if necessary.

Anything larger than 3 (or was it 4) actually just causes the client to request a chunk update - thus rendering this option ineffective (and making it akin to update radius 0).

[ProgrammerDan]

Anything larger than 3 (or was it 4) actually just causes the client to request a chunk update - thus rendering this option ineffective (and making it akin to update radius 0).

Even I learn something new every day :).

Interesting that it'd trigger a chunk update request -- a single request, or multiple? Would that increase load in a similar fashion? I should really dig into that code myself...

[Aleksey-Terzi]

Interesting that it'd trigger a chunk update request -- a single request, or multiple?

It triggers "send entire chunk" to client. Server's code is responsible for this not client's. We had issue (or lets call it confusion) year ago regarding this, so I have had to investigate this more detailed :) Even created this page to not forget what was found: https://github.com/lishid/Orebfuscator/wiki/About-deobfuscation-on-block-break

[ProgrammerDan]

That's my core question then -- will this increase the burden on the server (many chunk transmissions? reloads? potentially spanning many chunks...?) or will it be roughly the same burden as if it was set to 4.

[Aleksey-Terzi]

Do you mean is there difference between 4 and 6? In most cases no, but sometimes yes - because of some of blocks in "radius" could be in another chunk and therefore it will send one chunk to client and then will send changes about another chunk too. The higher number - the higher load on the system. I would not recommend to set radius to value more than 4. The best value for performance/quality is either 2 or 3 (at this value Server will not resend chunk by breaking each block).

[MrPowerGamerBR]

Thanks everyone for the responses!

I will test it later, can't do that right now because I'm on college atm.

Also, it was on radius 6 (and I think I had an derp moment right there, since I think my mind didn't really process that it was "radius")

However I know why I did change it, my server has an custom plugin (I made it) that, when you break an block with an custom pickaxe, it breaks all the blocks in an 3x3x3 region, and, if you already guess, using it with Orebfuscator causes a lot of fake ores.

Does Orebfuscator have an API so I can provoke an "deobfuscate this" on the updated blocks? That would allow me to decrease the update radius (but I will also decrease the update radius later to test Orebfuscator)

Em 27 de abr de 2017 06:22, "Aleksey-Terzi" notifications@github.com escreveu:

Do you mean is there difference between 4 and 6? In most cases no, but sometimes yes - because of some of blocks in "radius" could be in another chunk and therefore it will send one chunk to client and then will send changes about another chunk too. The higher number - the higher load on the system. I would not recommend to set radius to value more than 4. The best value for performance/quality is either 2 or 3 (at this value Server will not resend chunk by breaking each block).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lishid/Orebfuscator/issues/143#issuecomment-297661386, or mute the thread https://github.com/notifications/unsubscribe-auth/AJDnJzKCFzptcve86fkhAXxumVcC3Y3Tks5r0F5EgaJpZM4Lqc0e .

[Aleksey-Terzi]

Does Orebfuscator have an API so I can provoke an "deobfuscate this" on the updated blocks?

Yes it is. Look at this static method: https://github.com/lishid/Orebfuscator/blob/master/Plugin/src/main/java/com/lishid/orebfuscator/obfuscation/BlockUpdate.java#L85

Example of using:

import com.lishid.orebfuscator.obfuscation.BlockUpdate;

...

List<Location>locations = new ArrayList<Location>();
locations.add(block1.getLocation());
locations.add(block2.getLocation());

BlockUpdate.updateByLocations(locations, 1);

[MrPowerGamerBR]

Thanks! I will try using that later 😉

Em 27 de abr de 2017 06:55, "Aleksey-Terzi" notifications@github.com escreveu:

Does Orebfuscator have an API so I can provoke an "deobfuscate this" on the updated blocks?

Yes it is. Look at this static method: https://github.com/lishid/ Orebfuscator/blob/master/Plugin/src/main/java/com/lishid/orebfuscator/ obfuscation/BlockUpdate.java#L85

Example of using:

import com.lishid.orebfuscator.obfuscation.BlockUpdate;

...

List locations = new ArrayList(); locations.add(block1.getLocation()); locations.add(block2.getLocation());

BlockUpdate.updateByLocations(locations, 1);

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lishid/Orebfuscator/issues/143#issuecomment-297669217, or mute the thread https://github.com/notifications/unsubscribe-auth/AJDnJ4vR1ZYIhmg0icKhIgSqTLFxe2g-ks5r0GYjgaJpZM4Lqc0e .

[ProgrammerDan]

@jordanwilliams1 Any of this discussion proving helpful to you? Any chance you could supply us with your config / plugin layout / timings? Thank you!

[jordanwilliams1]

@ProgrammerDan Sorry I made this a long time ago... It seems that the problem was based on our SSD, We can are now hitting 110 players before our tps degrades. I Haven't really dug throw my timings in a while but i dont think this plugin is causing any problems for me anymore.

[ProgrammerDan]

@jordanwilliams1 Thanks for the followup! I'm very glad to hear your problem worked itself out, don't hesitate to ping us again if you run into trouble in the future.

@MrPowerGamerBR I'll leave this PR open for you for now; let us know how some of these suggestions sort out.

[MrPowerGamerBR]

I hope it isn't an SSD issue, I took at look at the SSD health and it is at 99% and the dedicated server is just one month old (from SoYouStart), and also I think I can't request a SSD swap since I don't have any real proof that is the SSD that is bad.

But maybe it is an SSD issue since I'm having a lot of Disk IO onTick lag.

[EvilOlaf]

Which way did you lookup the hard drive status? smartctl?

[MrPowerGamerBR]

@EvilOlaf using "smartmontools"

[MrPowerGamerBR]

Okay, after more testings, it seems it isn't a Orebfuscator issue (and is also not my dedicated server SSD issue). (and now I will try to figure out why is lagging the server)

This issue can be closed, thanks!

[ProgrammerDan]

@MrPowerGamerBR Good news for us -- good luck to you, sorry we couldn't be more help, but I hope you do figure out what's causing the problems!

I do have a recommendation if you think it's an in-game/plugin issue.

I've been working on some new tooling to diagnose issues of these sorts.

For now, I've embedded them in my "catch-all" tool, "SimpleAdminHacks" available here: https://github.com/DevotedMC/SimpleAdminHacks and build available here: https://build.devotedmc.com/job/SimpleAdminHacks-master/38/

Quick brag on how it looks in-game: http://imgur.com/a/zWNWo -- details in the readme and javadoc for the "hack".