Entity Loss/Duplicate UUID Issues

[aikar]

UPDATE: Most issues around duplicate entities, the source of them and dealing with them in chunks should now be resolved. A new config option "duplicate-uuid-resolver" is now in your config on how to handle them.

Build 1526 adds a new (and now default) mode called "Safe regen" for handling duplicates. This mode will delete any entity of same type that is near each other. This should help alleviate any risk of vanilla bugs that caused duplicate entities from surfacing them.

Please go edit your configs and set: duplicate-uuid-resolver: saferegen in your default world config

If you've used a recent build, you already have this in your config and need to update. If your updated from some older build pre 1500~ range to 1525+, then you will be on the new default

If you don't want to take any risk deleting an entity, then leave it on regen. If you don't want to take any risk of having a duplicate entity and would rather delete any potentials, then change it to delete. If you don't want to do anything to them and leave them in limbo state, change it to warn or nothing

In Limbo State, only 1 of the entities can be loaded into your world at a time. You may alternate which one ends up visible in the world due to Chunk Load Order, and will cause confusing situations. It is recommended you either use a regen strategy (so both live), or deletion to stop further log messages.

---

While #1199 chased a lot of issues and did identify legit bugs in the recent performance work of avoiding chunk lookups, it can not be that patch as the issue occurred on my server before that patch was ever even made. That patch is also near equiv to Sponges version now.

Suspect 1: (Pretty confident is not it) https://github.com/PaperMC/Paper/commit/ca0ec24d6482115e75a547115a75ed0ef4bae724 Analysis: This patch deals with Entity Removal, so I would suspect we would be having potential with duplicates rather than loss. Can not see how this can be source as the only thing that can set the flag to false is the called method.

Suspect 2: https://github.com/PaperMC/Paper/commit/e74e4584d6d1bd10a4d6929a0f1d90fb66fc0f69 Evidence against though is reports of no issues on 1468...

guaranteed issue found: Squids were causing the Random Seed to be corrupted, creating predictable and reused UUID's. But suspect that this wasn't the ONLY issue. ongoing research.

Interestingly, this also is likely the source of the issue for #1219 and what is being reported in #1208

[aikar]

Ok I do not believe it is that commit in question. This patch would impact removal, but were having issues in reverse direction.

[Drc-DEV]

(Using build 1500) While it seemed that the bug was fixed, after a restart some of the NPCs disappeared again. I noticed that it happens often after restarts and after purging chunks. So maybe it's related to something acting on chunk load.

[LogGits]

@DracarysPro this is related to #1208 I also noticed whilst testing that this happens often when the server has just restarted or the chunks haven't been loaded yet.

[electronicboy]

I had a few theories last night, one of them ranging to “is it possible that we’re saving multiple entities into chunks and trying to load them back, causing somewhat undefined behavior”, no idea if that’s possible or what side effects we’d see from that; I really need to find a way to replicate this locally, if somebody has a reasonably sized world save that they can provide that can replicate this, that might be handy, failing that I’ll try and get something setup to look into this when I have the time

On July 18, 2018 at 3:45:48 PM, Dracarys (notifications@github.com) wrote:

(Using build 1500) While it seemed that the bug was fixed, after a restart some of the NPCs disappeared again. I noticed that it happens often after restarts and after purging chunks. So maybe it's related to something acting on chunk load.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[aikar]

Copying @LogGits message from other ticket here:

@aikar This was using Paper 1500: https://www.youtube.com/watch?v=RuQYaCzm-l4

Forgot to /ver but if its needed, ill redo it the experiment. Wasn't able to test with no plugins at the time but i will tomorrow morning 👍

Same setup on 1468 works totally fine with no issues.

EDIT: Forgot to mention that this often happens after restarts

Steps to reproduce: EDIT: 1. Restart server.

2. Warp into a random chunk far away from other players so you don't make their mobs vanish.
3. Place the mobs (doesnt affect wolves for some reason)
4. fly away and roam around for abit.
5. come back to reveal missing mobs.

If step 4 didnt work. Restart the server and repeat step 2.

My paper.yml config:

# This is the main configuration file for Paper.
# As you can see, there's tons to configure. Some options may impact gameplay, so use
# with caution, and make sure you know what each option does before configuring.
# 
# If you need help with the configuration or have any questions related to Paper,
# join us in our IRC channel.
# 
# IRC: #paper @ irc.spi.gt ( http://irc.spi.gt/iris/?channels=paper )
# Wiki: https://paper.readthedocs.org/ 
# Paper Forums: https://aquifermc.org/ 

verbose: false
config-version: 13
settings:
  load-permissions-yml-before-plugins: true
  bungee-online-mode: true
  sleep-between-chunk-saves: false
  region-file-cache-size: 256
  save-empty-scoreboard-teams: false
  incoming-packet-spam-threshold: 300
  remove-invalid-statistics: true
  save-player-data: true
  use-alternative-luck-formula: false
  min-chunk-load-threads: 3
  suggest-player-names-when-null-tab-completions: true
  enable-player-collisions: true
  player-auto-save-rate: -1
  max-player-auto-save-per-tick: -1
messages:
  kick:
    authentication-servers-down: Mojang Authentication Servers are down.
    flying-player: Flying is not enabled on our server
    flying-vehicle: Flying is not enabled on our server
timings:
  enabled: true
  verbose: true
  server-name-privacy: true
  hidden-config-entries:
  - database
  - settings.bungeecord-addresses
  history-interval: 300
  history-length: 3600
world-settings:
  default:
    generator-settings:
      canyon: true
      caves: true
      dungeon: true
      fortress: true
      mineshaft: true
      monument: true
      stronghold: true
      temple: true
      village: true
      flat-bedrock: false
      disable-extreme-hills-emeralds: false
      disable-extreme-hills-monster-eggs: false
      disable-mesa-additional-gold: false
    use-vanilla-world-scoreboard-name-coloring: false
    remove-corrupt-tile-entities: true
    use-chunk-inhabited-timer: true
    queue-light-updates: false
    portal-search-radius: 128
    delay-chunk-unloads-by: 10s
    max-auto-save-chunks-per-tick: 24
    save-queue-limit-for-auto-save: 50
    max-chunk-sends-per-tick: 81
    max-chunk-gens-per-tick: 10
    game-mechanics:
      allow-permanent-chunk-loaders: false
      scan-for-legacy-ender-dragon: true
      disable-end-credits: false
      disable-player-crits: false
      disable-sprint-interruption-on-attack: false
      shield-blocking-delay: 5
      disable-chest-cat-detection: true
      disable-unloaded-chunk-enderpearl-exploit: true
    max-growth-height:
      cactus: 3
      reeds: 3
    fishing-time-range:
      MinimumTicks: 100
      MaximumTicks: 600
    despawn-ranges:
      soft: 32
      hard: 128
    falling-block-height-nerf: 0
    tnt-entity-height-nerf: 0
    water-over-lava-flow-speed: 5
    fast-drain:
      lava: false
      water: false
    optimize-explosions: true
    lava-flow-speed:
      normal: 30
      nether: 10
    lootables:
      auto-replenish: false
      restrict-player-reloot: true
      reset-seed-on-fill: true
      max-refills: -1
      refresh-min: 12h
      refresh-max: 2d
    filter-nbt-data-from-spawn-eggs-and-related: true
    max-entity-collisions: 8
    disable-creeper-lingering-effect: false
    hopper:
      cooldown-when-full: true
      disable-move-event: false
      push-based: false
    keep-spawn-loaded-range: 10
    auto-save-interval: -1
    nether-ceiling-void-damage: false
    allow-non-player-entities-on-scoreboards: false
    frosted-ice:
      enabled: true
      delay:
        min: 20
        max: 40
    container-update-tick-rate: 1
    parrots-are-unaffected-by-player-movement: false
    disable-explosion-knockback: false
    elytra-hit-wall-damage: true
    grass-spread-tick-rate: 1
    use-alternate-fallingblock-onGround-detection: false
    bed-search-radius: 1
    non-player-arrow-despawn-rate: 120
    prevent-tnt-from-moving-in-water: false
    keep-spawn-loaded: true
    anti-xray:
      enabled: true
      engine-mode: 2
      chunk-edge-mode: 1
      max-chunk-section-index: 3
      update-radius: 2
      hidden-blocks:
      - gold_ore
      - iron_ore
      - coal_ore
      - lapis_ore
      - mossy_cobblestone
      - obsidian
      - chest
      - diamond_ore
      - redstone_ore
      - lit_redstone_ore
      - clay
      - emerald_ore
      - ender_chest
      replacement-blocks:
      - stone
      - planks
    experience-merge-max-value: -1
    armor-stands-do-collision-entity-lookups: true
    disable-thunder: false
    skeleton-horse-thunder-spawn-chance: 0.01
    disable-ice-and-snow: false
    fire-physics-event-for-redstone: false
    skip-entity-ticking-in-chunks-scheduled-for-unload: false
    spawner-nerfed-mobs-should-jump: false
    baby-zombie-movement-speed: 0.5
    allow-leashing-undead-horse: false
    mob-spawner-tick-rate: 1
    all-chunks-are-slime-chunks: false
    squid-spawn-height:
      maximum: 63.0
      minimum: 45.0
    disable-teleportation-suffocation-check: false
    enable-treasure-maps: true
    treasure-maps-return-already-discovered: false

Same thing happens when using:

skip-entity-ticking-in-chunks-scheduled-for-unload: true

[aikar]

ok! looks like we had two issues.

Many moons ago, I added a patch to improve randomness of entities by making all entities share the same random instance. this was years ago... unknowingly, the squids have been out to get us...

Squids have been messing with the entity random seed. So every time a squid spawns, the entity random seed changes.

This results in UUID's generated for new entites to cycle through values that have already been used before for anything thats spawned around same time.

This is why i've seen it often in my spawn chunks with holos ,where my holos rebuild them selves on start.

Duplicate UUID's are just a nightmare all around...

Issue 2: Thanks to @bloodmc for telling us about this. We added the ability to delay chunk unloads, but this introduced another non vanilla issue:

In vanilla, if you teleport away or move far away fast, the chunk loads, the entity doesn't tick. But with our delay, if you move far enough away and the entity ticks for any reason, you now can trigger the vanilla "Despawn due to not being near a player".

This would only impact you if you had skip-entity-ticking-in-chunks-scheduled-for-unload: false though.

I've resolved the Squid Rebellion, as well as skipped despawn code if in an unloading chunk.

Please test build 1502 @LogGits and let me know if you still have issues. If you do.... Please check your server log and look for any messages about

WorldServer.a.error("Keeping entity {} that already exists with UUID {} - " + entity1, EntityTypes.a(entity1), uuid.toString()); // CraftBukkit // Paper WorldServer.a.error("Deleting duplicate entity {}", entity); // Paper

[aikar]

Still a major question is - why are we only seeing this now? We've had a shared random on entities for years.... This is NMS code so nothing new there either. Why does 1468 seem to behave correctly for LogGits? still some unknowns here

[LogGits]

Ok il do a test now and report my results. Could this squid issue also be the reason why a villager npc via the Citizens plugin, spawns in with a new uuid every restart?

Also would you like me to test with:

skip-entity-ticking-in-chunks-scheduled-for-unload: true

[LogGits]

Startup Build 1502:

Keeping entity minecraft:pig that already exists with UUID 54f2dbcc-b0aa-484a-bfca-5ed86333ae48 - EntityPig['Pig'/147, uuid='54f2dbcc-b0aa-484a-bfca-5ed86333ae48', l='world2', x=-1021.49, y=80.00, z=1021.33]

Deleting duplicate entity EntityPig['Pig'/204, uuid='54f2dbcc-b0aa-484a-bfca-5ed86333ae48', l='world2', x=-974.40, y=104.00, z=1158.08]

16:09:54Keeping entity minecraft:parrot that already exists with UUID 4711950b-a521-4630-ab9e-c41916b9e1d8 - EntityParrot['Parrot'/275, uuid='4711950b-a521-4630-ab9e-c41916b9e1d8', l='world', x=-1560.62, y=68.00, z=-2470.25]

Deleting duplicate entity EntityParrot['Parrot'/279, uuid='4711950b-a521-4630-ab9e-c41916b9e1d8', l='world', x=-1566.55, y=64.00, z=-2465.11]

16:09:55Keeping entity minecraft:parrot that already exists with UUID 4e932309-6a1e-456c-9234-fc88831dd3be - EntityParrot['Parrot'/278, uuid='4e932309-6a1e-456c-9234-fc88831dd3be', l='world', x=-1562.51, y=65.00, z=-2471.61]

Deleting duplicate entity EntityParrot['Parrot'/281, uuid='4e932309-6a1e-456c-9234-fc88831dd3be', l='world', x=-1516.57, y=63.00, z=-2515.46]

Did a restart and the same message about a parrot and pig popped up:

16:17:47Keeping entity minecraft:pig that already exists with UUID 54f2dbcc-b0aa-484a-bfca-5ed86333ae48 - EntityPig['Pig'/147, uuid='54f2dbcc-b0aa-484a-bfca-5ed86333ae48', l='world2', x=-1021.49, y=80.00, z=1021.33]
Deleting duplicate entity EntityPig['Pig'/202, uuid='54f2dbcc-b0aa-484a-bfca-5ed86333ae48', l='world2', x=-974.40, y=104.00, z=1158.08]

16:17:49Keeping entity minecraft:parrot that already exists with UUID 4711950b-a521-4630-ab9e-c41916b9e1d8 - EntityParrot['Parrot'/273, uuid='4711950b-a521-4630-ab9e-c41916b9e1d8', l='world', x=-1561.05, y=68.00, z=-2470.85]

Deleting duplicate entity EntityParrot['Parrot'/277, uuid='4711950b-a521-4630-ab9e-c41916b9e1d8', l='world', x=-1566.55, y=64.00, z=-2465.11]

Keeping entity minecraft:parrot that already exists with UUID 4e932309-6a1e-456c-9234-fc88831dd3be - EntityParrot['Parrot'/276, uuid='4e932309-6a1e-456c-9234-fc88831dd3be', l='world', x=-1562.51, y=65.00, z=-2471.61]

Deleting duplicate entity EntityParrot['Parrot'/279, uuid='4e932309-6a1e-456c-9234-fc88831dd3be', l='world', x=-1516.57, y=63.00, z=-2515.46]

With skip-entity-ticking-in-chunks-scheduled-for-unload: false mobs still vanish, I will do a test for true in a sec.

EDIT: both true and false values for skip-entity-ticking-in-chunks-scheduled-for-unload.

[LogGits]

Oddly enough, while testing this, it didnt announce the mobs that vanished, I tested with villagers, irongoles, horses, llamas, and wolves.

Even stranger, wolves are completely exempt from this whole vanishing fiasco.

@aikar whats your discord username, it may be easier for my to message you directly about this.

[MiniDigger]

you could join the paper discord https://discord.gg/jETyjUw

[LogGits]

@aikar I will be testing with a few other versions of paper to find which one actually caused this to start happening.

I will add, on 1468, players were reporting Villagers vanishing randomly not often and only villagers. This could also be because we were using a previous world folder to store data and transferred over to a new world folder hence breaking maps and potentially removing villagers. Therefore, this may not be related to this bug.

These builds had were the tests i did that functioned properly:

• [x] Build 1468
• [x] Build 1479
• [x] Build 1480

Whereas 1482+ all have issues with entities vanishing (I am assuming this as I tested 1482, 1491, 1492, 1499, 1500, 1502, and all of them had vanishing entity issues).

Ok, in my testing, I also saw that 1482 was marked as bugged, I tried it out and yes indeed this is where the vanishing mob issue started.

The thing that is the most interesting to me is why don't these mobs vanish?

• Pig
• Cow
• Sheep (any colour)
• wolf even if it is named and tamed

[0XE4]

In response to your question about the console errors: Very random, sometimes once a minute or more depending on the activity in that area. On average once every 5 minutes perhaps, it is very random. Oh, the mobs vanish the same as reported above too. Multiple reports from players of rideable entities like pigs/horses vanishing.

[aikar]

@0XE4 though have you found the entities that ended up disappearing the ones that trigger the new console error? IE does the entity type and position line up with missing entities.

and can you confirm that both the new and old lines are showing as different chunks

[0XE4]

I will have to log into the server and check all that when I am next free @aikar but I will get back to you to answer.

[aikar]

@andrewkm and anyone else tracking this. A major push was done today for build 1509 that adds an option to recover any lost entities due to the uuid duplication bug.

It's default on, so any missing entity should auto come back now.

If you would rather kill the lost entity and stop the spam, set duplicate-uuid-resolver: delete in paper world config.

If you rather leave it like it was and "don't tell me how broken things are, i don't want to be reminded I have a buggy world", set it to nothing.

It is recommended you either use regen or delete though.

Regen (default) will regenerate a new UUID for any entity with a conflicted UUID.

You will see red error text as entities are recovered, but as your world repairs itself, they should go away.

Please let me know @LogGits if you still are able to reproduce the loss issue on 1509, and if you see these warnings when you do reproduce.

If you are still having entity loss issues, also please add to your JVM startup flags: -Ddebug.entities=true and this will provide more data.

[axemwa]

"You will see red error text as entities are recovered, but as your world repairs itself, they should go away."

Usually those red errors never go away for me, I left my server on all day with 1 player standing in the same spot and when I got back from work the errors were still showing and popping up. If those errors are supposed to go away after a while like you say how long should it take for a world with a 20,000 block wide border (thats only been half explored) for them to go away?

It's good to know I can toggle those message off when I need to though thanx for that, sorry all I can do to help with those errors is send in the errors that are showing. I still havent had 1 entity dissapear on it's own yet... on any of the builds I've used, well not that I can see form testing anyway...

[axemwa]

And I'll try the latest build ot see if the messages go away after a while and I can't seem to find this section in paper.yml config (world section) even after deleting the old config duplicate-uuid-resolver: delete?

[stevmei]

Hello,

I tried to generate a new world with 1509 and I'm getting lots of duplication errors for entities:

[Server] ERROR Duplicate UUID found used by EntityMinecartChest['Minecart with Chest'/50458, uuid='7ec646b9-5db4-4b40-99de-1c1af9c010cb', l='testworld', x=102.50, y=31.50, z=1327.50, cx=6, cd=82, tl=4, v=true, d=false] [Server] ERROR Regenerated a new UUID for EntityMinecartChest['Minecart with Chest'/50469, uuid='a9f7f3db-8c2e-45c0-b63c-209129e956b7', l='testworld', x=102.50, y=31.50, z=1327.50, cx=6, cd=82, tl=0, v=false, d=false]

This should not happen to a new world, should it?

Edit: I haven't update for a while, I updated this morning from 1347 to 1509 on a testing server, generated a new testworld and these messages showed up.

Edit 2: Telepoted to a allegedly duplicated entity but there was only 1 entity generated... I tested it with a 'Minecart with Chest'

[axemwa]

@sigmaroot use paper version #1500 to completely remove errors, or #1506 (the errors will still come up but they're in process of fixing things the errors should go away after a while so I recommend having this on and just put up with the errors until everything in your servers worlds is fixed. Some sort of entity bug it's trying to fix I think.) Apparently there's a way to turn the spamming error messages off as aikar described above (scroll up) however I haven't been able to find the setting yet and like I said before they should go away eventually in version #1506 anyway

[stevmei]

The messages don't bother me. I just want to give @aikar the information that newly generated worlds are generating these duplicated entities when generating a new chunk. I dunno if that should happen but I don't think so. So the information here is: This does not only apply to existing worlds and entities.

[axemwa]

Oh sorry my bad :)

[aikar]

unless vanilla itself has a bug causing entities to duplicate, I suspect that Vanillas uuid generation is too weak and is still causing conflicts maybe, though its odd it would trigger this much.

I'm going to push a change soon to improve the randomness of UUID's to try to reduce this even more.

But if that doesn't solve it, then we got some nasty bug that causes entities to duplicate too. But there is def a case where entities that aren't duplicate of each other are sharing a UUID.

Note: That downgrading is not going to 'help' the problem. The only thing your doing by downgrading is hiding the messages, but you are still going to have uuid conflicts and losing entities by using 1500.

I strongly recommend using latest and ignoring the messages.

[aikar]

Also can someone try running with -Ddebug.entities=true too please.

[axemwa]

Yeah I'll have a look tomorrow once my internet is on again the weather lately I think has cut it off somehow, using my phone atm trying to use port forawrding on it with hotspot with no success

[aikar]

Relevant Vanilla Bugs: https://bugs.mojang.com/browse/MC-119971 https://bugs.mojang.com/browse/MC-22147 https://bugs.mojang.com/browse/MC-95649

[LogGits]

Will be testing tomorrow morning aswell, cheers Aikar or working so hard to resolve this :)

EDIT: wasn't able to test. I will be testing on Tuesday morning with debug -Ddebug.entities=true.

[axemwa]

Errors are finally clearing up overtime now in latest paper version, only this one appears now when other plugins are still starting up. but it probably changes I'll keep an eye on it

[10:23:40 ERROR]: Duplicate UUID found used by EntityCow['Cow'/303, uuid='0ae8ade7-28a8-4eb9-b7f9-88f4f11896df', l='Modernworld', x=-3131.78, y=65.00, z=2111.92, cx=-196, cd=131, tl=21186091, v=true, d=false] [10:23:40 ERROR]: Regenerated a new UUID for EntityCow['Cow'/306, uuid='a3bc1d3f-9630-46d0-aa55-57355ea1df96', l='Modernworld', x=-3132.56, y=65.00, z=2115.80, cx=-196, cd=132, tl=21182819, v=false, d=false]

[andrewkm]

Awesome, will be throwing 1509 online soon and see how things go with the default option :)

[LogGits]

Ok I am doing my testing now, I will report my findings shortly.

EDIT:Latest build 1511 still causes entities to vanish.

[DUPE-UUID] Duplicate UUID found used by EntityItemFrame['entity.ItemFrame.name'/78, uuid='81d82dff-9219-4eba-a164-d9c93c33a5a4', l='world2', x=-1131.50, y=13.50, z=1263.03, cx=-71, cd=78, tl=1794, v=true, d=false], regenerated UUID for EntityItemFrame['entity.ItemFrame.name'/2598, uuid='bd3c9d2f-4b7f-4c09-a947-eaea795cd4a2', l='world2', x=-1131.50, y=13.50, z=1263.03, cx=-71, cd=78, tl=0, v=false, d=false]. See https://github.com/PaperMC/Paper/issues/1223 for discussion on what this is about.
[DUPE-UUID] Duplicate UUID found used by EntityItemFrame['entity.ItemFrame.name'/79, uuid='c9a7529d-90bc-45e4-a781-5ac12a9efef9', l='world2', x=-1131.50, y=14.50, z=1263.03, cx=-71, cd=78, tl=1794, v=true, d=false], regenerated UUID for EntityItemFrame['entity.ItemFrame.name'/2599, uuid='9486d3ed-a3c7-48b0-ad00-ffe40205953b', l='world2', x=-1131.50, y=14.50, z=1263.03, cx=-71, cd=78, tl=0, v=false, d=false]. See https://github.com/PaperMC/Paper/issues/1223 for discussion on what this is about.
[DUPE-UUID] Duplicate UUID found used by EntityItemFrame['entity.ItemFrame.name'/80, uuid='2be2adc2-ee60-471f-a866-2628b76883a0', l='world2', x=-1127.50, y=13.50, z=1263.03, cx=-71, cd=78, tl=1794, v=true, d=false], regenerated UUID for EntityItemFrame['entity.ItemFrame.name'/2600, uuid='5fd05c94-f309-4361-b4ad-542ff61f79a5', l='world2', x=-1127.50, y=13.50, z=1263.03, cx=-71, cd=78, tl=0, v=false, d=false]. See https://github.com/PaperMC/Paper/issues/1223 for discussion on what this is about.
[DUPE-UUID] Duplicate UUID found used by EntityItemFrame['entity.ItemFrame.name'/81, uuid='533c6bf6-86b6-4e8f-b26a-69e483b012e1', l='world2', x=-1127.50, y=14.50, z=1263.03, cx=-71, cd=78, tl=1794, v=true, d=false], regenerated UUID for EntityItemFrame['entity.ItemFrame.name'/2601, uuid='8b3ad7dd-703f-439e-ae81-ac07298afe1e', l='world2', x=-1127.50, y=14.50, z=1263.03, cx=-71, cd=78, tl=0, v=false, d=false]. See https://github.com/PaperMC/Paper/issues/1223 for discussion on what this is about.
[DUPE-UUID] Duplicate UUID found used by EntityParrot['Parrot'/127, uuid='218f06fb-798f-4e22-8d23-a8d59dfb2508', l='world2', x=-1121.57, y=59.00, z=1262.61, cx=-71, cd=78, tl=90739971, v=true, d=false], regenerated UUID for EntityParrot['Parrot'/2602, uuid='68b2447f-efb0-4f3b-89f1-45c4a4768a0a', l='world2', x=-1121.57, y=59.00, z=1262.61, cx=-71, cd=78, tl=90739971, v=false, d=false]. See https://github.com/PaperMC/Paper/issues/1223 for discussion on what this is about.

I will now test debug mode as mobs that vanish don't show up in the console. Running with the debug mode I am receiving no additional information:

[DUPE-UUID] Duplicate UUID found used by EntityGuardian['Bush'/83, uuid='8f12d364-ea9a-4693-bfae-cc255596893a', l='world2', x=-1130.71, y=12.00, z=1272.43, cx=-71, cd=79, tl=217690376, v=true, d=false], regenerated UUID for EntityGuardian['Bush'/2635, uuid='29d58b8f-36b0-43b9-95d7-5e8821f9f30c', l='world2', x=-1130.71, y=12.00, z=1272.43, cx=-71, cd=79, tl=217690376, v=false, d=false]. See https://github.com/PaperMC/Paper/issues/1223 for discussion on what this is about.
[DUPE-UUID] Duplicate UUID found used by EntityGuardian['Regan'/84, uuid='2725213d-ea74-4976-8d1f-1494a9fcc50e', l='world2', x=-1127.43, y=12.00, z=1279.57, cx=-71, cd=79, tl=217690330, v=true, d=false], regenerated UUID for EntityGuardian['Regan'/2636, uuid='1fe1447d-c4bf-4e29-975d-36b1b6ae74c3', l='world2', x=-1127.43, y=12.00, z=1279.57, cx=-71, cd=79, tl=217690330, v=false, d=false]. See https://github.com/PaperMC/Paper/issues/1223 for discussion on what this is about.
[DUPE-UUID] Duplicate UUID found used by EntityItemFrame['entity.ItemFrame.name'/85, uuid='2a07b651-fc13-4aae-a2b1-538fe788e14a', l='world2', x=-1133.97, y=12.50, z=1268.50, cx=-71, cd=79, tl=1801, v=true, d=false], regenerated UUID for EntityItemFrame['entity.ItemFrame.name'/2637, uuid='42be3773-bedc-4dfa-8229-af5e3579571f', l='world2', x=-1133.97, y=12.50, z=1268.50, cx=-71, cd=79, tl=0, v=false, d=false]. See https://github.com/PaperMC/Paper/issues/1223 for discussion on what this is about.
[DUPE-UUID] Duplicate UUID found used by EntityItemFrame['entity.ItemFrame.name'/86, uuid='ea9c3979-88cb-4f83-b7ba-3f91c3669de6', l='world2', x=-1133.97, y=12.50, z=1266.50, cx=-71, cd=79, tl=1801, v=true, d=false], regenerated UUID for EntityItemFrame['entity.ItemFrame.name'/2638, uuid='1763f679-755b-45d3-b183-ab155dd512fe', l='world2', x=-1133.97, y=12.50, z=1266.50, cx=-71, cd=79, tl=0, v=false, d=false]. See https://github.com/PaperMC/Paper/issues/1223 for discussion on what this is about.

EDIT: Just now received random messages

ERROR 24/07/2018 17:20:43
17:20:43EntityHumanNPC['PartyPete'/2639, uuid='bf818229-ebef-2582-a190-78a20f714930', l='world2', x=-1128.50, y=11.00, z=1268.18, cx=-71, cd=72, tl=4198, v=true, d=false](PartyPete at -1128.501391103238,11.0,1268.175450872001) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
Console 24/07/2018 17:20:43
17:20:43java.lang.Throwable: null
at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkProviderServer.a(ChunkProviderServer.java:311) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.WorldServer.save(WorldServer.java:1082) ~[patched_1.12.2.jar:git-Paper-1511]
at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.save(CraftWorld.java:813) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:805) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-1511]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
ERROR 24/07/2018 17:22:31
17:22:31EntityHumanNPC['usersname'/3802, uuid='d7f89ce7-1283-211c-95e7-ccd89ebb3904', l='world2', x=-1156.30, y=72.06, z=1166.51, cx=-71, cd=73, tl=780, v=true, d=false](usersname at -1156.2998,72.0625,1166.5135) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
Console 24/07/2018 17:22:31
17:22:31java.lang.Throwable: null
at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkProviderServer.a(ChunkProviderServer.java:311) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.WorldServer.save(WorldServer.java:1082) ~[patched_1.12.2.jar:git-Paper-1511]
at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.save(CraftWorld.java:813) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:805) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-1511]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
ERROR 24/07/2018 17:26:52
17:26:52EntityHumanNPC['usersname2'/3799, uuid='3c642697-a0c7-2771-bbc4-94b7a94e3f07', l='world2', x=-1156.73, y=72.00, z=1164.58, cx=-70, cd=72, tl=5999, v=true, d=false](usersname2 at -1156.7264,72.0,1164.5788) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
Console 24/07/2018 17:26:52
17:26:52java.lang.Throwable: null
at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkProviderServer.a(ChunkProviderServer.java:311) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.WorldServer.save(WorldServer.java:1082) ~[patched_1.12.2.jar:git-Paper-1511]
at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.save(CraftWorld.java:813) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:805) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-1511]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
ERROR 24/07/2018 17:27:31
17:27:31EntityHumanNPC['usersname'/3802, uuid='d7f89ce7-1283-211c-95e7-ccd89ebb3904', l='world2', x=-1156.30, y=72.06, z=1166.51, cx=-71, cd=73, tl=6780, v=true, d=false](usersname at -1156.2998,72.0625,1166.5135) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
Console 24/07/2018 17:27:31
17:27:31java.lang.Throwable: null
at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkProviderServer.a(ChunkProviderServer.java:311) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.WorldServer.save(WorldServer.java:1082) ~[patched_1.12.2.jar:git-Paper-1511]
at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.save(CraftWorld.java:813) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:805) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-1511]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
ERROR 24/07/2018 17:32:32
17:32:32EntityHumanNPC['usersname'/3802, uuid='d7f89ce7-1283-211c-95e7-ccd89ebb3904', l='world2', x=-1156.30, y=72.06, z=1166.51, cx=-71, cd=73, tl=12780, v=true, d=false](usersname at -1156.2998,72.0625,1166.5135) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
Console 24/07/2018 17:32:32
17:32:32java.lang.Throwable: null
at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.ChunkProviderServer.a(ChunkProviderServer.java:311) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.WorldServer.save(WorldServer.java:1082) ~[patched_1.12.2.jar:git-Paper-1511]
at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.save(CraftWorld.java:813) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:805) ~[patched_1.12.2.jar:git-Paper-1511]
at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-1511]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

I have also noticed now, when I restart and join. The mobs in my test area disappear with no info in the console about it.

[axemwa]

Dam wish I could help test this but my home internet has been down last few days due to weather conditions and have been using my phones internet instead via tethering. Have been trying to find out how to port forward from my phone but it doesn't look like I can as the recommended android apps I couldn't get to work :(

[Cryptite]

I also get this on my server. FWIW "Halfthal" is a Citizens NPC for us (same errors as above looks like):

[08:51:36] [Server thread/ERROR]: EntityHumanNPC['Halfthal'/3747, uuid='0f47d9fd-ed3e-28ca-863c-1dffadf5d20b', l='bigboi', x=2297.75, y=66.00, z=3028.15, cx=143, cd=187, tl=252, v=true, d=false](Halfthal at 2297.7492,66.0,3028.1543) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223 java.lang.Throwable: null at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-"d155366"] at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-"d155366"] at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-"d155366"] at net.minecraft.server.v1_12_R1.ChunkProviderServer.unloadChunk(ChunkProviderServer.java:410) ~[patched_1.12.2.jar:git-Paper-"d155366"] at net.minecraft.server.v1_12_R1.ChunkProviderServer.unloadChunks(ChunkProviderServer.java:348) ~[patched_1.12.2.jar:git-Paper-"d155366"] at net.minecraft.server.v1_12_R1.WorldServer.doTick(WorldServer.java:310) ~[patched_1.12.2.jar:git-Paper-"d155366"] at net.minecraft.server.v1_12_R1.MinecraftServer.D(MinecraftServer.java:904) ~[patched_1.12.2.jar:git-Paper-"d155366"] at net.minecraft.server.v1_12_R1.DedicatedServer.D(DedicatedServer.java:423) ~[patched_1.12.2.jar:git-Paper-"d155366"] at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:773) ~[patched_1.12.2.jar:git-Paper-"d155366"] at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-"d155366"] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]

[mibby]

This error spams quite often with Paper 1511 and Citizens. :(

[06:02:03] [Server thread/ERROR]: EntityHumanNPC['Jobs Clerk'/949189, uuid='1cc9626d-0378-2943-8c5a-1597d7aaf2b5', l='spawn', x=20.50, y=48.00, z=0.50, cx=0, cd=0, tl=227, v=true, d=false](Jobs Clerk at 20.5,48.0,0.5) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
java.lang.Throwable: null
    at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.unloadChunk(ChunkProviderServer.java:410) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.unloadChunks(ChunkProviderServer.java:348) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.WorldServer.doTick(WorldServer.java:310) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.D(MinecraftServer.java:904) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.DedicatedServer.D(DedicatedServer.java:423) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:773) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-1511]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[06:02:03] [Server thread/ERROR]: EntityHumanNPC['Dark Zone'/949188, uuid='8c00952c-92af-266a-afb0-3680301a1769', l='spawn', x=-30.50, y=54.00, z=9.50, cx=1, cd=0, tl=227, v=true, d=false](Dark Zone at -30.5,54.0,9.5) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
java.lang.Throwable: null
    at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.unloadChunk(ChunkProviderServer.java:410) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.unloadChunks(ChunkProviderServer.java:348) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.WorldServer.doTick(WorldServer.java:310) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.D(MinecraftServer.java:904) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.DedicatedServer.D(DedicatedServer.java:423) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:773) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-1511]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[06:02:03] [Server thread/ERROR]: EntityHumanNPC['Maintenance'/949197, uuid='fa53e359-5877-232b-ab6d-bfb07e744070', l='spawn', x=-32.50, y=54.00, z=-5.50, cx=1, cd=0, tl=227, v=true, d=false](Maintenance at -32.5,54.0,-5.5) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
java.lang.Throwable: null
    at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.unloadChunk(ChunkProviderServer.java:410) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.unloadChunks(ChunkProviderServer.java:348) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.WorldServer.doTick(WorldServer.java:310) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.D(MinecraftServer.java:904) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.DedicatedServer.D(DedicatedServer.java:423) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:773) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-1511]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

[aikar]

Well those Cititzens logs are surely interesting... Do these NPC's move around or stand still?

I could not get these logs to show up in a fly through the world, even when setting up contraptions to make entities ride minecarts on a 4 chunk corner.

So thats interesting that citizens is triggering it often.

Have you also seen dupe uuid issues with these citizens?

@LogGits those Bush Regan NPC's, have you encountered one of them been missing before?

I'm also highly suspect about entities disappearing without logs... im wondering if its plugins removing them.

Has anyone triggered it for non citizens?

FYI None of these logs should of changed behavior vs previous builds, so please don't go downgrading. It was just purely more informative messages.

Have those with the citizens lines encountered citizens ever go missing?

[Cryptite]

In my case, none of my Citizens move; however, my plugin does despawn/spawn them based on some conditions (anybody nearby, etc)

[aikar]

@Cryptite yours triggered in a chunk unload. Then that's very odd.

Your Cititzen was registered to a chunk 2 chunks away from its current locX and locZ values...

It SAYS it should be at x=2297.75, y=66.00, z=3028.15 Can you go there and get your citizens ACTUAL x/y/z for me?

[Cryptite]

Those coordinates are, in fact, exactly where that NPC is. Attached is me inside of the guy:

[aikar]

well that is interesting. The Citizen is currently REGISTERED to the chunk with the middle location 2296,66,3000

Can you go see if anything stands out there? It's about 2 chunks away.

[Cryptite]

Nothing of real note there...

[stevmei]

Could you open the sand? Check for height y = 66 😄

[Cryptite]

Here's that region file as requested by Aikar: https://1drv.ms/u/s!ArrIMXzP7wfBhbY-DhnVN4GU6_E1IA

[LogGits]

@aikar some of my npcs are moving and I haven't seen any of them disappear as they are in spawn area.

Also something that i find interesting is that wolves, tamed, untamed, named, unnamed, don't vanish. Is there something there that could be worth looking at?

[Baktus79]

[18:18:48 ERROR]: EntityHumanNPC['Baktus_79'/8850, uuid='9d9e5a90-5555-2dd7-b9fd-a6a5649a3875', l='world', x=460.21, y=75.00, z=-529.48, cx=-3, cz=10, tl=1, v=true, d=false](Baktus_79 at 460.2115,75.0,-529.4823) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
java.lang.Throwable: null
        at net.minecraft.server.v1_13_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:575) ~[patched_1.13.jar:git-Paper-32]
        at net.minecraft.server.v1_13_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:303) ~[patched_1.13.jar:git-Paper-32]
        at net.minecraft.server.v1_13_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:236) ~[patched_1.13.jar:git-Paper-32]
        at net.minecraft.server.v1_13_R1.ChunkProviderServer.a(ChunkProviderServer.java:297) ~[patched_1.13.jar:git-Paper-32]
        at net.minecraft.server.v1_13_R1.WorldServer.save(WorldServer.java:900) ~[patched_1.13.jar:git-Paper-32]
        at org.bukkit.craftbukkit.v1_13_R1.CraftWorld.save(CraftWorld.java:799) ~[patched_1.13.jar:git-Paper-32]
        at net.minecraft.server.v1_13_R1.MinecraftServer.v(MinecraftServer.java:946) ~[patched_1.13.jar:git-Paper-32]
        at net.minecraft.server.v1_13_R1.MinecraftServer.run(MinecraftServer.java:812) ~[patched_1.13.jar:git-Paper-32]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

Im not sure if this is needed :D I am doing as told! Can add that this entity is created with Citizens.

[0XE4]

[00:20:27 ERROR]: EntityHorse['Horse'/48540, uuid='f047dd34-a0ff-4c82-9e86-b481356a68cb', l='world', x=-xxxxxx.14, y=64.00, z=xxxxx.11, cx=-xxxxx, cd=xxxx, tl=xxxxx, v=true, d=false] will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
java.lang.Throwable: null
        at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-1511]
        at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-1511]
        at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-1511]
        at net.minecraft.server.v1_12_R1.ChunkProviderServer.a(ChunkProviderServer.java:311) ~[patched_1.12.2.jar:git-Paper-1511]
        at net.minecraft.server.v1_12_R1.WorldServer.save(WorldServer.java:1082) ~[patched_1.12.2.jar:git-Paper-1511]
        at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.save(CraftWorld.java:813) ~[patched_1.12.2.jar:git-Paper-1511]
        at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:805) ~[patched_1.12.2.jar:git-Paper-1511]
        at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-1511]
        at java.lang.Thread.run(Unknown Source) [?:1.8.0_172]

This has started showing up now so I thought I would post in case it helps.

[aikar]

@0XE4 thanks, it def helps to see it triggered on NOT a Citizen....

Citizens is def triggering the condition often, but seeing it on a normal entity leads to my suspicions on entity duping/uuid duping.

after you reload the world, does that horse then trigger uuid regeneration message?

[Baktus79]

[12:40:58] [Server thread/ERROR]: EntityPlayer['Cam3lot'/585992, uuid='928c658a-9441-4212-b81e-7b35c39d330a', l='world', x=7806.10, y=65.38, z=579.37, cx=-347, cd=-132, tl=20723051, v=true, d=false](Cam3lot at 7806.1016778160565,65.38020878090896,579.3726262315796) will be leaving this chunk but saved to it. Report this to https://github.com/PaperMC/Paper/issues/1223
java.lang.Throwable: null
    at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveEntities(ChunkRegionLoader.java:359) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkRegionLoader.saveChunk(ChunkRegionLoader.java:150) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.saveChunk(ChunkProviderServer.java:251) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.ChunkProviderServer.unloadChunk(ChunkProviderServer.java:410) ~[patched_1.12.2.jar:git-Paper-1511]
    at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.lambda$unloadChunk0$0(CraftWorld.java:255) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MCUtil.ensureMain(MCUtil.java:69) ~[patched_1.12.2.jar:git-Paper-1511]
    at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.unloadChunk0(CraftWorld.java:248) ~[patched_1.12.2.jar:git-Paper-1511]
    at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.unloadChunk(CraftWorld.java:244) ~[patched_1.12.2.jar:git-Paper-1511]
    at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.unloadChunk(CraftWorld.java:217) ~[patched_1.12.2.jar:git-Paper-1511]
    at org.bukkit.craftbukkit.v1_12_R1.CraftWorld.unloadChunk(CraftWorld.java:213) ~[patched_1.12.2.jar:git-Paper-1511]
    at org.bukkit.craftbukkit.v1_12_R1.CraftChunk.unload(CraftChunk.java:130) ~[patched_1.12.2.jar:git-Paper-1511]
    at com.volmit.react.Gate.unloadChunk(Gate.java:806) ~[?:?]
    at com.volmit.react.action.ActionPurgeChunks.purge(ActionPurgeChunks.java:150) ~[?:?]
    at com.volmit.react.action.ActionPurgeChunks$2.run(ActionPurgeChunks.java:86) ~[?:?]
    at com.volmit.react.util.Task$3.run(Task.java:80) ~[?:?]
    at org.bukkit.craftbukkit.v1_12_R1.scheduler.CraftTask.run(CraftTask.java:63) ~[patched_1.12.2.jar:git-Paper-1511]
    at org.bukkit.craftbukkit.v1_12_R1.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:423) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.D(MinecraftServer.java:839) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.DedicatedServer.D(DedicatedServer.java:423) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.C(MinecraftServer.java:773) ~[patched_1.12.2.jar:git-Paper-1511]
    at net.minecraft.server.v1_12_R1.MinecraftServer.run(MinecraftServer.java:665) ~[patched_1.12.2.jar:git-Paper-1511]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

[aikar]

@Baktus79 well that's scary.

That log indicates the player teleported, and React unloaded the chunk while the player was still registered to it...

It seems we have something that's moving Entities but doesn't process chunk registrations instantly....

So we then have entities saved into 1 chunk, thats then also saved in another :/

Note these are vanilla level issues, but I'm suspecting that because of our regeneration of UUIDs, that this could duplicate entities.

has anyone encountered entities duplicating? Previously in vanilla, the dupe would be ignored and only 1 loaded.

The issue is, we have some non dupe entities sharing the uuid, so we do want to regenerate those....

might do something like if this.distance(That) < 16 AND abs(this.ticksLived - that.ticksLived) < 1000, then delete the dupe, otherwise regen.

But id like to stop the duping risk all together, which means fixing the vanilla bug. These logs have helped give some indication on what the problem is, so thanks :)

[stevmei]

Thanks @aikar for looking into it dealing with Mojangs ... ehm... yea. 😈

[Resoluciones]

Hi!

Follows log in case it's useful: https://hastebin.com/uvagacibib.cs

Thank you!!!

EDIT: I think these type of errors are always happening when there is any source of lag, in example, when starting the server, or after a random teleport, when the server needs to create or load chunks. Sometimes after those errors happen the players are kicked with a "time out" error. And by the way, chunk loading in 1.13 is extremely heavier in comparison with same plugins installed in a 1.12 server, even with 10GB of memory and with just 1 player; have we somehow "involuted" with the new version?

[Brokkonaut]

https://pastebin.com/raw/A5Yq1kFB

1.13, no plugins, no players online - it just showed up in the console after the server was idle for more than one hour