Glitches with Citizens2

SlimeDog commented 3 years ago

Expected behavior

No WARNs for existing NPCs.

Observed/Actual behavior

The paper fix related to Implement-Chunk-Priority-Urgency-System-for-Chunks" (Fixes #5980) found in paper-186 (and paper-185, and probably paper-183 (not verified)) does not play well with Citizens. The issue does not exist in paper-184, or in comparable versions of Spigot. This issue did not exist in or prior to Paper-182.

Steps/models to reproduce

Copy server content from existing up-to-date Spigot server (3217-Spigot-b166a49-4f6bcc8). This is our SOP, to ensure that we are testing comparatively on Spigot and Paper.
Change server software from Spigot to Paper. No other changes.
Update from Paper-184 to Paper-186.
Start server.
Observe WARN messages -- see link above.

Plugin list

A very long list. Available if really necessary. Relevant plugins: Citizens 2.0.28-SNAPSHOT (build 2362) Denizen 1.2.1-SNAPSHOT (build 5703-DEV) Sentinel 2.3.5-SNAPSHOT (build 425)

Paper version

version [15:24:40 INFO]: This server is running Paper version git-Paper-186 (MC: 1.17.1) (Implementing API version 1.17.1-R0.1-SNAPSHOT) (Git: 72ce684) You are running the latest version Previous version: git-Paper-184 (MC: 1.17.1)

Agreements

[X] I am running the latest version of Paper available from https://papermc.io/downloads.
[X] I have searched for and ensured there isn't already an open issue regarding this.
[X] My version of Minecraft is supported by Paper.

Other

No response

SlimeDog commented 3 years ago

Issue persists in Paper-187.

SlimeDog commented 3 years ago

Even worse, sometimes the server just hangs on start-up, trying to build the world where the identified NPCs reside.

electronicboy commented 3 years ago

Those warnings won't cause the server to hang, if the server is hanging, you'd need to provide jstack reports or something

the removed patch is likely 100% unrelated to this as it did not touch the entity management system at all, but, god knows here

SlimeDog commented 3 years ago

Please give a link to specific instructions for what you want me to provide. The server does not crash, so there is no crash reporting. I will produce a jstack report if instructions are available. In the meantime, I have reverted to paper-184, which does not exhibit either problem.

e-im commented 3 years ago

Depending on how you're hosting it, doing that may be pretty much impossible. However assuming you have access to the machine, see this guide - just skip the jira specific stuff and scroll down to the jstack section.

SlimeDog commented 3 years ago

I host my own test servers. I'll give it a try. Thanks.

SlimeDog commented 3 years ago

I've spent the better part of today trying unsuccessfully to generate a hung server (the WARN messages persist). I'll keep trying occasionally, but there are other tasks to which I need to attend.

SlimeDog commented 3 years ago

OK, finally. Here is jstack output for a hung server. And here is the associated log. I will leave the server in this state until you tell me to free it, in case you want additional information.

SlimeDog commented 3 years ago

Here are the last few entries in the console (which is, of course, unresponsive at this point)

[18:14:26 INFO]: [Server Tutorial Plus] PlaceholderAPI has been found!
[18:14:26 INFO]: [Server Tutorial Plus] Servertutorial enabled successfully!
[18:14:26 INFO]: [SavageGraveyards] Enabling SavageGraveyards v1.16.1
[18:14:26 INFO]: [SavageGraveyards] Language file en-US.yml successfully loaded.
[18:14:26 INFO]: [SavageGraveyards] Multiverse-Core detected.
[18:14:26 INFO]: [SavageGraveyards] Sound file sounds.yml successfully loaded.
[18:14:26 INFO]: [SavageGraveyards] SQLite datastore initialized.
[18:14:26 INFO]: [CommandSigns] Enabling CommandSigns v2.2.0
[18:14:26 INFO]: [CommandSigns] Plugin vault detected
[18:14:26 INFO]: [CommandSigns] Vault economy [EssentialsX Economy] linked with command signs !
[18:14:26 INFO]: [Minecraft.CityWorld] [CityWorld] Loading settings for 'world_city'

electronicboy commented 3 years ago

at me.daddychurchill.CityWorld.Plugins.WorldEdit.PasteProvider_WorldEdit.loadWorldEdit(PasteProvider_WorldEdit.java:121)

That plugin caused a deadlock by calling getPlugin on a gen thread

at be.nokorbis.spigot.commandsigns.utils.CommandBlockValidator.isPlate(CommandBlockValidator.java:27)

Attempted to load a chunk in it's onEnable, hence why the plugin manager was already locked, and why the gen thread got stuck

SlimeDog commented 3 years ago

That resolves the hanging server issue. Unless I am mistaken, the issues with Citizens remain. In most cases, the server did not hang, but the Citizens NPCs trigger WARN messages, and have incomplete data. Reminder, this does not occur on previous Paper releases, or on Spigot releases.

SlimeDog commented 3 years ago

Issue persists with

> version
[18:15:02 INFO]: This server is running Paper version git-Paper-196 (MC: 1.17.1) (Implementing API version 1.17.1-R0.1-SNAPSHOT) (Git: 59d449d)
You are running the latest version
> version Citizens
[18:15:15 INFO]: Citizens version 2.0.28-SNAPSHOT (build 2362)
[18:15:15 INFO]: Website: http://www.citizensnpcs.co
[18:15:15 INFO]: Author: fullwall
> version sentinel
[18:15:19 INFO]: Sentinel version 2.3.5-SNAPSHOT (build 425)
[18:15:19 INFO]: Author: mcmonkey

Versions of plugins have not changed since originally reported.

SlimeDog commented 3 years ago

This issue persists on Paper 1.17.1-214. As a consequence, I cannot run Paper on my servers, since between 2 and 5 Sentinels are reliably broken. Fortunately, everything works on Spigot 1.17.1, with the same content (just a change of Minecraft variant).

electronicboy commented 3 years ago

The patch which you're saying is to blame makes very little sense unless Citizens is doing something really weird, in which our inclination to dig deep into citizens to work out what it's doing is really minimal given that the only changes that should have been made are prioritisation stuff which aren't really exposed to plugins, but, that patch is also being restored soon too

SlimeDog commented 3 years ago

The word "blame" is so fraught that I decline to use it. My only goal is to run parallel Spigot and Paper test servers (with the same content) that I have been doing since MC 1.16.5 without any issues, up to and including Paper-182. As stated above, if I revert to Paper-182, all is well (I just retested to confirm). Since then, not. I am doing my best to provide information that would allow efficient debugging.

SlimeDog commented 3 years ago

This issue persists on Paper 1.17.1-249 with

Citizens 2.0.28-SNAPSHOT (build 2367)
Sentinel version 2.3.5-SNAPSHOT (build 431)

As before, everything continues to work on Spigot 1.17.1-3236, with the same content (just a change of Minecraft variant).

ScuroK commented 3 years ago

I can confirm the same problems.

SlimeDog commented 3 years ago

This issue persists on Paper 1.17.1-266 with

Citizens 2.0.28-SNAPSHOT (build 2371)
Sentinel version 2.3.5-SNAPSHOT (build 431)

As before, everything continues to work on Spigot 1.17.1-3241, with the same content (just a change of Minecraft variant).

SlimeDog commented 3 years ago

This issue persists on Paper 1.17.1-275 with

Citizens 2.0.28-SNAPSHOT (build 2382)
Sentinel version 2.3.5-SNAPSHOT (build 432)

Consequently, Paper cannot be used in production, since various sentinels are glitched. As before, everything continues to work on Spigot 1.17.1-3242, with the same content (just a change of Minecraft variant).

SlimeDog commented 3 years ago

Further analysis at https://github.com/CitizensDev/Citizens2/issues/2635#issuecomment-926989052

Owen1212055 commented 2 years ago

Because it seems you have come up with a solution and that there is not a very easy way to reproduce this will be closed. Feel free to open another issue if you are able to reproduce this with clearer reproduction steps.

PaperMC / Paper