Closed Teleriann closed 3 years ago
Hello @Teleriann ! 👋🏻
Sorry to hear this is not going smoothly.
Server was running just fine, we are using achievements plugin for a longer time. But couple of days ago server started to experience lag spikes.
What happened a couple of days ago? Did you update the plugin or its configuration? Did you install or update any other plugins? Did you upgrade or change anything else on your server?
We have updated lots of plugins 9 days ago, achievements with cmi, residence and along with others... We are updating our database version and things like that as soon as possible, so all of that "system" things on our dedicated servers too. When we updated the plugin, we let the plugin recreate its config and then we have put our achievements there, it was running all right. Aach plugin lags started like 2 days ago, players started to reporting issue with that lag spikes, so I have looked to timings and there it was. If I remember correctly, only thing related to minecraft, at the day when lag spikes occurred was installing dynmap plugin to our server. But i already tried to remove that plugin, and let server run without it, but there was no difference in performance, so thats probably not it.
There you have timings from just now: https://timings.aikar.co/?id=2d636c1a984a4fba9088107ebf53e8ab#plugins It is not that high right now, because lag spike happened just for like 4 times in that 4 minutes. Like i said, i do not understand, why it is just that random, I have opened console on one monitor, then server tps with minecraft on other one. I it interesting that even when some players gets lots of achievements at the same time, it is running all right, but then at one random moment when there is not happening anything about aach at all, it just lag spikes.
And of course i tried to remove aach plugin, without it there were not any lag spikes at all. And again, sorry for my terrible english.
We are updating our database version and things like that as soon as possible, so all of that "system" things on our dedicated servers too
Did this happen by any chance a couple of days ago?
Could you share a little more about your configuration? Based on the latest timing report, it's somewhat bizarre that Crafts
achievements have such a big performance hit, but on the other hand Kills
, Places
and Breaks
have almost negligible impact. Underneath the hood, whether it be in the database structure or the actual listener code, all four categories are very similar. I'm surprised there is a gap off several orders of magnitude on average - either there's something peculiar in the config, either a lot of different players are performing craft actions, whereas all the kills, places and breaks are performed by only a few different players.
We have updated our debian 6 days ago, nothing else really.
Btw, our server specs are: Intel® Core™ i9-9900K DDR4 RAM Nvme ssd 1 GBit/s connection
I can send you whole aach configuration with achievements in some private message if needed. If that would help.
There are our plugins, that might help too.
I can send you whole aach configuration with achievements in some private message if needed
That may be useful. Send it by mail, you can find it on my GitHub profile.
There are our plugins, that might help too.
Out of fascination, do the lags happen if you disable all other plugins? This may be useful to tests, though admittedly a little impractical. :)
So, I sent you our whole config through email. Also, I can try today to stay up late, and wait for player count to be low, and then examine whether the lag spikes will be smaller, or maybe not at all. Then, if there will be, even if "there are not players on server", I will disable all plugins to see if it will stop.
So, when there is low player count on the server, lag spikes are almost not visible at all. But when I open timings, most of the times there are still aach taking most of the server performance. More players = bigger lag spikes. or probably more noticable ones. But it gets interesting when player count is reaching 10 to 0 players. There are not lag spikes at all, or anything like that, not even in timings. So it is probably something related to some player data, and that is causing that lag spikes. Im sure that you will understand it better than me.
Thank you for your help, i will be glad for any suggestions.
There are some timings with lower player count: https://timings.aikar.co/?id=23d5774397b14d99bdba911013ccbe1a#plugins Timings with player count close to 0: https://timings.aikar.co/?id=23d5774397b14d99bdba911013ccbe1a#plugins
I had a look at your config. One thing that I noticed is that some achievement Name
parameters are really long, for example: places_10000_white_wool|orange_wool|red_wool|magenta_wool|light_blue_wool|yellow_wool|lime_wool|pink_wool|gray_wool|light_gray_wool|cyan_wool|purple_wool|blue_wool|brown_wool|green_wool|black_wool
Do they actually work? I would expect an error to be thrown in the console, there's a 64 character limit in their name imposed in the database: https://github.com/PyvesB/advanced-achievements/blob/e56acc1f7e5eb3aab50c2631c9c6576895f42fe2/advanced-achievements-plugin/src/main/java/com/hm/achievement/db/DatabaseUpdater.java#L104
However, I don't think this would explain any of the performance problems.
Looking at the lastest timing (https://timings.aikar.co/?id=23d5774397b14d99bdba911013ccbe1a#plugins), I noticed there was exactly one instance of ConnectionsListener
in that timeframe. Could you tell me a little more about the player who connected at that time? Was it their first connection during the day, or had they already connected previously?
I also noticed that some of the other plugins had quite high resource usage in that timing extract. Do those other top plugins also happen to use MySQL? If so, did you notice them slow down in the past few days, compared to a few weeks ago for example?
So, yes, they do work, we had some problems with that ones, but we have manually changed character length in database, so they work just fine.
About that timing and players, 2 players have connected during that timings period, one of them was already on the server before, like couple of minutes ago, and sent 2000 messages/commands to the chat. Second one have connected for the first time during that day.
And for that other plugins not really, everything is normal, you are probably pointing to leaderheads, we are using that plugin for couple of years, yes, it is connected to the database. But its behaviour has not changed at all, during the day, it is not that visible in timings, but when there are no players, and "nothing" is happening on the server, then it will pop up, of course that it is taking some server performance too, but there are not any lag spikes or anything like that related to other plugins.
When im looking at timings right now, it is quite all right, server is not really lagging, i do not understand it: https://timings.aikar.co/?id=8fd01e94efc84ac5822ff77fdf9590bb#timings
Im once again sorry for my english, I hope that you understand everything.
About that timing and players, 2 players have connected during that timings period, one of them was already on the server before, like couple of minutes ago, and sent 2000 messages/commands to the chat. Second one have connected for the first time during that day.
Are you sure about that? The timings indicates only one count of ConnectionsListener
, that should correspond to only one player.
When im looking at timings right now, it is quite all right, server is not really lagging, i do not understand it
Advanced Achievements caches a lot of what it reads from the database. If the same players stay connected for a long time, you'll get a lot of cache hits, speeding things up. If there's a big player turnover, then that means the plugin needs to go back to the database a lot more often.
Yes, about that connections, im pretty sure.
Right now, there are 120 players online, and aach have 7% in timings, second one is cmi with 1.3%. It is lagging, but not that much. I had one more idea, we will try to move some things from our database to another database server, so there will be less data, I do not think that it will solve problems with lag spikes, but I'll give it a chance.
I had one more idea, we will try to move some things from our database to another database server, so there will be less data
Okay, let me know how that goes.
In the meantime, a few other questions to get a better understanding:
Right now we are using 10.3.27-MariaDB, but we will be updating soon. Our localhost is local right now. No, we are not sharing it between multiple servers, each server has own achievements.
Btw, look at this timings, thats horrible: https://timings.aikar.co/?id=6ae1f0b1bedd4e1ab6497acf86c2ce66#plugins
Sorry for delayed answer, we tried some tests with aach, but nothing really helped. Thanks for your help.
If each server has its own achievements and aren't sharing, is there any specific reason why you chose MySQL? I would expect MySQL/MariaDB to be the slowest of all four database types supported by Advanced Achievements, with the default H2 probably the fastest in most cases.
If you want to stick with the current database, it would be good to give the 10.5.x MariaDB series a go, that may well help.
Based on some of the timings you provided, I have a few ideas to improve performance a little. Will try to work on them this week. I don't expect a miracle, database reads are regularly taking over 100ms each on your server, but it should help a little. Stay tuned!
Sure, thank you very much. We are used to use MySQL for all plugins that support this feature, we are kinda used to it, work with data/edit them etc... in phpmyadmin. But if it would help to heavily reduce lags from aach, we can try it.
Also, i do not really work with this, whats the best way to transfer data from our mysql to h2 or just basic sql?
I've never done the conversion myself, but a SQL dump may be a good starting point. Alternatively, these resources may help:
As you'll notice from the above commit references, I'm also pushing some small performance improvements under this issue. Will aim at testing and releasing later this week.
Just to be sure, we are using premium aach plugin from spigot, i should just download it from github now? I know that you have that plugin in some other version for free too. Thanks and sorry for stupid question.
I recommend only downloading the releases from Spigot, as you've been doing until now.
If you're feeling adventurous, you can recompile the source code on GitHub yourself, but it's a bit of extra effort, and there is no guarantee that the code you'll compile has yet been fully tested. Use with care if you decide to do so, you'll effectively become a beta/alpha tester! 😉
Release 7.0.4 is available for download. Again, I'm not expecting a miracle, but it should help with performance a bit. In particular, ConnectionsListener
shouldn't be as bad as in some of the timings you have shared so far.
It is still not perfect, but it is much much better. Thank you very much for your support, here are some screenshots too, i have also set distance and onlinetime checking to once per 30minutes, so that probably helped too.
Thank you for your support once again.
Ok so, it is not as good as in morning, when server is in prime time, lots of players are connecting or disconnecting, more players are online so probably because of that it runs poorly.
7% is quite too much, compared to other plugins that have around 1%, but it is still better than 30% tho
Also, there is one more thing I just wanted to ask, how much time should aach take to load on server start? It is usually around 2 minutes, second most time consuming plugin to load is residence that takes around 15seconds. It is not that important, it is just server restart at 4am no one cares about that, but sometimes quite annoying when I need server restart around prime time.
Thanks for the feedback. Glad to here that the work has paid off and that things have improved a little.
Also, there is one more thing I just wanted to ask, how much time should aach take to load on server start? It is usually around 2 minutes, second most time consuming plugin to load is residence that takes around 15seconds. It is not that important, it is just server restart at 4am no one cares about that, but sometimes quite annoying when I need server restart around prime time.
That's a really useful observation! I had only read through the config you sent by mail and hadn't actually tried it on my own server. The plugin normally takes at most a few seconds to launch, and with your config, it takes 5 minutes on my laptop!
The step which seems to be ridiculously slow is Registering permissions...
. Is this consistent with the behaviour you observe on your server? That step has got nothing to do with the database. It goes through all achievement names, and registers one permission for each one of them. The code is really simple, even someone who isn't familiar with programming could make sense of it:
Permission achievementParent = new Permission("achievement.*", PermissionDefault.OP);
for (String name : achievementMap.getAllNames()) {
String permissionNode = "achievement." + name;
if (pluginManager.getPermission(permissionNode) == null) {
Permission perm = new Permission(permissionNode, PermissionDefault.TRUE);
perm.addParent(achievementParent, true);
Bukkit.getPluginManager().addPermission(perm);
}
}
I've done a bit of micro-benchmarking: adding the first few permissions is fast, addParent
takes ~2ms and addPermission
takes ~1ms on my machine. When you reach the 5000th permission, things has dramatically slowed down: those calls take ~80ms and ~55ms respectively. Spigot/Bukkit's permissions system does not scale very well, taking over 0.1s to register a single new permission with a parent is not acceptable.
Given the above analysis, I'm no longer sure the database is actually the root cause of the slowness. Could you please install the Spark plugin, run /spark profiler
for a few minutes when the server is having trouble, and share the resulting link here?
Sure, i will tomorrow, right now only 60 players are online, and server is running smoothly. We have spark already installed, we are using it for things like that or heap dumps.
Thanks for your support <3 .
Based on some further analysis of your timings, I've pushed a couple of commits that should help out a little more - again no miracles to expect.
I've also raised a Spigot bug to flag the performance issue with the permissions system. Feel free to upvote it or add any information that you think may be useful. 😉
Once again, thanks for help. So, there are two spark profiles as you requested: 1. https://spark.lucko.me/yvCAQ2NbmA
Im not really familiar with spark profiles, so you will know better. Server was not having trouble really tho, it has around 19 TPS. But there were some drops, so there might be something useful. That profiles are made on latest spigot version, without these last optimizations. And again, im sorry for my english, i hope that you can understand everything.
Thanks for sharing the profiles. They confirm some of the previous analysis. As the next steps, I suggest:
And again, im sorry for my english, i hope that you can understand everything.
No worries, I'm not had any trouble understanding so far. 😉
Version 7.0.5 has been released, you can progress the first bullet point from https://github.com/PyvesB/advanced-achievements/issues/1003#issuecomment-826067645.
Haven't heard back for a while, so I'll assume things are good enough in the latest releases or that you're still waiting to carry out the MariaDB refresh. Will close for now.
Thanks for the great interactions on this issue! 😉
:beetle: Description
Server was running just fine, we are using achievements plugin for a longer time. But couple of days ago server started to experience lag spikes. Heavy lag spikes that tps went from 20 to 7 with 3-4seconds of lag spike. Of course that i have looked to server timings, and i found that achievements are causing unusual heavy server load. We have over 5000 achievements on our server, so we tried to reduce that to lower number, but it did not help. Iam reading our timings alot, to optimize our server performance, achievements always had like 2% of server performance, but right now, sometimes it hits 20%.
I have tried to put larger numbers in seconds to config to reduce the lag, but it did not help at all, i even tried to look for some pattern, when the server lag spikes, but it looks like that it is "random".
I will be glad for every piece of help that i could get, thank you. Also, im sorry for my english.
Timings for example: 1. https://timings.aikar.co/?id=c13a7de340b440eebc82951f6164c318#plugins 2. https://timings.aikar.co/?id=0308f55ee0234c708dd0d1f4e547f32d#plugins
:page_facing_up: Server logs
There are not any errors or anything like that in our console or logs, thats why i have problem to resolve that issue.
:wrench: Configuration