Investigation of server lag

[inklit]

Hi, I've been investigating some probable causes of server lag using a JVM Sampler, and I found out that about 16% of my server's CPU usage is from CCTweaks.

This is a rather alarming discovery, considering how much computers are used on the server.

Check out this for some more details on what it is doing. http://i.imgur.com/LsSf7c7.png Here you can see an Aikar Timings Report that backs up the report from WarmRoast https://timings.aikar.co/v2/?id=43320e34dd6842fc9ada150bc8d5d80d&section=lag

For comparison, our OC networks use no noticable amount of CPU time, and this amount of CPU time is the equivalent of like 20K buildcraft pipes.

Is this behavior expected and normal? Do you think that it could be reduced?

[SquidDev]

Ouch. Networks should be using as little CPU as possible. By the looks of it the slowest bit is the controller validator: it would probably be sensible for me to allow disabling it.

It it possible for you to give me the data behind the above image so I can see what else is slow? I'm seeing wiredmodems are taking 20% of tick time which is awful but confusing as they don't do much - I'll see if there is a way prevent cables from ticking.

[inklit]

I can't seem to get my profiler to work accurately again for some reason, it broke so I can't really give you the data behind the above image.

I am currently in the process of destroying a MASSIVE network (see #117)

This network is over 4000 blocks big...I can't even believe my players would create a network this big but don't realize that it doesn't even really work since there's a limit on how far data can be sent...

Once it's removed, it might be a little harder to test for this issue since the data will be less accurate without this gigantic network.

[inklit]

https://timings.aikar.co/v2/?id=7395aa5c664548c6ac32f5ff0e79d210 The network was removed, and you can see a noticeable difference. So much so that I don't even consider them as "lagging the server" any more.

[SquidDev]

Well this shouldn't be too hard to reproduce 😄: just create a lot of nodes.

[inklit]

I think this is a little bit more than just "a lot" of nodes...hahaha.

If you're interested in finding out what else may be causing lag, check out WarmRoast at https://github.com/sk89q/WarmRoast, it is the profiler I used to figure out what the source of the lag was.

Sponge also has Aikar's timing system built in, so you could use that as well by simply running /sponge timings report, and it should automatically upload the timings report to his site. I only use this as a re-confirmation of what WarmRoast reported, since it is a little bit inaccurate (or rather, WarmRoast is easier to find the exact source) as far as I can tell.

[SquidDev]

So a quick /fill ~ ~ ~ ~50 ~ ~50 computercraft:CC-Cable 12 (create a lot of interconnected cabled) confirmed my suspicions: the network controller is slow. I've set it to be disabled by default.

However this hasn't solved the problem: my network of ~10000 nodes takes about 100 seconds when a node is destroyed. This is obviously not great.

Edit: I've found room for some more optimisations and reduced most events to near instant. There was some legacy code lying around which I hadn't updated to benefit from the new system. There is still room for some improvement though.

[SquidDev]

The above commit says it all: I've done fair rewrite of the network controller which reduces a lot of unnecessary calls and makes everything more deterministic. I'm able to add and remove 2.5k nodes to a network of 20k at a time with unnoticeable delay so hopefully this fixes everything :smile:.

[inklit]

Sweet! Thanks!