search

Creators-of-Create / Create

[Forge Mod] Building Tools and Aesthetic Technology
MIT License
2.96k stars 912 forks source link

RailwaySavedData.save() takes over 200ms on large train networks affecting server tickrate #4732

Open uberswe opened 1 year ago

uberswe commented 1 year ago

Describe the Bug

Our server has a large train network and when the state of the network is saved it causes a spike in the time it takes for that tick to finish processing. This occurs every 5 minutes.

Here is a spark profile of a tick that takes over 200ms https://spark.lucko.me/elXa0YUvvt

462e3d999480d502a860232d0aeeccd9

The train network for the spark profile above looks like this:

> /create trains
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: -+------<< Train Summary: >>------+-
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: Track Networks: 22
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: Signal Groups: 6226
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: Trains: 51
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: Nearest Graphs:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: 5df6d with 6558 Nodes
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> 1316 Signals
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> 162 Stations
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: 031a3 with 32 Nodes
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> 1 Stations
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: c401c with 41 Nodes
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: d442e with 1 Nodes
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: 7da3a with 1 Nodes
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: [...]
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: Nearest Trains:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: 0301a: Polar Line 02, 2 Wagons
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> On Track: 5df6d
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> Travelling to Bigswe Spawn Station (3m away)
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> Schedule, Entry 1, IN TRANSIT
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: [Remove]
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: af0b3: Polar Express 3, 3 Wagons
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> On Track: 5df6d
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> Waiting at: Uberswe Spawn Station
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> Schedule, Entry 4, POST TRANSIT
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: [Remove]
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: ef4c3: Polar Express 1, 3 Wagons
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> On Track: 5df6d
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> Travelling to Uberswe Passenger 1A (3378m away)
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> Schedule, Entry 0, IN TRANSIT
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: [Remove]
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: c007c: Polar Line 03, 2 Wagons
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> On Track: 5df6d
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> Travelling to BigSwe Pass - B (3416m away)
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> Schedule, Entry 0, IN TRANSIT
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: [Remove]
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: 1c179: Polar Express 4, 3 Wagons
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> On Track: 5df6d
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> Travelling to Uberswe -> Spawn (1001m away)
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:  -> Schedule, Entry 2, IN TRANSIT
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: [Remove]
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: [...]
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]:
[12:51:06] [Server thread/INFO] [minecraft/DedicatedServer]: -+--------------------------------+-

Reproduction Steps

  1. Create a large train network
  2. Wait for state to save
  3. See a tick that takes over 200ms to process

Expected Result

I would expect not to see spikes.

But I understand that a server won't be able to support an unlimited number of trains and networks. Perhaps the load could be spread over more ticks in some way.

Screenshots and Videos

No response

Crash Report or Log

No response

Operating System

Ubuntu 20.04.6 LTS (amd64)

Mod Version

0.5.0i

Minecraft Version

1.18.2

Forge Version

40.1.92

Other Mods

There was no crash but here are the mods

create_things_and_misc (v1.0.0) supermartijn642configlib (v1.1.6) cccbridge (v1.3.1) architects_palette (v1.3.1) connectivity (v1.18.2-3.2) mcwwindows (v2.1.1) laserio (v1.4.5) ctm (v1.18.2-1.1.5+5) mcwdoors (v1.0.9) pickupnotifier (v3.2.1) railways (v1.2.4+forge-mc1.18.2) mekanismgenerators (v10.2.5) jeresources (v0.14.1.171) cloth_config (v6.4.90) buddycards (v1.18.2-3.2.1) refinedstorage (v1.10.5) chipped (v2.0.1) industrialforegoing (v3.3.1.6) farmersdelight (v1.18.2-1.2.0) crashutilities (v4.1) simpleshops (v1.1.4) mcwfences (v1.0.7) supermartijn642corelib (v1.1.7) spark (v1.10.38) curios (v1.18.2-5.0.9.0) patchouli (v1.18.2-71.1) collective (v6.53) advgenerators (v1.1.0.6) worldedit (v7.2.10+1742f98) mekanismtools (v10.2.5) mcwroofs (v2.2.3) architectury (v4.11.90) curiouselytra (v1.18.1-5.0.1.0) computercraft (v1.101.2) aiimprovements (v0.5.2) lightoverlay (v6.0.5) trashcans (v1.0.17a) polylib (v1801.0.2-build.13) observable (v2.2.3) fastleafdecay (v28) codechickenlib (v4.1.3.480) sliceanddice (v1.1.3) extrautilitiesrebirth (v1.7.3) betteradvancements (v0.2.0.146) borderlesswindow (v1.18-1.4.0) handoveryouritems (v3.0) ftblibrary (v1802.3.11-build.177) ftbteams (v1802.2.10-build.96) itemfilters (v1802.2.8-build.47) jei (v9.7.1.255) tesseract (v1.0.32) mekanism (v10.2.5) caelus (v1.18.1-3.0.0.2) ae2extras (v2-1.18.2) bdlib (v1.19.3.7) clumps (v8.0.0+17) naturescompass (v1.18.2-1.9.7-forge) decorative_blocks (v2.1.2) ftbbackups2 (v1.0.18) beautifiedchatserver (v2.2) starlight (v1.0.2+forge.83663de) mcjtylib (v1.18-6.0.20) rftoolsbase (v1.18-3.0.12) xnet (v1.18-4.0.9) rftoolscontrol (v1.18-5.0.10) farsight_view (v1.18.2-1.9) enderstorage (v2.9.0.182) ftbchunks (v1802.3.17-build.265) forge (v40.1.92) inzhefopcore (v3.0.0) zerocore (v1.18.2-2.1.31) bigreactors (v1.18.2-2.0.61) minecraft (v1.18.2) cofh_core (v9.2.1) thermal (v9.2.0.46) thermal_foundation (v9.2.0.46) thermal_integration (v9.2.0.16) thermal_innovation (v9.2.0.17) thermal_expansion (v9.2.0.20) thermal_locomotion (v9.2.0.13) tconstruct (v3.6.4.113) smoothchunk (v1.18.2-1.9) antimobgriefing (v1.18.2-1.0.0) rftoolsutility (v1.18-4.0.23) theoneprobe (v1.18-5.1.2) ae2 (v11.7.3) ae2wtlib (v11.6.3) titanium (v3.5.9) ftbquests (v1802.3.14-build.191) immersiveengineering (v1.18.2-8.4.0-161) creativecore (v0.0NONE) archers_paradox (v3.2.0.12) kotlinforforge (v3.11.0) jeiintegration (v9.0.0.37) flywheel (v0.6.8.a) createdeco (v1.3.1-1.18.2) materialis (v1.18.2-2.8.0) mantle (v1.9.45) gravestone (v1.18.2-1.0.1) thermal_cultivation (v9.2.0.16) polymorph (v1.18.2-0.46) storagedrawers (v10.2.1) fluxnetworks (v7.0.8.12) appleskin (v2.4.1+mc1.18.2) lootr (v0.3.24.61) ferritecore (v4.2.2) puzzleslib (v3.3.6) appmek (v1.2.2) cosmeticarmorreworked (v1.18.2-v2a) createaddition (v1.18.2-20230426a)

Additional Context

We do keep a lot of chunks loaded on the server using FTBChunks but I don't believe this affects this behavior.

jaynotgay commented 9 months ago

Did you manage to fix this issue?

uberswe commented 9 months ago

Did you manage to fix this issue?

No, we moved to a new server and just have less trains

IThundxr commented 1 month ago

I'm not sure if much can be done about this apart from maybe saving off thread, i'll leave this open in case anyone has ideas on how this could be handled better