Immersive Petroleum slows down world loading in a significant manner

carbonatek commented 7 months ago

Minecraft Version

1.19.2

Immersive Engineering Version

9.2.4-170

Immersive Petroleum Version

4.2.0-28

Current Behaviour

Basically, when I restart my server, the server spends most of its loading time (upwards of 30 seconds) loading IP Reservoir regions. I don't know why is that, since the CPU in the server is not weak (Xeon E-2136) but for some reason I had about 6130 files in the ReservoirRegions folder and IP took about 42 seconds to initialize before letting the world load. Removing about 4000 redundant files in the directory led to the reduction to about 15-20 seconds. It's still a lot for about 13MB of data to load, I think, and it adds up when I have to do a restart multiple times a day due to me changing the configuration of other mods. I have found out about IP doing this by profiling the start process with Spark thanks to ModernFix. Server is installed on an HPE SATA 3 SSD, and the server has 10G of memory allocated, Java dist GraalVM 17.

Expected Behaviour

Server loads the .dat files in a more reasonable amount of time depending on the amount of files.

Reproduction Steps

Pregenerate 20000 radius with Chunky
Stop the server
Start it
Server hangs for upwards of 30 seconds on 'Preparing level "world"'

Debug Crash Log

https://bytebin.lucko.me/X5teIxr0e9 (excluding client-sided optimization mods)

Information for the reader.

Add a :thumbsup: to the bug report if you are also affected. This helps show the severity of this bug report and doesn't clutter the comments.
Add a comment if you have any insights or background information that isn't already part of the conversation.

TwistedGate commented 7 months ago

20k Radius?! Yeah, no wonder with a 40000² Chunk (or 640000² Block) area. That'd be 1.56 million region files.

mindforger commented 7 months ago

i think radius is given in blocks not chunks, but .....

is there a reason for the reservoir region files to be this "small" ? vanilla region files cover 32x32 chunks, this would make around 1500 region files for that area if i did the math correct, in your case this would be still around 97500 reservoir files (based on your numbers), which seems way too many to begin with, but not unreasonable to ever be reached, based on the project size of your server (or at least based on servers i have worked with)

carbonatek commented 7 months ago

20k Radius?! Yeah, no wonder with a 40000² Chunk (or 640000² Block) area. That'd be 1.56 million region files.

Don't worry, it's in blocks, not chunks. If it were in chunks, it would be probably still generating now, and I started the server 8 months ago.

Edit: As a matter of fact, even the 20k block radius took about 23 hours to finish, and that was on a Ryzen 7 5700X, which has much higher single-threaded performance, just thought I should add this info :P And you may ask, why did I pregenerate so much? Well, I play with about 3 or 4 other people on the server and we live separately, quite a long distance away from each other, and if I had not pregenerated those chunks, the server would've probably slowed down to a crawl when someone took off from spawn.

TwistedGate commented 7 months ago

i think radius is given in blocks not chunks, but .....

is there a reason for the reservoir region files to be this "small" ? vanilla region files cover 32x32 chunks, this would make around 1500 region files for that area if i did the math correct, in your case this would be still around 97500 reservoir files (based on your numbers), which seems way too many to begin with, but not unreasonable to ever be reached, based on the project size of your server (or at least based on servers i have worked with)

Now knowing its in blocks now it should be 6084~ files, not 97500. I am using the same trick vanilla uses to figure out if a block is in a certain region. Though in this case it uses the reservoir bounding-box centre instead. =P

And yes, there may be a lot of small files but it's better than one gigantic files which would have been the case with the previous system. (Also a nightmare to find specific reservoirs should something got wrong)

And i guess the loading tanks a bit because of it but i think its a fair trade-off knowing that with the old system the main array that holds all the reservoirs would at some point be so large that it would take quite a while for the reservoir handler to find and get certain reservoirs.

A cache was/is being used for both, but still.. the initial look-up would be noticable as server-lag.

And i can't make the loading faster, it's as fast as it (or minecraft) can go and/or is willing to go. (I can't even compress the data being stored either, its as compressed as it can be Lol)

carbonatek commented 7 months ago

Well the only thing that I really came up with was loading the reservoir data on-demand only for regions (or chunks, but the regions would be less demanding) that are loaded by players rather than load it all at once, but I also came up with the disadvantage that it would probably spike the CPU usage quite a lot every time someone would visit a region that was not previously loaded yet, and I haven't even thought about that data unloading from memory/cache. From what I understand, there is no better way than the current implementation, so it means I will just have to stick with it :P Not really a programmer, the most complex thing I've ever done was Mekanism reactor management in ComputerCraft, so don't expect any great ideas that would be possible within a reasonable span of time from me.

TwistedGate / ImmersivePetroleum