Strange issue when pregenerating

[embeddedt]

Hi, I'm trying out GT6 in a private 1.7.10 modpack and running into some issues with pregeneration. Full disclosure, I am using OptiFine and it might be what causes this. I am also using Speiger's chunk pregenerator mod as that is what was recommended.

My issue is that during pregeneration, millions of NextTickListEntry are being created and stored in the world's pending tick sets. However, these never seem to be processed, even if I cancel the pregeneration task. The result is that my heap is completely filled after only a few minutes of pregenerating and the game is unplayable till I restart.

I do have a heap dump and know how to use VisualVM to diagnose this to some degree, so I did some investigating and it seems that all of the tick entries are corresponding to GregTech blocks. Here is a screenshot showing one of the entries and the chain of objects containing it:

Here is the mod list. Some of them were disabled to try and help with testing, they have .disabled in the file name.

 1.7.10                                                       Controlling-1.7.10-1.0.0.jar                       logisticspipes-0.9.3.132.jar                             Railcraft_1.7.10-9.12.2.1.jar
'[1.7.10] SecurityCraft v1.8.13.jar'                          CoreTweaks-1.7.10-0.2.2.jar                        LookingGlass-1.7.10-0.2.0.01.jar                         RandomThings-2.2.4.jar
 ActuallyAdditions-1.7.10-r21.jar                             CosmeticArmorReworked-1.7.10-v6b.jar               LumySkinPatch-1.0.10.jar                                'RedstoneArsenal-[1.7.10]1.1.2-92.jar'
 ae2stuff-rv3-0.5.1.11-GTNH-mc1.7.10.jar                      craftingtweaks-mc1.7.10-1.0.88.jar                 Mantle-1.7.10-0.3.2b.jar                                 rftools-4.23.jar
 aether-1.7.10-v1.1.2.2.jar                                   CraftTweaker-1.7.10-3.1.0-legacy.jar               mcheli                                                   sdnf-1.0.jar
 alreadydecoratingfix-1.0a.jar                                DefenseTech-1.7.10-1.0.1.46.jar                    McHeliPrivacyShield-1.7.10-1.0.1-universal.jar           secretroomsmod-1.7.10-4.7.1.413.jar
 animfix-mc1.7.10-1.4.0.jar                                   Draconic-Evolution-1.7.10-1.1.0-gt.jar             mcjtylib-1.8.1.jar                                       SimplyJetpacks-MC1.7.10-1.5.3.jar
 AntiIdConflict-1.3.5-1.7.10.jar                              DummyCore1.13.jar.disabled                         Mekanism-1.7.10-9.1.2-kmecpp.3.jar                       SkinPort-1.7.10-v10d.jar
 appliedenergistics2-1.7.10-rv3-beta-82-GTNH-8-g08bc7a5.jar   endercore-1.7.10-0.2.6-1-g1aea890.jar              MicdoodleCore-1.7-3.0.12.504.jar.disabled                SolarFlux-1.7.10-0.8b.jar
 archaicfix-1.0.jar                                          'EnderIO-1.7.10-2.3.1.32-8-g6a00ab5 (1).jar'       'MineFactoryReloaded-[1.7.10]2.8.1-174.jar'              'soundreloader-1.1-1.7.10 (1).jar'
 ArchitectureCraft-1.7.2-mc1.7.10.jar                         EnderStorage-1.7.10-1.4.9.31-universal.jar         mixingasm-1.7.10-0.2.1.jar                               SpongeMixins-1.5.0.jar
 Aroma1997Core-1.7.10-1.0.2.16.jar                            EnderTech-1.7.10-0.3.2.405.jar                     ModTweaker2-0.9.6.jar                                    squidless-1.7.10-0.6.1.jar
 Aroma1997s-Dimensional-World-1.7.10-1.1.0.1.jar             'Enhanced Biomes 2.5 for MC 1.7.10.jar'             ModularPowersuits-1.7.10-0.11.1.117.jar                  ssmlegacy-mc1.7.10-1.1.0.dirty.jar
 AromaBackup-1.7.10-0.1.0.0.jar                               Et_Futurum_Requiem-2.4.0-SNAPSHOT-93d2fda.jar      More-Planets-1.7.10-1.4.4-GC502.jar.disabled             StevesAddons-1.7.10-0.10.23.jar
 AsieLib-1.7.10-0.4.9.jar                                    "Ewy's Workshop-1.1.6.jar"                          MouseTweaks-2.4.4-mc1.7.10.jar                           StevesCarts-1.7.10-2.0.1.jar
 Atum-1.7.10-0.6.77.jar                                       ExtraCells-1.7.10-2.5.10-5-ge2469e1.jar           'MrCrayfishFurnitureModv3.4.7(1.7.10).jar'                StevesFactoryManager-1.7.10-1.1.0-GTNH.jar
 Automagy-1.7.10-0.28.2.jar.disabled                          ExtraTiC-1.7.10-1.4.6.jar                          NaturesCompass-1.7.10-1.3.1.jar                          StorageDrawers-1.7.10-1.11.13-GTNH-1-g2bf1fa2.jar
 Avaritia-1.7.10-1.25-2-g1a10110.jar                          falsepatternlib-mc1.7.10-0.5.0.jar                 neiaddons-1.12.15-DEV-mc1.7.10.jar                       Tainted-Magic-1.7.10-8.1.1-1-g4424096.jar.disabled
 Avaritiaddons-1.7.10-1.5.3-GTNH-1-g9e31f04.jar               fastcraft-1.25.jar                                 NEIIntegration-1.7.10-1.1.4.jar                          tcinventoryscan-mc1.7.10-1.0.11.jar.disabled
 bdlib-1.9.4.109-mc1.7.10.jar                                 FloodLights-1.7.10-1.2.8-137.jar                   netherlicious-3.2.1.jar                                  tcneiadditions-1.7.10-1.1.0.6-2-g7b19714.jar.disabled
 BetterAchievements-1.7.10-0.1.0.jar                          FoamFix-1.7.10-universal-1.0.0-beta3.jar           netherportalfix-mc1.7.10-1.1.1.jar                       tcnodetracker-1.7.10-1.1.5.jar.disabled
 BetterBuildersWands-1.7.10-0.9.2-GTNH-1-g57e364d.jar         Forestry-1.7.10-4.4.6-2-g14f61b5.jar               NoMobSpawningOnTrees-1.2.0-mc1.7.10.jar                  TConstruct-1.7.10-1.9.0.11-GTNH-1-gc824664.jar
 BetterFps-1.0.1.jar                                          FpsReducer-mc1.7.10-1.10.3.jar                     NotEnoughEnergistics-1.7.10-1.3.11-1-g86ce155.jar        Thaumcraft-1.7.10-4.2.3.5.jar.disabled
 betterp2p-1.7.10-1.0.7-3-g00855f0.jar                        FTBLib-1.7.10-1.0.18.3.jar                         NotEnoughItems-1.7.10-2.2.10-GTNH-3-g7d71173.jar         Thaumcraft4Tweaks-1.7.10-1.4.16.jar.disabled
'BiblioCraft[v1.11.7][MC1.7.10].jar'                          FTBUtilities-1.7.10-1.0.18.3.jar                   Numina-0.4.1.0-kmecpp-3.jar                              ThaumcraftMobAspects-1.7.10-1.0.0-GTNH.jar.disabled
 biggerpacketsplz-upto1.7.10-1.2.jar                          gadomancy-1.7.10-1.0.8.4-1-gd3d2ca3.jar.disabled   OpenBlocks-1.7.10-1.6.8-GTNH-1-g9209e4b.jar              thaumcraftneiplugin-1.7.10-1.7a.jar.disabled
 BigReactors-0.4.3A.jar                                       GalacticraftCore-1.7-3.0.12.504.jar.disabled       OpenComputers-MC1.7.10-1.7.5.16-GTNH.163-universal.jar   thaumicenergistics-1.7.10-1.3.16-GTNH.jar.disabled
 BrandonsCore-1.0.0.12.jar                                    Galacticraft-Planets-1.7-3.0.12.504.jar.disabled   OpenGlasses-1.1-GTNH.1.jar                               Thaumic-Exploration-1.7.10-1.1.85-GTNH.jar.disabled
 bugtorch-1.7.10-1.1.6.jar                                    gendustry-1.6.5.1-GTNH-mc1.7.10.jar                OpenModsLib-1.7.10-0.10.1.jar                           'Thaumic Machina-1.7.10-0.2.1.jar.disabled'
 buildcraft-7.1.24.jar                                        gregtech_1.7.10-6.15.01.jar                        OpenModularTurrets-1.7.10-2.2.11-245.jar                 ThaumicTinkerer-1.7.10-2.6.4.jar.disabled
 buildcraft-compat-7.1.7.jar                                  Grimoire-mc1.7.10-3.2.7.jar                        OpenPeripheral-1.7.10-AIO-8.jar                         'ThermalDynamics-[1.7.10]1.2.1-172.jar'
 Careful+Cast+Corrector-2.0.jar                               hodgepodge-1.7.10-1.7.2-7-g737a639.jar             OptiFine_1.7.10_HD_U_E7.jar                             'ThermalFoundation-[1.7.10]1.2.6-118.jar'
 carpentersblocks                                             ic2                                                OreExcavation-1.1.134.jar                                TiC-Tooltips-1.7.10-1.2.6.jar
"Carpenter's Blocks v3.3.8.2 - MC 1.7.10.jar"                 IC2NuclearControl-2.4.3a.jar                       p455w0rdslib-1.7.10-1.0.4.jar                            TMechworks-1.7.10-0.2.16.2.jar
 CaveControl-0.2.jar                                          IguanaTinkerTweaks-1.7.10-2.1.6.jar                Peripherals++-1.3.6.jar                                  triangulator-mc1.7.10-1.4.0.jar
 CCTweaks-1.7.10-1.1.1.jar                                    IGW-Mod-1.7.10-1.1.12-34-universal.jar             PlayerAPI-1.7.10-1.4.jar                                 TwilightForest-1.7.10-2.3.8.14.jar
 Chicken+Chunk+Patcher+1.1.jar                                ImmersiveEngineering-0.7.7.jar                     PPP                                                      Waila-1.5.10_1.7.10.jar
 chisel-1.7.10-2.10.9-GTNH-2-g3c07295.jar                     industrialcraft-2-2.2.827-experimental.jar         ProjectBlue-1.1.6-mc1.7.10.jar                           WailaHarvestability-mc1.7.10-1.1.6.jar
 ChiselTones-1.7.10-1.0-3.jar                                'INpureCore-[1.7.10]1.0.0B9-62.jar'                 ProjectRed-1.7.10-4.7.0pre14.18-Base.jar                 WAILAPlugins-1.7.10-0.2.5.jar
'Chunk Pregenerator V1.7.10-2.5.2.jar'                        InventoryTweaks-1.59-SI-release-2.jar              ProjectRed-1.7.10-4.7.0pre14.18-Compat.jar               WanionLib-1.7.10-1.8.2.jar
 CodeChickenCore-1.7.10-1.1.3.jar                             IronBackpacks-1.7.10-1.2.20.jar                    ProjectRed-1.7.10-4.7.0pre14.18-Fabrication.jar          WarpTheory-1.7.10-1.2.6-GTNH.jar.disabled
'CoFHCore-[1.7.10]3.1.4-329.jar'                              ironchest-1.7.10-6.0.62.742-universal.jar          ProjectRed-1.7.10-4.7.0pre14.18-Integration.jar          Wawla-1.7.10-1.1.1-GTNH.jar
'CoFHLib-[1.7.10]1.2.1-185.jar'                               Jabba-1.2.9-1.7.10.jar                             ProjectRed-1.7.10-4.7.0pre14.18-Lighting.jar             WirelessCraftingTerminal-1.7.10-1.8.8.4-2-gba49449.jar
 CollectiveFramework-1.0.0-132.jar                            l4j_jndi_fix-oldforge-1.0.0.jar                    ProjectRed-1.7.10-4.7.0pre14.18-Mechanical.jar           WorldBorder-Forge-1.0.2.jar
 compactmachines-1.7.10-1.21.jar                              LanHostSkinReFixed-1.7.10-rv0.jar                  ProjectRed-1.7.10-4.7.0pre14.18-World.jar                WR-CBE-1.7.10-1.4.3.jar
 ComputerCraft1.75.jar                                        letsencryptcraft-1.7.10-1.2.0.jar                  QuarryPlus-1.7.10-2.1.0.jar                              zettaindustries-1.1-120.jar
 Computronics-1.7.10-1.6.7-GTNH.jar                           llibrary-1.5.1-1.7.10.jar                          railcraft                                                Ztones-1.7.10-2.2.1.jar

I know this likely isn't a bug in GregTech itself, but if you have any suggestions of where to begin looking/changing settings, I'd appreciate it!

[GregoriusT]

So the Issue is you are running out of RAM and the swap activates which tries to write RAM to the Drive, lagging you to death. Is that correct?

@Speiger did tell me there was a Feature to skip scheduled block updates or something along those lines, not sure what it was, but apparently that Feature fails for you. I know that GT6 is kinda spamming those updates in Oceans especially, so that might be the source of the RAM usage.

Edit: Wait what, why does this happen with PrefixBlock, I didn't look closely at the screenshot until just now. I need to check what the hell happens there for it to be in something called "NextTickList" because i never saw an actual Class File named like that in MC.

[embeddedt]

I have my Java heap set to 4GB. The heap eventually fills up with NextTickListEntry objects and that causes the standard continuous garbage collection as Java fails to free them. I did try increasing the heap to 6/8GB but this just increased the amount of time till the heap filled up and didn't solve the underlying problem.

Swap isn't an issue in my case; my system has quite a bit more RAM than what I'm giving Minecraft.

I know that GT6 is kinda spamming those updates in Oceans especially, so that might be the source of the RAM usage.

A previous heap dump I had taken showed that a lot of these objects were pointing to FluidOcean blocks so it's good to hear confirmation on that.

It's also worth noting that I'm pregenerating in a singleplayer world.

or it to be in something called "NextTickList" because i never saw an actual Class File named like that in MC.

The objects are being added to pendingTickListEntriesHashSet and pendingTickListEntriesTreeSet inside WorldServer and are created in WorldServer.scheduleBlockUpdateWithPriority.

did tell me there was a Feature to skip scheduled block updates or something along those lines, not sure what it was, but apparently that Feature fails for you.

Yes, it seems that the block updates are being scheduled, and then never processed. To solve this there either needs to be some task that processes block updates once the set gets large enough, or scheduling should be completely disabled during pregeneration.

[GregoriusT]

Thing is PrefixBlocks do not tick, so why the hell do they get added to the pending tick list whatsoever, they very much should NOT be! (im still looking into that right now)

[embeddedt]

In the meantime, I'll try again without OptiFine just to be absolutely sure it isn't the cause of the issue.

[GregoriusT]

Okay for the Gravity Block Logic (Sand/Gravel) there is a scheduled Block Update in PrefixBlocks, I forgot about that one. But it shouldn't accumulate tons of those...

[embeddedt]

It takes a lot longer to happen without OptiFine installed, I was just about to write a message saying that OptiFine was the problem, but it does still get there. NextTickListEntry objects still take up at least 600MB in the heap dump. I waited a bit longer, cancelled the pregen task, took another heap dump, and the objects now took up about a gigabyte.

[GregoriusT]

Yeah I am in the process of rewriting PrefixBlock so that most Ore Blocks should at least not fill up your Heap.

[GregoriusT]

Okay this Version should have less Issues with the Ore Blocks at the very least.

https://gregtech.mechaenetia.com/secretdownloads/

[embeddedt]

I tried opening my original testing world and the heap filled almost instantly when I began pregenerating. That might just be from all the scheduled ticks being saved (I believe they are persisted at least in newer versions). I'll now try a fresh world and see what happens.

[embeddedt]

Okay, here are some numbers. I made a fresh world and about 3GB of the heap was in use. I took a heap dump to force a full GC and that brought it down to 2GB. So before pregeneration starts, about 2GB of the heap is occupied as a baseline. This seems reasonable.

Unfortunately, the issue still occurs with the snapshot you provided. I sampled a few of the 22 million NextTickListEntry objects and here are the block classes I saw:

• net.minecraft.block.BlockSand
• net.minecraft.block.BlockGravel
• gregtech.blocks.fluids.BlockOcean

I did not see any more PrefixBlock entries now, so your fix for that was likely not a waste of time. However, the issue seems to be deeper than just GregTech since even vanilla blocks are being included in this list.

I should probably try running the pregenerator without GregTech installed and see what happens. It might be a bad interaction between it and one of my other mods. I thought I had used it before but maybe it was in my 1.12 pack instead.

[embeddedt]

The issue is still there without GregTech, but isn't pronounced enough to cause issues. There are still a couple million NextTickListEntry objects but they're only taking up 200MB of heap total.

I guess the last thing to do is to try the pregenerator on a vanilla world and see if I get the same behavior.

[embeddedt]

Some more testing shows that the issue doesn't occur in a world with only GregTech and the pregenerator, so it must be one of my other mods. I may end up just having to write a patch that silently drops block updates once the queue gets too full because I don't like the idea of going through 200 mods to find the problem...

Anyways, thank you for the ore fix and this can probably be closed since it isn't really your mod's issue.

[GregoriusT]

Wait did you try only dropping Optifine?

[embeddedt]

Yes, and the issue still occured, so it wasn't an Optifine issue for once. :laughing:

[GregoriusT]

I am still actually curious as to which Mod causes it then, if its not GT6 or Chunk Pregenerator or Optifine.

[embeddedt]

I wrote a small piece of code that iterates the block update queue each tick and prints a warning when it's too large. This is what I'm seeing:

[13:16:54] [Server thread/WARN] [org.embeddedt.archaicfix.ArchaicFix]: There are 12475 pending block updates in dimension 0
[13:16:54] [Server thread/WARN] [org.embeddedt.archaicfix.ArchaicFix]: These are too many block updates. A breakdown will now follow.
[13:16:54] [Server thread/WARN] [org.embeddedt.archaicfix.ArchaicFix]: gregapi.block.prefixblock.PrefixBlock_ - 4
[13:16:54] [Server thread/WARN] [org.embeddedt.archaicfix.ArchaicFix]: net.minecraft.block.BlockGravel - 103
[13:16:54] [Server thread/WARN] [org.embeddedt.archaicfix.ArchaicFix]: gregtech.blocks.fluids.BlockOcean - 12368
[13:16:54] [Server thread/WARN] [org.embeddedt.archaicfix.ArchaicFix]: Now forcing the queue to be cleared in dimension 0
[13:16:54] [Server thread/WARN] [org.embeddedt.archaicfix.ArchaicFix]: After update GC, there are are 12368 pending block updates

The pattern I noted in the logs is that the server is completely unable to process any BlockOcean block updates even if world.tickUpdates(true) is manually called. This is probably the reason for the leak.

Here's the code for reference:

    @SubscribeEvent
    public void onServerTick(TickEvent.ServerTickEvent event) {
        if(event.phase == TickEvent.Phase.START) {
            for(WorldServer world : MinecraftServer.getServer().worldServers) {
                Set<NextTickListEntry> tickEntries = ReflectionHelper.getPrivateValue(WorldServer.class, world, "field_73065_O", "pendingTickListEntriesTreeSet");
                if(tickEntries.size() > 0) {
                    if(tickEntries.size() > 3000) {
                        LOGGER.warn("There are " + tickEntries.size() + " pending block updates in dimension " + world.provider.dimensionId);
                        LOGGER.warn("These are too many block updates. A breakdown will now follow.");
                        IdentityHashMap<Class<? extends Block>, Integer> numBlocks = new IdentityHashMap<>();
                        for(NextTickListEntry entry : tickEntries) {
                            Class<? extends Block> clz = entry.func_151351_a().getClass();
                            numBlocks.put(clz, numBlocks.getOrDefault(clz, 0) + 1);
                        }
                        numBlocks.keySet().forEach(key -> {
                            LOGGER.warn(key.getName() + " - " + numBlocks.get(key));
                        });
                        LOGGER.warn("Now forcing the queue to be cleared in dimension " + world.provider.dimensionId);
                        world.tickUpdates(true);
                        LOGGER.warn("After update GC, there are are " + tickEntries.size() + " pending block updates");
                    }
                }
            }
        }
    }

I'm now trying to see if forcefully loading the chunks that contain the block updates will hide the issue, to see if it's an issue with block updates being created for unloaded chunks.

[embeddedt]

Forcefully retrieving the chunk using world.getChunkFromBlockCoords(entry.xCoord, entry.zCoord) does not help. The updates are still not processed, or the blocks keep readding themselves to the queue somehow.

[GregoriusT]

I would highly suggest to bisect the Mods Folder to see which of the Mods causes the leak, that way it should be easier to find the culprit. ;)

[GregoriusT]

Oh and maybe try to ONLY process BlockFalling aka Sand and Gravel and see if they get cleared out at least. But do that after bisecting the Mod List. ^^

[embeddedt]

All block updates seem to be cleared without issues except BlockOcean. I could try bisecting but it's going to take a very long time, relaunching is quite slow.

[GregoriusT]

Bisecting means you remove half the Mods to see if that fixes the Issue, that way you end up knowing which bunch of Mods causes it and then you half that bunch too, etc ;)

[Speiger]

@embeddedt chunkpregen for example does random tick cleanup too. Though before it creates new chunks instead of after.

    public void cleanUp()
    {
        if(tickUpdates.size() > 0)
        {
            int count = 1 + tickUpdates.size() / 1000;
            while(count > 0)
            {
                try
                {
                    world.tickUpdates(true);
                }
                catch(Exception e)
                {
                    FMLLog.getLogger().info("Prevented Server Crash of Random Tick Cleanup");
                    e.printStackTrace();
                }
                count--;
            }
        }
    }

So this code is called 1 tick delayed. And so far it had been working fine.

[Speiger]

Small info about random ticks. Blocks can create indefinite loops with random ticks. Fire for example will always keep 1 NextTickEntry object in the world. You can not get rid of it. A while(true) loop would just kill your server because it had been 1 minute since the last tick. Also until 1.10.2 (Could be 1 version later) Minecraft only processes up to 1k NextTickEntries while in later versions its up to 65k. So yeah greg is doing his ocean water stuff in the wrong version since minecraft will do stuff very slowly....

[GregoriusT]

I know my Ocean Blocks use that Tick Update similar to how Fire does it, though the Update itself does die down NORMALLY, which somehow after cleanup is not the case with this particular Modpack.

[GregoriusT]

Okay super dumb Question, you ARE using the same Seed in all of your tests RIGHT?

[embeddedt]

I actually haven't been using the same seed (I will from now on), but I've used the exact same set of steps to create the world and the behavior is consistent each time: only the ocean blocks (and sometimes rivers) keep increasing in the update queue. Everything else clears out like it should.

[embeddedt]

Part of the issue with debugging is that there are always thousands of block updates queued, even before I run the pregenerator. I assume this is because of the looping @Speiger mentioned. I've removed a bunch of mods and am waiting to see if the issue reoccurs.

[Speiger]

@embeddedt no the "Looping" i do is only running during pregeneration. That code is only running through a active pregen task. The mod is designed to turn everything off that isn't needed. If something is needed it will turn it on.

[embeddedt]

Maybe I wasn't clear, I was referring to the looping caused by a block scheduling an update to itself inside updateTick.

[Speiger]

ahhh ok

[embeddedt]

After some time spent fiddling with mod combinations, I have made a couple changes that have hidden the issue for now. I have two performance mods installed, CoreTweaks and FoamFix, and changed two settings in their configs that I thought may have been related.

The first change was to disable the getPendingBlockUpdates optimization in coretweaks.cfg. The second change was to disable the ghost chunkloading fixes in FoamFix by setting enableFixes to false in the ghostbuster section.

After doing this I don't see the heap usage spiking anymore. There are still many, many block updates but the game seems to perform more normally.

This is not really a conclusive fix but I don't have a lot more time to spend on the issue, so this will do the trick for now. If I continue to have issues I'll likely just increase my heap size or not use the pregenerator at all.

Thank you @GregoriusT and @Speiger for taking the time to help despite this not being a bug in either of your mods. I appreciate your help!

[GregoriusT]

Oh nice, can you attach the two Config Files to a Reply so I can look through them and maybe add them to my Default Config Pack? ^^

[embeddedt]

I'm still not 100% convinced that this is a real fix, but here you go.

configs.zip

[GregoriusT]

Yeah I know it might not be a real fix, but every little bit helps ;)

[GregoriusT]

Uh both of these are inside the root config directory right? not in some subdirectory like many Mods like to do?

[embeddedt]

Yes, both go directly in the config folder.

[GregoriusT]

I doubt any of the ghostbuster fixes touch the Lists themselves, pretty sure they mess with the Code of those Blocks directly, at least the way I read that out of the Config implies that.

[embeddedt]

After playing around some more with this, I have found that the block updates do settle down over time as Greg mentioned. However, @Speiger, I think the pregenerator can be improved somewhat to avoid this situation.

Here is what is happening currently:

1. Some chunks are generated that contain ocean blocks, etc.
2. In some cases, up to 50,000 block updates are immediately scheduled at these locations.
3. The pregenerator immediately tries to clean up these updates, however, since they are fluids, they flow once and requeue themselves for further updates later.
4. The pregenerator moves on and starts generating more chunks, and the cycle repeats from step 2.

Nothing is technically leaking. If you have an infinitely sized heap and stop the pregenerator, the block updates will eventually run. The problem is that the way GT6 worldgen works, block updates are queued faster than the game can deal with them, causing the queue to grow too much.

I think a simple fix here is for the pregenerator to stop generating chunks while there are more than about 50,000 block updates queued. This will prevent the runaway queuing I described and shouldn't break anything.

I would happily submit a PR but it doesn't appear that Chunk Pregenerator is open-source. :confused:

[Speiger]

@embeddedt the issue here is that gregtech schueleded tick spam is kinda excessive and 50k blockupdates honestly isn't even enough. With vanilla water we reach that all the time. Also when a chunk unloads it also includes all the NextTickEntries. Meaning that unloaded chunks do not contribute to that factor. Also the biggest question is: How long should I wait to cleanup updates? Because I can't make it a while true loop or a while(x amount) because at that point i am dealing with a lot more waiting time then usual.

I think one thing @GregoriusT can do is just increase the amount of NextTickEntry objects being processed per tick to 1.10.2 or newer amounts that way minecraft on its own can deal with a lot more of these. Since that number is 65k instead of 1k in newer versions. Or maybe for his water implement a system that can deal with more elements that works on a per chunk basis.

Also yes ChunkPregen is closed source for a lot of reasons.

[embeddedt]

Also the biggest question is: How long should I wait to cleanup updates? Because I can't make it a while true loop or a while(x amount) because at that point i am dealing with a lot more waiting time then usual.

I think you have no choice but to wait until the number reaches a reasonable level. The current system allows the heap to be exhausted fairly easily since chunks are generated faster than they are unloaded (at least from what I can tell), and I would rather have my pregeneration take 3 times longer than have my game freeze, need to be restarted, and need a very large heap. Unless you have a way of forcing the chunks to unload?

Even stopping at 1 million queued updates would be better, the number seemed to reach 20 million in my case.

[Speiger]

CP marks chunk unloaded the moment they are done generating. The only chunks it doesn't mark unloaded are the required 3 chunks infront of it. X+1 Z, X, Z+1, X+1, Z+1

That is one thing I could fix since it would cause 64 loaded chunks to stay over per region file, which averages out to 48-32 the closer to it is to finishing the pregeneration. So the patch i could do is once a region file is done it would just mark all chunks unloaded that aren't protected by players/spawn. (Note that minecraft unloads all loaded chunks that are not protected by players ever 900 ticks)

But outside of that there is nothing I can do without doing a while true loop which will result in your game crashing the moment you have to many infinite NextTickEntry elements in the queue because then its a while(true) loop.

[embeddedt]

There is some sort of logic you use to pause pregenerating when the MSPT gets too high. Can you not add a second check there to pause if the number of NextTickEntry elements exceeds a certain threshold?

[Speiger]

Depending on the implementation this either could take a lot of time or be not as effective as the current implementation. Remember what I am doing is a lot more then Vanilla Minecraft does. The Bug you reported with the "MemoryLeak" could be achieved in Vanilla + GT just by flying around really fast over an ocean. Since 900 ticks are 45 seconds, and only 1k NextTickEntries are processed per tick. If it spawns 50k Entries thats 50 ticks of updates until it caches up with X amount of chunks. And since I assume this is a "layer" and not all guess how long that will take to fully complete.

[GregoriusT]

Just letting Pregenerator do infinite update processing wont help, especially for things that requeue themselves, for which Fire would end up making a deadlock. And GT6 Ocean Blocks do have SOME conditions in which they will requeue themselves too (when chunks around them aren't loaded), and not just the Block below them (to fix LIGHT issues because Minecraft sucks...)

Yeah this whole Issue is not as easy to solve as one might think... I am just glad that disabling that one Mod that has the changes to the scheduled block update queue actually removes the leakage for the most part.

[embeddedt]

Infinite update processing won't be effective in this situation, yes. But without some sort of hard cap on the block update queue, the pregenerator will inevitably starve the game of memory. I don't think a speed comparison is fair, since the faster pregenerator just fills up a heap and leads to having to restart the game.

I am just glad that disabling that one Mod that has the changes to the scheduled block update queue actually removes the leakage for the most part.

I actually haven't tried running the pregenerator for a long time again since figuring all this out. The last world I tried it on was so full of block updates that it was unusable afterwards, I'd prefer that not to happen to another world. Generation is fast enough currently that I'm just trying to avoid pregenerating.

[Speiger]

You are missing the point that CP is not the cause of the issue, it is just a amplification especially on higher speed settings. You do know that you have control over speed settings too right?

Infinite update processing won't be effective in this situation, yes. But without some sort of hard cap on the block update queue, the pregenerator will inevitably starve the game of memory. I don't think a speed comparison is fair, since the faster pregenerator just fills up a heap and leads to having to restart the game.

About that. Why do you think I iterate over the list once per tick? To make sure nothing needed is gone. Also I kinda have to let chunks loaded for a certain amount of time otherwise they do not load correctly. On top of that at such a scale you can be pretty sure that nextTickEntries are not cheap on performance so i can only run it so many times before I crash servers due to lag. Or slow down the pregeneration so much that it is not worth doing a pregeneration at all.

I would suggest you try player teleportation pregeneration and look how well that performs in comparason.

[embeddedt]

You are missing the point that CP is not the cause of the issue, it is just a amplification especially on higher speed settings.

I understand, as I mentioned earlier I don't think there is an actual leak or bug. I think it's just a fundamental limitation of the way block updates are handled.

You do know that you have control over speed settings too right?

I didn't know that, thanks. I'll see if I can tune that if I do decide to pregenerate again at some point. That should solve my issue since I can just slow it down enough to let the block updates be processed.