Crash when running Generate Max Height data for tamriel.

[robertgk2017]

Loading up just skyrim.esm and generating max height data for Tamriel worldspace, and hitting no to Not reuse existing data, generates for a time, but eventually it throws an assertion and crashes. Build 538 did not have this crash.

Here is the .zip that it generated.
CreationKit.exe_20240105_022004.zip

It successfully generates for every other worldspace without issue.

[robertgk2017]

Upon furth testing and re-attempts the assertion that gets thrown has the following text.

E:\Projects\Creation-Kit-Platform-Extended\Creation Kit Platform Extended Core\Patches\MemoryManagerPatch.cpp(367):

false

A memory allocation failed. This is due to memory leaks in the Creation Kit or not having enough free RAM. Requested chunk size: 1398288 bytes.

I just checked in task manager and its attempting to use over 200 GB of ram. without CKPE installed running a Generate Max Height data task for the Tamriel worldspace only ends upp using about 5 GB of ram at the most.

[Perchik71]

multiplies the pages in the pools, it then I'll put a limiter. Edit: For high-speed memory.

[Perchik71]

vmm-x64.zip maybe fixed

[Perchik71]

ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) TEXTURES: malehead.nif : MaleHeadIMF is missing a facegen tint map TEXTURES: malehead.nif : MaleHeadIMF is missing a facegen detail map

The memory manager is disabled. And here there are already mistakes, how you assembled them without CKPE at all, I do not know.

[Perchik71]

ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) ASSERTION: aSize != 0 (e:_skyrimhd\code\gamesln\bscore\memorymanager.cpp line 708) TEXTURES: 1stPersonGauntletsKha_1.nif : KhaHands is missing a specular map TEXTURES: 1stPersonGauntletsKha_0.nif : KhaHands is missing a specular map

Just a bunch... Edit: I don't think he did anything to you at all. But he eats a lot of memory even with the memory manager disabled. I'll mark this as a bug, but it's not clear yet what caused it, however, with ASSERTION, your CK should have crashed.

[robertgk2017]

ive generated max height data for Tamriel with build 538 of CKPE and it didn't use more than about 6 GB of ram for the whole process. something is definitely messed up.

[Perchik71]

hmm.. CK allocates a lot of memory in 0 bytes. need to check the commits.

[robertgk2017]

With that updated vmm dll you sent here, it is still memory leaking, but it seems to be a bit slower now. i have it running an its sitting at 120 GB of ram usage.

[Perchik71]

I made a limiter, but I will not add it to the project, the problem is not in it. The problem is that CK allocates a lot of such memory in 0 bytes, somewhere in some commit I made something to do with alignment. Edit: As a result, it is not 0 bytes, but something more useful has become. Edit2: Test 538 there are suspicions about cell view It is constantly being updated...

[Perchik71]

Although this assembly eats decently. And this is me where he told me about 11400 cell. Already 30 GB

[robertgk2017]

oh my, yea something definitely odd there. It shouldn't be using that much memory for something like this. I'm not much of a programmer but in particular for the Generate Max Height data function, when its done processing a given cell(s) it should remove the data for them from memory.

[Perchik71]

Well on 538 build, I caught a crash, but it's not due to lack of memory. But the usual one. but in general, he did not eat more than 40 GB, but he freed up 13 GB, etc. I noticed that the Cell View window is frequently updated and redrawn. In general, I think I can reduce the amount of consumption, but I won't be able to make him eat as little as 6GB, that's for sure.

[robertgk2017]

going that low while probably ideal is not realistic. Whatever optimization that can be done i'll take. And i'll happily test and report back whatever you try.

The cell view rapidly updating is because that function has to load every single cell in the worldspace in sequence, several times. so 11400 total, times however many times it hits each one, is gonna be an update to the cell view.

You could probably make it not call any updates to the cell view at all, while this function is running? if that's possible? The cells aren't being completely loaded like if you load one up manually to make edits. So the view being spammed like that is completely unecessary

[Perchik71]

if that's possible? Yep. I need to think about it.

[Perchik71]

patch.zip Reduced the total consumption by 4 times, taking into account requests with size 0. However, I still catch a crash in some kind of Cell. Try it yourself.

[robertgk2017]

Will do. I'll report back what happens.

Edit: It's about half way through the worldspace, which was where it was crashing for me. Its using about 20 GB of physical ram and it has about 75GB stashed in the pagefile. So about 95 GB total now. Certainly an improvement, but still egregiously wasteful.

[robertgk2017]

Nope, it crashed again.
CreationKit.exe_20240105_022004.zip

This function should not be allocating any memory at all beyond a few MB to load the cell its generating at any given time. and the references its reading the collision from. after each cell is generated that data should be removed from memory, which by the looks of it the vanilla CK does.

[Perchik71]

Creation Kit Platform Extended Runtime: Initialize (Version: 0.1.0.594, OS: 10.0 Build 19045) Use what I sent you.

[robertgk2017]

That is what you sent me. in that patch.zip i copied both of those dlls to my game folder before running this test hmm let me run it again. Except it says 599 in the title bar not 594.

[Perchik71]

no, there is a 599 version in the archive. either this is the wrong dump

[robertgk2017]

might have been the wrong dump im re running it in the CK and will send you that one when it crashes.

[robertgk2017]

Alright here is a fresh one, CreationKit.exe_20240106_044218.zip

This one has 599 in it so it's from the right dll.

[robertgk2017]

okay ive done a bunch of repeat tests. for some reason it appears as though data its discarding, it is sending to the Pagefile for eternity. Assuming this is true, this should not happen, any such data that it is discarding should actually be deleted from memory, both physical and pagefile? If that makes sense?

EDIT: it's specifically crashing as soon as the pagefile hits 180gb. I'ma turn on pagefile on an extra drive and see what happens.

Edit 2: yup that did the trick. I installed another 1tb drive and set the whole thing to be used for the pagefile. It's now cruising past the halfway point in the worldspace, which is is where it was crashing. As of this Edit it's using about 350GB of total Memory. Which is pretty fascinating considering it's all completely unecessary I think.

[Perchik71]

A lot, even by the standards of Fallout 4, where it is necessary to calculate the previs. (240GB). Later, over a cup of tea, in my free time (many my time), I'll look at where they don't free up memory. i'll connect the sniffer, etc. it is discarding should actually be deleted from memory, both physical and pagefile? If that makes sense? yep. Windows usually puts the virtual address there if it is rarely used.