Program slows to crawl when working with merges + getmodbyname

[Morribyte]

Hi there again! ESLify in general is working muuuuuch better now and im so grateful to the effort you and Noggog have put in to fix the issues. Thank you. You guys are seriously awesome for this. It's a total gamechanger

I'm not actually sure if this is an issue, but I have a fairly big merge that has something like 200k records in it.

I left it running for it to cache for over an hour and it a was still working on voice files. I also ran it several times with different sized merges to see if it was a specific mod that was a problem. It would just sit on some voice files for as long as 5-6 minutes and in some cases just get stuck on a specific voice file (usually .xwm I think it was). Is it slower because it's reading from .json?

If it is because caching it from reading json files is necessarily slower than using Mutagen's reading functions, I wonder if it's possible to have an option to disable caching the files from merges? Is it always necessary to save and rewrite the facegen/voice files?

I personally use MergeMapper so that relinking is unnecessary. I believe that means that files wouldn't need to be cached. Scripts could be a different story since scripts inside a merge could have formID calls to a mod outside of a merge.

And before I forget, I don't want to bombard you with issues so I'll just tack it on here: GetModByName. It's generally not used with newer scripts now that people understand the issues with it, but for older mods like SkyUI, FNIS, RaceMenu and some SexLab mods are ones ive noticed. Usually they are only there to check if mods are enabled so not a big deal but if they are in the fe space that will never pass the check. Is that something people using ESLify Everything has to keep in mind (ie edit and recompile scripts / check all the scripts for calls) or can it be changed to deal with them in some way such as through a warning or some logging that tells you which mods have those calls and should not be ESL'd?

[Michael-wigontherun]

The Cache gets read from only once and outputted once. Its doesn't read the json each time it checks something. The Caching should work fine. Does it ever go into a not responding state?

[Michael-wigontherun]

Mutegen cant read what ESLify needs to change records. ESLify Everything uses the forms output by zMerge and xEdit. Mutegen cant see what was changed after the plugin is saved that's why hard written data needs to exist from the thing changing the plugin.

[Michael-wigontherun]

MergeMapper needs to be used inside extended into what ever is getting those files its isn't a do all fix. It was split off of SPID so that it could be used by more things. DLLs may become aware of remapped ids if a SKSE DLL has been written to use MergeMapper. See Developer Usage section below for how to convert..

[Michael-wigontherun]

GetModByName I have zero clue what that is. Its GetFormFromFile that is the issue. And you are correct most have stopped but there are files outside of that spectrum of new mods that still do and then there are people who just code what ever they want. You issue isnt with Papyrus though its with the Voice ESLify. If you send me a stopped working devlog I'll look over what I need. to change to make it faster.

[Michael-wigontherun]

On another note why are you merging that much stuff that is just waste of stuff. Make a few small merges that can sit in light plugin space.

[Michael-wigontherun]

Also also send the merged cache file in your compactedforms folder.

[Morribyte]

The Cache gets read from only once and outputted once. Its doesn't read the json each time it checks something. The Caching should work fine. Does it ever go into a not responding state?

I honestly can't tell. I think so? It just sits there with a blinking cursor and nothing I press except for CTRL+C to exit the program does anything.

GetModByName I have zero clue what that is. Its GetFormFromFile that is the issue. And you are correct most have stopped but there are files outside of that spectrum of new mods that still do and then there are people who just code what ever they want. You issue isnt with Papyrus though its with the Voice ESLify. If you send me a stopped working devlog I'll look over what I need. to change to make it faster.

GetModByname is a function where you can put in a mod name such as SkyUI.esp and SKSE will look over the load order (not in the FE space) and check to see if there's a mod by that name that exists. You can check it against a certain spot in the load order, but most of the time it's used to just check < 255. The result is that any script that's ESLify'd won't be considered a part of the load order, and thus any functions with GetModByName will cease to function. That script function is just a hold over from older scripts usually, but often scripts that call SkyUI / RaceMenu / FNIS tend to use them to check if those mods are inside the load order.

On another note why are you merging that much stuff that is just waste of stuff. Make a few small merges that can sit in light plugin space.

Right now it was some follower mods from before ESLify Everything was a thing to save on load order slots and I think a few of them were a lot bigger than I expected (AnnaNPCs is the main culprit here I think). Now that it seems like most of the errors that I was having are gone, I think I am going to be dismantling them all and figuring out what I can ESLify and seeing what my load order looks like then and merge smaller mods like you said.

I will re-run the merge through ESLify Everything until it freezes and send you the dev log / merged cache as well. Just give me a few minutes

[Michael-wigontherun]

Oooooooo ok I can add getmodbyname didn't know it was a thing ok.

[Morribyte]

I'm running ESLify on a merge I just threw together (apparently I already had deleted the follower merge I had before) with a bunch of the popular follower mods and other bigger follower mods and for whatever reason the voice ESLifier seems to be working OK. No idea what's different today vs yesterday. I will need to see if it gets replicated as I work and once that happens I'll give you the log. It's actually kind of very annoying that it seems to want to cooperate today lol.

[Morribyte]

Oooooooo ok I can add getmodbyname didn't know it was a thing ok.

I'm not actually sure you can fix it. Since it won't check the FE space for mods by that name at all (that's why they always say to not ESL FNIS / SkyUI / RaceMenu), I'm not really sure how it's going to be fixable.

[Michael-wigontherun]

I can only find one thing that says that and its never referenced or commented on other then that, I just added a check to change the name in that log line. If it has nothing to do with it why mention it.

[Morribyte]

I can only find one thing that says that and its never referenced or commented on other then that, I just added a check to change the name in that log line. If it has nothing to do with it why mention it.

Eh? Yeah, like I said at the start of the post I wasn't really sure these things were an issue I just wanted to bring it to your attention just in case it's something you need to fix down the road. I know that GetModByName is a fairly uncommon call, but they are there sometimes especially in older mods (which, why are you using older mods anyway right? lol) but again I just wanted to pass along what I've learned. I didn't want to make 2 issues especially for a thing that's really probably not much of an issue in practice outside of niche mods.

[Morribyte]

It's been running for 3+ hours now. Still going over voice files. It's slowed down to a crawl from moving really fast at first, so I decided to just send you the log and stuff now. I couldn't send the files in a pastebin or over Github because they were far too large -- so I just decided to send them via a .zip file.

I have a feeling it's just because the merge is so large. Like I said before, I am disassembling the merges and will be making smaller ones since I can ESL far more mods, so this might not even be an issue at all. I did notice that the mods that had bsas seemed to process the voice files far faster, if that means anything.

Log and Merge Cache.zip

[Michael-wigontherun]

Ok so this kind of merge that isn't gonna even run well in game. This is just a test I get that but this is the kind of thing that makes modders hate there users for. This is the kind of thing that is just moronic to think is a good idea. But people actually make these kinds of merges and then blames the Modders for it not working well or at all. These tools are not intelligent, there tools running a set of repeatable instructions there not equip well enough to know this is a bad idea. I can add a few extra checks to skip over CompactedModData if folders are not found but beyond that I am not going to try to support something like this.

[Morribyte]

Ok so this kind of merge that isn't gonna even run well in game. This is just a test I get that but this is the kind of thing that makes modders hate there users for. This is the kind of thing that is just moronic to think is a good idea. But people actually make these kinds of merges and then blames the Modders for it not working well or at all. These tools are not intelligent, there tools running a set of repeatable instructions there not equip well enough to know this is a bad idea. I can add a few extra checks to skip over CompactedModData if folders are not found but beyond that I am not going to try to support something like this.

Ya, I sort of figured tbh. I didn't think it would run that well either lol. this particular merge I just threw a bunch of mods together to sort of show you what happens in cases when the speed slows down -- the original merges were split into several smaller merges to still save a few slots, but still had massive slow downs. I'm guessing even those merges were probably too big based on what you're saying.

Thanks again for the help!

[Michael-wigontherun]

I wouldn't merge any of the mods you add in that merge cache. All of them are massive mods adding quests lines, Voiced NPCs and Cells. Just chose a few to play with at a time.

I like my massive load orders too but sometimes enough is truly enough, I don't even know how you don't run out of string space with all that and SexLab mods as well.

[Morribyte]

I wouldn't merge any of the mods you add in that merge cache. All of them are massive mods adding quests lines, Voiced NPCs and Cells. Just chose a few to play with at a time.

I like my massive load orders too but sometimes enough is truly enough, I don't even know how you don't run out of string space with all that and SexLab mods as well.

Ya, I agree with you.

lol and Honestly I'm just trying to see how many mods I can stuff together without having the game break. I don't want to run with everything in an actual playthrough tbh, it seems like it'll be way too much.