Automatic Load Order for Optimization.

[Felgrey]

When you have many mods installed in my case +200 its very important that certain mods load before others like for example:

Ai Behavior after Mods adding new Enemies

The Programm already gives you a load order when you subscribe to a lot of mods a the same time.

It would be a benefit if there was an option to automatically determine a Load order based on mod types and priority.

[Dragon-32]

an option to automatically determine a Load order based on mod types and priority

That's a monumentally huge undertaking right there. I'd say that enabling the AML to read and follow mod dependency information maintained by the modders is the only realistic way to go. As per #170 and #56

If you're thinking of LOOT then XCOM's got a fundamentally different data structure to TES games. IMO, it'd have to be more of a BOSS / mlox situation; and those require significant, active and dedicated maintenance (speaking from experience).

[Felgrey]

Ok, i see your point and you are correct. I guess this should be on a low priority because of the amount of time it would take and other issues being more urgent.

[RevZero]

Regarding load order in general: AML handles "load order" by changing the order in that the mods are listed in the corresponding ini file. However, this order might not even be relevant. Some people mentioned, that the subscription date may be what determines the actual load order (probably because in some cases it helps to un- and re-subcribe to a specific mod to fix issues?). I never investigated this and wouldn't even know a good way to test this, but it would be nice to know for sure. Just pointing out here, that AML might not even be capable of changing the load order anyway.

[Dragon-32]

I've read Workshop posts that the load order in AML doesn't actually change anything. The .ini change it makes is ignored or is irrelevant to how mods are actually loaded.

Stretching my memory (getting on for four years ago now) but I'm pretty sure when Solariz implemented the load ordering it was very much a first attempt / beta.

On load ordering there's also the issue that (from what I can work out) for people with the SDK non-Workshop mods load after Workshop mods, but for people like me without the SDK Workshop mods load last.

[RandDenner]

So I have tested how loadorder works using local files. It is 100% determined by the foldernames of the mods, using an alphanumeric type function. Controlled by the file/folder handling routines of the OS, not the game. If you have three mods with conflicts in the local "mods" folder, using subfolders 001, 002, and 003, then the one in folder 003 will win every time. Filenames of any mod files don't matter. The order of the mods in the XComModOptions.ini file doesn't matter. The dates of files or folders (creation, last modified, or accessed) doesn't matter. Any data or entry in the mod's files doesn't matter. The exact same files simply swapped between the folders without any other changes causes the one in the "last" folder, alphanumerically (which is B-tree order for Windows) to win every time.

It is believed, but not proven as yet, that the same holds true for files in the local Steam Workshop folder. Because the ID numbers are incremented for later posted mods, and the ID number is used to set the folder name, it would hold that newer versions with newer dates also have higher Steam ID numbers, and therefore win conflicts due to the higher folder name number.

However, it is as yet unknown exactly how the enumeration process handles foldernames. A 1:1 character by character comparison irrespective of different lengths, or normalizing the names to the longest length and treating shorter filenames as if they have null characters for the nonexistent preceding positions. And it may be different for pure numbers, letters, and mixtures of the two. Edit: it is not normalized as to length. The folders are compared character by character without regard to the full length of the folder name. The lexicographical winner is the one with the earliest highest characters regardless of length of the full folder names.

The only control the XComModOptions.ini seems to exert is that when all the mod folders are enumerated and returned by the OS, mod folders with .XComMod control files that do not match the names in the .ini file list are simply not processed for inclusion into the game setup. The order is determined by the enumeration of folders by the OS and the game doesn't seem to ever alter that order once set by the file/mod retrieval process and in fact uses that exact order to integrate the contents into the game setup.

The devs were lazy and instead of coding a positive control function to enumerate mods using the control file (the .ini), they just ask the OS to enumerate the folders and ignore all mod files except the ones matching entries in the control file. But the enumerated order remains as given by the OS. There are several chances to positively control the process, and they missed all of them. This wastes time and resources and introduces multiple opportunities for errors. I had thought that this may explain later issues with the official mod loader being slow as hell and unreliable, but it turns out that other issues are the factors. Still terrible code though, in both cases.

Edited for greater clarity and to negate a mistaken conjecture.

[robojumper]

@RandDenner

These observations appear mostly correct to me and match what I had written down about load order a few months ago at https://robojumper.github.io/too-real/load-order/ even down to details like the B-Tree order. Nice work. I found a way to control whether workshop mods or local mods load first in that post.

I would note two things:

I think this explains later issues with the official mod loader being slow as hell

This doesn't immediately follow. The old base launcher cached no info and retrieved workshop details on every launch before showing the UI, which made it take several minutes to start for me. I think the new Firaxis/2K/whatever launcher is better in that respect.

Because the ID numbers are incremented for updates, and the ID number is used to set the folder name, it would hold that newer versions with newer dates also have higher Steam ID numbers, and therefore win conflicts due to the higher folder name number.

The folder name is unaffected by updates, it's the same over a mod's lifecycle. However, later mods can have an ID that compares lexicographically "less than" to an older mod -- LW SMG pack has an UGC ID of 577409322 while a recent WotC mod has the UGC ID 2368109302, and '2' < '5' even though the later mod has a higher number.

[RandDenner]

I suspected that mod numbers may be permanent once set. I was being unclear and more or less attempting to say that if a revised mod was separately uploaded, it would have a new higher assigned number and would therefore win conflicts as a result. But I incorporated a statement about updating the numbers based upon something someone else said without investigating it properly myself. I will update my post to fix this error.

I see what you mean about the lexicographical issue. Although I do note that some file handling routines do not process character by character but instead process the name as a whole, so in the examples you supply, they could be processed in the numerically correct order since the folder that begins with a 5 would be treated as a 05... since its length is one less than the folder beginning in 23... However, without testing under all circumstances and OSes, I would not trust that occurring.

Anyway, this does present some potential problems in practice, but nothing serious as yet.

And since I have literally never used either of the official launchers, it's good to know something more concrete about their issues. However my comment on the inadequacy of the system still stands correct, I think: There were indeed several chances to positively control the process, and they missed all of them. This process does waste time and resources and introduces multiple opportunities for errors.

EDIT: I misunderstood something robojumper was commenting on and edited to fix my mistake.

[robojumper]

@RandDenner Just to make it clear, there's no filling in numbers. 2368109302 is a higher number than 577409322 because it's more recent, but the lexicographic comparison loads it first because the most significant digit is lower.

[RandDenner]

Yeah, I realized that after my post and updated my post as a result. Good catch.

[Dragon-32]

@robojumper

I found a way to control whether workshop mods or local mods load first in that post.

I will admit to a little "are you kidding me?!" moment when I read that.

[RandDenner]

Yeah. Actually, I had the exact same thought when I found the .ini entry for the mods folder and I had planned to do an experiment myself to see what the result was. Robojumper got there first, though. Way first. Lol.

From Robojumper's post: "It is obvious that base-game config is loaded first. DLCs happen to be loaded after that, and then it’s mod directories. The game knows of the mod directories through a config array ModRootDirs in XComEngine.ini, and it turns out that the order there determines whether workshop mods or local mods are loaded first!"