Discussion for next year changes

[erri120]

Do discuss mostly on GitHub so the messages don't get lost on Discord

Overview

Core changes:

• Abstract the hell out of both the Compiler and Installer.
• Step away from meta.ini files that tell WJ what to do with an archive
• make updating all Compilers and Installers easier

PRs that I will do next

• Configurable/Modular compilation/installation
• Manual Compiler/Installer

In-depth

Everything above is tied together and will require some tinkering so I don't have one PR doing everything. Anyway, lets talk about the Compilers: We already abstracted a lot of code and created ACompiler so we can have multiple Compilers. The problem is that most of the time when a new features gets merged, the VortexCompiler doesn't get updated. Most recent example was the introduction of UpdateTracker.NextStep and VFS.IntegrateFromFile. I want to go further. Let's adapt the concept of the "Compilation Stack" to the entire Compilation and Installation. This ties in with the configurable/modular PR I want to do:

this picture took way too long to make...

My vision is only use WJ and nothing else. Meaning that you slap all your Mods into MO2, Vortex, or whatever, run Wabbajack and configure everythin in Wabbajack. Making Wabbajack an Editor for Modlists essentially. We currently expect everything to be setup beforehand and than just run the Compiler. What if we don't run the entire thing but step by step like in the picture you can see the Status. The status of the entire modlist and the status for each step. Have Wabbajack first check the Downloads, then check the installs and so on.

Instead of having

// in _Begin()
UpdateTracker.NextStep("Step 1");
DoSomething();

UpdateTracker.NextStep("Step 2"):
AnotherOneBitesTheDust();
// ..

We use either use System.Collections.Generic.Stack<T> or a custom implementation of it to and just shove each step into the stack and run it down like the current Compilation Stack. We'd have to change the naming so that the current Compilation Stack will have a different name.

On the installation side this will enable us to create a modular experience. When we have an Editor for creating Modlist than we can easily create one for installing Modlists. Maybe even pack it with some basic Logic or Choices and be like If User Selects Option A Then Install A Else Install B. Endless possibilities.

[halgari]

Can I get a writeup of the proposed workflow? Would someone start with MO2, then edit the metadata in WJ? Or would you be installing mods via WJ?

[erri120]

I'd like to go away from being MO2 centered so let me phrase it this way: Someone used whatever tool to createa a mod setup. Be it MO2, Vortex, or whatever we have support for (as long as we can parse what has been installed). Then the potential modlist author goes into Wabbajack, points to their setup (MO2's profile ini or Vortex installation or whatever) and the new currently unexisting Editor screen will pop up. There the modlist author can change (like shown in the 1st picture) properties for different items in different sections. We might have every downloaded file in a Downloads section where the modlist author can adjust from where the file came from (without having to create inis). In the Install section you might have options for creating choices like either Option A or Option B. Than we can have Patching, Steamworkshop Items, or whatever steps we have.

[Noggog]

We already abstracted a lot of code and created ACompiler so we can have multiple Compilers. The problem is that most of the time when a new features gets merged, the VortexCompiler doesn't get updated.

Totally agree that as things progress, the common functionality should be recognized and siphoned "upwards" so that it gets inherited as naturally as possible by all.

I do like the concept of adding jobs to do to a Queue initially, and then pulling them off and doing the work afterwards. For me, this also opens up a nice avenue to more easily implement other tasks, too, like https://trello.com/c/pk1zvoDu/224-improve-progress-bar-stage-prediction. When we put jobs to do on the queue, they can perhaps also specify in some way how long or how many steps they intend to take, so we can scale the progress bar more intelligently. That's a side tangent, but just bring it up as this would be a nice side-benefit to the mentality of "going over" everything we're going to do before we actually do it.

TLDR: I do think adding something like Queue<(Func<Task> Job, string JobName, int StepCount)> into the ABatchProcessor framework would be a good step forward.

---

Most of my questions stem from the need to understand a use case and/or gritty specific of how the flexibility of opting out of a Compilation step entirely would be utilized. For example, what steps inside MO2Compiler._Begin() would be potentially omitted? I feel like all the steps in that function are needed for compilation to be successful. There's not much room for optionality there, but perhaps there's some concepts I'm not thinking of.

What it seems like you're more after is not making the -large- steps in a compilation optional, but rather improving different "ICompilationStep implementations to be optional and/or have more flexibility? Eg, we wouldn't just add the stack step of InstallMod, maybe. That step might have more mechanics where the compilation author could specify that this might be optional, and that would open up the ability to choose whether to install it on the other side. (Both backed by a UI, ofc)

[erri120]

I just want to go away from this big one time run function _Begin(). The Installer should mostly be one click, but for the Compiler I think an Editor would be better. Meaning that we don't expect the setup to be perfect for the Compiler and than just run it but that the Modlist author can change stuff in Wabbajack, save to some sorts of 'project file' and export the finished Modlist at the end. I do like the term ExecutionQueue/ExecutionStack @Noggog.

Most of my questions stem from the need to understand a use case and/or gritty specific of how the flexibility of opting out of a Compilation step entirely would be utilized. For example, what steps inside MO2Compiler._Begin() would be potentially omitted?

With Compilation step I assume you don't mean ICompilationStep but rather some Execution step: If we have some sort of Editor and expose some/most/all Execution Steps for the Modlist author to tweak (like pic 1 but with more sections) than we can dynamically add/remove execution steps. Currently for the MO2 Compiler you might not need zEdit patches or crazy BSA manipulations but, like I said at least 3 times already: I don't want WJ to be MO2 centered, in Vortex there is a lot of shit that can be left out if you don't need them. I also plan on doing Manual support. This right here is to create a foundation for writing extensions. You should take a look at how hacky I did the VortexCompiler/Installer. It works, but its hacky and I don't like it. Tbh if you @halgari say that we should 100% focus on only MO2 and Bethesda Games than fine, we don't need this.

Anyway: Editor instead of Compiler. Can we do this?

[Cyclonit]

This is a braindump containing my thoughts regarding customizability and Wabbajack's position within the modlist creation workflow. Given that the topic of customization is part of this issue, I think it fits here. If it doesn't I'll gladly open up a new issue instead.

Currently, modlists are created using a mod manager. Once the modlist has been created and installed through the mod manager, Wabbajack analyizes the current state of the game and computes the delta to the original state without a modlist. The result is a list of instructions that can be appleid to other original games, to transform them into the modded version.

This workflow profits off of the maturity of existing mod managers and puts Wabbajack into a comfortable position, where modlist creation and installation become very straight forward processes. However, it also means that Wabbajack is constrained by the modlist creation workflow itself. Since the entire modlist has to be created prior to Wabbajack compiling its list of instructions, there is no trivial way of making WJ modlists customizable.

My initial idea is to cut out the mod managers from the process of creating modlists entirely. For this purpose, Wabbajack would have to be expanded to have an editor itself and it would have to be able to create installation steps from definitions instead of the current state of a modded game.

Next I will outline what these definitions will have to be.

Mod Information

For a modlist there are two important things to a mod: Meta information and alterations that have to be performed on that mod to make it work properly.

Firstly, meta information consists of the source from where to download the mod from, hashes for consistency checking, version, authors and the mod description. Using this information, the installer can download the mod and verify that it has downloaded the correct files as well as presenting credits and the description to the player.

Secondly, many mods require additional alterations to work in a modlist. Thus all of these alterations must be specified, such that the installer may apply them to the mod. Most alterations can be fully automated based on specifications made by the author (e.g. excluding files from archives or merging mods together). Other alterations do require the modlist author to modify the mod using tools like SEEDIT manually, such that the compiler can figure out what the installer will have to do.

Some well known installation instructions are as follows. Many will already know these instructions from modlists like Lexy's LOTD.

• Exclude files from an archive
• Include files into an archive
• Extract BSAs from a mod
• Flag ESL
• Optimize Assets
• Search and Replace in .ini-files
• Merge mods
• Convert ESP to Form 44
• Patch Binary
• Install FOMOD

The Mod Graph

A modlist in MO2 or Vortex may be be represented by a simple flow graph without any branches. The mod manager starts by installing the first mod and adds subsequent mods overwriting contents of previous mods. For customizability the graph would have to be expanded to allow conditional branching.

Input

To facilitate customizability modlist maintainers must be able to define input parameters during modlist creation. When a player installs a modlist, they are first presented with a dialog to make their choices for all of the input parameters. There may be both radio boxes and checkboxes with each box being treated as a boolean variable, similarly to a FOMOD installer.

Branches

Branches within the graph represent the various paths that the installer may take based on the player's choices. Whenever the graph splits into multiple branches a switch instruction with conditions for each branch is placed at that intersection.

Conditions for branches must be mutualy exclusive, such that there is no ambiguity as to which path the installer will take.

If all conditions combined are not exhaustive, an additional else branch bypassing all of the other branches may be created. Otherwise, the installer must prevent the player from making choices that are not supported.

Nodes

Each node in the graph may either represent a mod including alterations to be performed on that mod or individual alteration themselves.

Mod nodes first of all contain all of the meta information, such that the installer can download the files. Additionally, if there are actions that must be performed on a mod regardless of customization, the actions may be added to the mod node directly.

If different alterations are required depending on the input parameters, they may be added to the graph independently. This allows for higher customizability without having to repeatedly add a mod in multiple branches. One case where this may come in handy, is with binary patches for making ESPs compatible with one another.

The mod graph as shown in the image above can easily be represented in human readable JSON. If any binary patches must be included, these can either be base 64 encoded right into the json or they may be added as blob files next to the modlist json.

[Noggog]

I think it would be good to scope and clarify the discussion, as I think there's a few threads muddled together in this so far. Here are the topics/goals I feel are being discussed so far: 1) General code reuse improvement in Compiler classes. How do we want to reuse common concepts (between Vortex/MO2, say), and how to omit concepts that are N/A for a certain compiler. I think this is a red herring discussion, as furthering that cause would require some refactors, but nothing absolutely fundamental like we're discussing here. 2) Abstracting our compilation concepts away from MO2/Vortex to be even more detached and unrelated to either source. This would just mean identifying the common concepts between MO2/Vortex that we want to identify as common between -any- source. A MO2/Vortex compiler would be constructed from these smaller building blocks.
3) Compilation/Installation decision trees/graph. Providing customization concepts during compilation, so as to offer customization concepts during installation.

1 and 2 are talking about modifying Begin() and its contents. Either refactoring some calls to the abstract base class, or migrating to some Queue<Action> system.

I believe 3 is more talking about improving the Compilation stack systems (ICompilationStep) to have more features in the steps. Eg.. IncludeMod A step might be only run if high resolution route is in effect, etc.

I feel like this discussion is veering towards 3).

Here's my thoughts on expanding WJ compilation/installation to be a decision graph/tree:

I think it would be awesome and I'd love to see it come to fruition. I think the proposed solutions have a lot of considerations and complexity that still need to be addressed.

Let's try to set up a very simplistic example use case.

User has two texture mods, one high rez, one low rez. They then want fix one of the textures they dont like by adjusting the saturation or something. That's the modlist; Very simple.

Currently

User cannot do both the low texture or high texture setups. They have to pick one to install, and do so via MO2. Then they open up the image editor and adjust the saturation on the setup they chose. Compile. WJ finds source download locations, and binary diff of the texture change. Done.

New Setup

The new system can theoretically now install low texture or high texture setup now. They also don't need to use MO2. They use a WJ "editor" now.

First task the user wants to accomplish is to offer low/high resolution alternatives. They say they're interested in the high rez texture pack, and the low rez texture pack through some WJ UI we develop? Cool, decent amount of work, but do-able. They add the "decision" for the user to either install the low res or the high res, and specify one texture pack to be installed based on the user's choice. Cool. Low/high res setup complete. WJ will offer the decision up to the user, and install one texture pack or the other.

Okay, now the user wants to do the saturation change. Where/how does the user do this? Where is the texture file to modify? The texture packs only exist as download locations that WJ is pointed to, and perhaps they're also downloaded somewhere and stored as zips, maybe. But their contents are not manifested to exist on the disk yet anywhere for the user to actually open up one of the textures to do the modification. Which install choice would be the one that "existed" and was made available for modification? Do they just open the zip up themselves, find the texture, unzip to their desktop, do the change, and then tell WJ to somehow absorb the difference as "Patch binary" step? There is nowhere all the files actually exist for the user to do any manual tweaking.

This isn't too bad when it's just a texture being changed. What if it's a custom manually created patch? How are all the alternative choices "manifested" into an actual installation that the user can open up in xEdit to do some patching? Which set of WJ installation choices are they picking when they finally get into xEdit to do some tweaking? Do they have to do this again for every possible combination of installation choices?

---

Part of the beauty of WJ is the fact that it doesn't care how the modlist got the way it is. It just replicates it.
Once WJ does have to care, and especially when exponential choices come into play, the problem of how the user actually goes about making tweaks becomes a really tough problem.

The graph you posted above works fantastically in a more simplistic modding setup like doing Witcher 3 or something. But as soon as tiny tweaks from the user or from 100s of different tools need to be applied at different stages, many of which expect a legit install to read in to process, the problem gets real rough. Not saying absolutely impossible, just that there's a lot more to discuss and hammer down

Other Considerations

• WJ would likely lose its "byte to byte" correctness "guarantee", as all the possible decision combinations might not be able to be hashed to ensure correctness.

[Cyclonit]

I'll continue focusing on point 3 for now.

Could we maybe use different terms for users who use WJ to compile lists and those who use it to install them? I suggest calling the former "modlist maintainers" or simply "maintainers" and the later "players". I'll use those terms hereafter.

You raise valid points. The entire workflow for creating a new modlist would look like I outline in the following paragraphs.

Starting a new modlist

The mainter would first have to define a download location where all of the mods are supposed to go. Next, they will be presented with a near empty graph editor, containing only the start node. From there they can add new nodes.

Mods

When the maintainer adds a new mod, they can download it straight through WJ. All of the downloaders are already in place and some are able to fetch all of the mod metadata at the same time. If they choose a mod from a source that doesn't provide the metadata, they will have to add it themself.

Next WJ will download the mod to its download location. From there on the maintainer can either add alterations to the mod node or add further nodes to the graph.

Alterations

By default, WJ will merely install the mod without any changes to it. However if the maintainer decides to define alterations WJ can process these prior to installing the mod. With most alterations, like removing files from an archive, WJ can do this fully automatically. All the maintainer has to do is to specify the path relative to the archive.

If alterations require external tools, WJ can set up a sandbox environment using a pseudo installation up to that alteration's point in the graph. Next it can lauch the external application and observe the changes to the mod's files. From there on, it computes the delta as usual and adds the delta to the modlist.

Branches

As soon as branches are available, the maintainer will have to specify which version of the modlist they are working on. This way WJ can always setup a suitable sandbox to monitor the maintainers manual changes.

Byte to Byte correctness

aany possible path through a modlist graph is deterministic and can thus be vetted just like WJ does today.

[halgari]

I'd like to avoid any situations were we start encoding common modding operations into wabbajack. There's such a variance involved that it's impossible to define them in code, instead in WJ we currently just perform a diff. No one really knows what "form 44" conversion means, which is why all mod guides leave that up to the CK. So it's not really possible to tell WJ "go convert form 44", when that involves performing operations inside the CK.

The same mostly applies to ESL conversion, I don't want to start digging into the specifics of parsing and modifying ESP files in WJ, because then we have to ensure our code is correct. Instead we currently provide a single method to diff files, and that works for any number of situations.

[Noggog]

I do think we'd need to have a "checkout" concept where WJ installs a modlist up until a certain node, yielding a "pseudo" installation of what it looks like up until that node. Then the user can do any manual alterations (xEdit CR, whatever), and then hit some sorta "lock in binary changes" button in WJ?

I do suppose that the added complexity of branches and whether the user needs to do CR for every branch combination is up to them and how complex they make their tree. If all they offer is some texture alternatives, then this featureset helps them a lot, without adding too much headache. It would only start to bite users that really push the choices concepts to the limit

[halgari]

To @Cyclonit 's proposal, I not sure I understand the point. I think I want to step back a bit and define the problems in play. @erri120 did a good job in describing the problem of:

"Problem: Vortex support in WJ is a bastard child, anything implemented for MO2 has to be re-implemented for Vortex"

So let's start there. Can we get one-sentence problem statements, in the form of: "Problem: ...". That will help us drive how possible solutions may or may not help us reach those goals.

[erri120]

Problem: Wabbajack updates but modlist Dev is absent Solution: Wabbajack CLI + custom Docker image + CI

[halgari]

Let's keep it as problem statements for now: each problem can have at least 2-3 solutions. If we discuss solutions to early on we end up pre-conditioning ourselves to a certain approach, that we may have to shoe-horn other problems into when they crop up.

The way to do software design on large changes like this is to:

1) list all the problems in play, and let this sit for a day or so so we're sure we caught most of them 2) create at least 3 solutions that cover all the problems to one degree or the other. 3) create a matrix of solutions and problems noting the pros and cons of each solution for each problem 4) normally at this point one solution will appear as the clear winner. 5) deep design/coding begins at this point

I normally wouldn't be so much of a stickler for process, but what we're discussing here is a complete rewrite of the guts of WJ, a program that mostly works today. We're talking about months of work and the introduction of dozens of bugs, perhaps with a certain amount of time where the old codebase is unusable or at least un-maintained.

[LivelyDismay]

I don't understand a ton of the stuff I just read, but I can give suggestions from my perspective as to what I would see as a "Maintainer." I'll even use your format but with slightly more sentences (and without Erri's fancy "solution" part because I am not very intelligent).

1. Problem: Modlists break a lot from updated mods. It would be great so have some sort of system in place to tell us what exact mod broke, maybe even with a link from the meta file so we can quickly go look into it. It's usually a matter of reinstall/replace, but we still have to check the changelog for the mod in question and possibly fix our patches to accommodate those changes, so if we did have some sort of automated update, we'd probably need to flag which mods are ok to auto update and which ones we need to look at first.

2. Problem: Modularity scares the hell out of me. I don't want users to be choosing options in WJ (I imagine textures would be fine, but I DO NOT want users to be able to pick and choose mods). I have enough trouble maintaining the list(s) I'm already responsible for, I don't want to maintain multiple versions of the same list for picky users. We already discourage modlist changes, have a rule for the bot to say so, and actively encourage other potential modlist maintainers to "Make the game you would want to play." Open it up to textures, and it won't be long before we get "Can I get Lexy's to just not install Frostfall?" and then suddenly Mdc is either maintaining 20 different butchered versions of Lexy's or we're all beating people away with a barrage of questions to do so.

3. Problem: Google drive. We're tired of saying "UI isn't updated, get it from gdrive." I know you've made steps to remedy this, and I imagine Cyclonit's servers will eliminate this entirely, but still worth mentioning.

4. Problem: WJ is constantly getting slideshow cache errors. Nexus server issue or WJ issue?

5. Problem: Whitelisted links. It seems pretty random whether WJ will actually grab LOD files from gdrive. Again, I expect this will go away with the new hosting, but still mentioning it for posterity.

6. Problem: VNV is still using the Beta 4 build. Get in touch with Zeefa about that one.

7. Problem: Updating modlists (installing WJ on top of an existing installation) tends to leave empty mod folders at the bottom of the mod pane in MO2, which confuses users. These folders are old versions/removed mods and are not necessary.

8. Problem (sorta): Everyone thinks merged mods are Optional mods they can "activate" and add to the modlist. They're in Optional solely so MO2 doesn't see them. Is there a better way to hide these or idiot-proof them so we can stop dealing with it?

I hope that was sort of what you were looking for from me. If not, then....well, sorry for wasting your collective time.

[ghost]

Modularity scares the hell out of me. I don't want users to be choosing options in WJ (I imagine textures would be fine, but I DO NOT want users to be able to pick and choose mods). I have enough trouble maintaining the list(s) I'm already responsible for, I don't want to maintain multiple versions of the same list for picky users.

I'm in agreement here. It would be akin to supporting hundreds of possible lists instead of one or two. With NOISE being the way it is, I support people already adding their own mods, when it's been stated I can't handhold every small problem users may have when adding onto it.

[erri120]

Response to @LivelyDismay: 1: see #328 2: You can still choose what should be optional or not. Not like you will need to have at least 1 choice in the Modlist at the end. If you only want to use it for Low/High res textures than fine. If you feel daring and want to try doing UNB or CBBE choice also fine. If you are a masochist and want to do some crazy choice between weather or gameplay mods than fine by me. It's the Modlist authors decision on how much they want to torture themself, we just create the opportunity for it 3: With "UI isn't updated, get it from gdrive.", do you mean the current process of hopping on GitHub, changing the modlists.json file and waiting for someone to access the PR? 4: see #331 Apparently some users find it scary so I "lowered" the threat of it. Tbh its just a message telling you we couldn't get an image

[LivelyDismay]

1. I saw that last night. Awesome. Thank you.
2. I understand that, but simply being in the support channels is already often a masochistic hellhole. People won't ask politely, they'll say, "Well, Wabbajack can do it, so go do it for me." It isn't the possibility of implementation that worries me, it's the demanding people that will expect us to treat them all like they're special, and that we're obligated to do what they want.
3. Yes, though this is largely a non-issue the last few days since modlist devs have access to merge our own changes. We used to often have to wait for Timbo, you, or Halgari to catch it, so we'd have to forward people to gdrive in the process. Not at all a slight against anyone here, just pointing out that it has been a mild annoyance in the past. People join the discord to say WJ failed, and we have to reply, "Yeah, the github isn't merged yet, grab the manual download link."
4. I know it doesn't hurt anything. I tell people to ignore it all the time. I just didn't know if it was a Wabbajack problem or a Nexus problem. I'm guessing Nexus, but I don't know anything about how this stuff works.

[Cyclonit]

Problem: Modlists are direct copies of their creator's install and thus geared towards the performance of their own PC. Users with significantly better or worse PCs can either not take advantage of their power or may not be able to play the modlist.

@LivelyDismay To be honest, if someone demands that you do something from them, just tell them no. Those who demand special treatment will have to learn that its up to you to decide if you want to make your list customizable or not.

[erri120]

Getting a Wabbajack CLI up and running, see #352. We can expand that very easily and maybe create a CI for updating modlists when a new WJ version comes around.

Ofc we have problems where a mod is unavailable but if its just about updating to a newer WJ version than this can make updating very easy.

[Noggog]

Do we want to close this out? Or migrate it potentially to an area that wont clog up our issues page? Seems like a nebulous item that will never really have a "reason" to be closed