Simple Mod Framework v3

[atampy25]

Development of version 3.0.0 of SMF is now in progress. It might finish by the end of the month, the end of the year or months from now. Perhaps it will never finish, but I'll do my best to avoid that.

As a new major version, it brings with it breaking changes to hopefully improve the SMF experience. Here's a summary of the changes made so far.

Breaking Changes

Update checking overhaul

Update URLs have been removed, in favour of a new top-level url key in the manifest. All mods must now specify a URL that links to their mod page. This link will be shown in the GUI and also used for mod update checking.

If the URL links to a page on Nexus Mods (https://nexusmods.com/hitman3/mods/453), users will be notified when the Nexus version changes, and will have to manually redownload the mod to receive updates. This means that while Nexus update checking is now possible, it's still better to use one of the alternative URLs which provide fully automatic updating.

If the URL links to a GitHub repository (https://github.com/Notexe/Portable-Chair), users will be notified of new releases (assuming the release is tagged with a version like 1.0.0 - no v at the start) and the framework will be able to automatically update the mod.

If the URL links to a ModWorkshop page (https://modworkshop.net/mod/45444), users will be notified of new releases and the framework will be able to automatically update the mod.

{
    "id": "Notex.PortableChair",
-   "updateCheck": "https://github.com/Notexe/Portable-Chair/releases/latest/download/updates.json"
+   "url": "https://github.com/Notexe/Portable-Chair"
}

Framework data moved to data key

Properties like contentFolders and localisation have now been moved into the top-level data key (this applies to options as well, so the option now has a data key instead of directly having contentFolders).

{
    "id": "Notex.PortableChair",
-   "contentFolders": ["content"],
+   "data": {
+       "contentFolders": ["content"]
+   }
}

Options overhaul

Mod options have been completely overhauled. The easiest way by far to understand the new options system will be through the use of a schema and an editor like VS Code, but here are the basics.

Options now require IDs. The ID can be anything (not including braces or colons), and there's no set convention currently (though there may be one on release).

Most options now have the ability to be referenced within content files. This is done through a simple find-and-replace. "bla": "#{option:TheOptionID}" will be transformed into "bla": true for a boolean option, or "bla": 1 for a number, etc. If a find-and-replace is inside of a string, like "bla": "something #{option:TheOptionID} something", it will be replaced inline, like "bla": "something 1 something".

Option groups now exist, which are sections of options that can be displayed based on a condition (such as another option being on or off, or another mod being enabled).

Option tooltips have been replaced with descriptions, which may or may not be displayed differently, depending on how the GUI is ultimately designed.

Removal of thumbs

The thumbs manifest key no longer exists. It was previously used to disable dynamic resources to make REPO mods work online. Instead, it has been replaced with a new boolean disableDynamicResources key which the framework will manage, and which users can choose to ignore.

Removal of dependencies and introduction of portResources

The dependencies key has been removed. Necessary resources are now automatically made available to whatever files they are referenced in, without having to manually specify dependencies to port. You can still port resources yourself if you have a different reason to do so, using the portResources key:

{
-   "dependencies": [
-       "00123456789ABCDE",
-       { "runtimeID": "00123456789ABCDE", "toChunk": 0, "portFromChunk1": true }
-   ],
+   "portResources": [
+       "00123456789ABCDE",
+       { "resource": "00123456789ABCDE", "forPartition": "chunk0" }
+   ]
}

Only resources inaccessible by the given partition will be ported, an improvement over v2's unnecessary porting of some non-chunk0/1 dependencies.

Mod ID version range syntax change

The requirements, incompatibilities, loadBefore and loadAfter keys accept specific version ranges for the referenced mods. The syntax for defining these has changed.

{
-   "requirements": [
-       ["SomeoneElse.TheirMod", "^1.3.1"]
-   ],
+   "requirements": ["SomeoneElse.TheirMod@^1.3.1"]
}

Packagedefinition changes

The packagedefinition key has been renamed to packageDefinition and support for custom partitions has been removed, meaning the type key in its entries has been removed as well (since it would always be entity).

blobsFolders renamed to blobFolders

Self-explanatory.

Condition changes and script overhaul

Scripts and conditions have both been changed to use the Rhai language. This means they are now fully sandboxed and therefore safe for users; the framework will no longer give a warning when scripts are used in a mod.

Conditions should now be formatted as Rhai expressions (config.loadOrder.contains("Mod") rather than "Mod" in config.loadOrder).

The script interface has changed significantly.

Old

export async function analysis()
export async function preDeploy()
export async function postDeploy()
export const cachePolicy

New

fn data(config: Config, data: ManifestData) -> ManifestData;
fn operations(config: Config, data: ManifestData) -> Vec<(String, Operation)>;

The first function, data, receives the config and the manifest data (after options have been merged into it), and has the opportunity to modify the manifest data. For example, it can add a content folder based on a complex condition. The second function, operations, receives the same information but has the ability to return a series of (String, Operation) tuples. Both must be defined.

The mention of Operation may be confusing. It's due to the revamp of framework internals, which will be talked about later.

Removal of delta special file type

The XYZ.delta special file type has been removed due to it being largely useless and highly brittle to any change in the file. It was not used in any mod on the Nexus. There is no replacement.

Removal of all QuickEntity files prior to version 3.1

Support for QuickEntity files (entity.json and entity.patch.json) which use a version prior to 3.1 has been removed.  

Mod archive format change

Mod archives (e.g. ZIP) no longer include a folder inside them which contains the manifest. Instead, the manifest.json file exists at the root of the archive.

Improvements

So, you've heard all the reasons why the new update will be a headache for you (don't worry, this should actually all be automatically update-able by SMF on release, so you won't actually have a headache updating). What's the benefit? The framework has been completely redesigned.

New programming language

If you're fond of programming, you've probably heard of Rust. Simply switching to idiomatic Rust should significantly improve both the speed of deploys and the quality and clarity of errors. It also reduces the file size of Deploy.exe by at least half.

Graph model

The framework will no longer simply deploy mods one by one. All mods are now deployed, all at once. You might ask whether this impacts the load order functionality - load order is still respected because the framework now represents each deploy as a dependency graph.

Essentially, instead of deploying files, the framework now applies operations like ApplyQNPatch or OverwriteMaterial in parallel across a graph which represents deploy order as dependencies (a later mod in the order depends on an earlier mod's patches finishing in order to perform its patches).

This also allows for more granular caching of data (such as specific manifest keys rather than the whole manifest being re-deployed), which should further improve deployment speed. The framework will also now be able to store the whole cache in a single compressed file, instead of the multi-gigabyte folder it previously had.

This is also where those two functions from the script interface come in. An Operation is in effect a node in the dependency graph, and the String its ID. A mod which wants to apply a dynamically-built patch, for example, can return an ApplyQNPatch operation. The ID is used for caching, which is another thing the framework could not previously do for scripts.

Safety

In addition to scripts now being safe, other improvements have been made. Internal framework data structures can no longer be constructed at all with invalid data, meaning that a manifest with an invalid path will now immediately throw an error with a clear message explaining why.

Synthesis

The speed of deploys should be significantly improved in this version, and the framework should now use far less disk I/O.

Scripts should now be a usable tool for more advanced mod developers.

Errors are now far, far clearer.

A note on the GUI

The Mod Manager will also receive a redesign as part of this update. The details of this have not yet been worked out, but much of it should stay the same. The mod option interface will obviously be revamped, and the underlying technology will likely be switched to Tauri.

So what?

This post has been made to ask for your feedback. If you have any issues with the ideas above, any ideas of your own, any new features you'd like to see added or any issues you have with current SMF, feel free to share below!

Also, polls will be running on some changes in the Glacier 2 Modding discord server, which you can join here. A poll is currently running regarding where manifest metadata should be stored.

Other notes

If you'd like to play around with the current iteration of the new manifest format, you can use this JSON schema with VS Code or in an online editor.

[martesi]

The present Mod Manager takes forever to load mods with large mount of files.

E.g., Any NPC Replaces Any Suit. Each mod has full index for every suit in game, the only difference between them would be the replace target. For better experience, can we expect at least one of below:

A dynamic replace behavior

Mods can require a form schema in the manifest, which would create a visual form in Mod Manager for this mod. In previous scenario, mod creator should supply a list for replace from, and convert the current indexes to list replace to. This design should be flexible for future expansion.

Shared index

If the previous one is hard to implement, maybe a shared index key can be specified so Mod Manager will reuse instead of recreate.

Non-blocking loading

It's understandable mods with lots of file take more time on I/O, but the UI blocking is really annoying. Maybe a background work queue can be introduced, all the file system operation can be avoided in UI? I mean the UI really only needs a small parts of the Mod information, which can be transformed from worker_threads(just example) to main then to UI.

And what's the latest as for Jul 2024?

[atampy25]

I have no idea what you're talking about with regards to a "shared index key" or what replace from and replace to are.

None of the code, or performance issues, from v2's Mod Manager are retained in v3. They stem from large amounts of IO done by the frontend, which is JS and which has to reimplement many parts of the framework and is therefore slow. The mod manager in v3, being a Tauri app, is integrated with the same logic used for deploy and can perform necessary tasks like validating mods more efficiently.

[martesi]

Replace from and replace to

For example, replace the signature suit to Abel De silva:

• replace from: the signature suit
• replace to: Abel De Silva

So we can have a form like:

<Form>
    {USER_CAN_ADD.map(() => (
        <Row>
            <Form.Item name={"from"}>
                <Select value={"The Signature Suit"} options={DEVELOPER_PROVIDED_0} />
            </Form.Item>
            <Form.Item name={"to"}>
                <Select value={"Abel De Silva"} options={DEVELOPER_PROVIDED_1} />
            </Form.Item>
        </Row>
    ))}
</Form>

This will leads to a config passed to mods like:

const config = {
    form: {
        // the key data is config by developer
        // value of from/to can be different from their labels
        data: [{ from: "The Signature Suit", to: "Abel De Silva" }]
    }
}

Mods can use this config and return the actual deploy payload using operations() mentioned in your changes.

Shared index key

Never mind. It assumes the Mod Manager processed every mod with their individual content/blob on start up. I just roughly looked the code, it doesn't do this on start up, and Deploy process seems only take the mod option as its target to patch. They looks fine.

IO done by frontend

This is what I'm talking about. I don't think fs should be used directly from the frontend though it's easier to operate without more verbose ipc communication. But it's sync and in the main thread, which might be the cause to the frontend freeze.

It's also wired since I can only see the frontend reads mod manifest on start up, yet the speed is really frustrating.

Imagine a download manager, with multiple download tasks executing in the same time. You can see how far did it go, interrupt and restart with new config. For example, running multiple mod installation concurrently. This is what's I am suggesting regarding non-blocking.

[atampy25]

The first part you've described is just how the options system already works in v3.

I know what non blocking IO is. The frontend in v3, being a Tauri app, does not do any filesystem operations because the point of Tauri is to offload most model and controller concerns to Rust via asynchronous IPC. In regards to "running multiple mod installation concurrently", the v2 architecture cannot support it because load order must be respected, not because of synchronous IO, and the v3 architecture already does that as explained in the "Graph model" section.

[martesi]

I was concerning about the direct usage for window.__TAURI__.fs, which is just the replacement in Tauri. Good to know that it's not the case. I thought the graph model is describing deploy only. What I mentioned is actually the add mod operation, which is different and way easier to implement.

It's excited to know that v3 has so much improvement, can we get the first public test version within this year?