LRN
commented
3 years ago May i suggest MO to replace git in this regard?
That is, define a special kind of attribute for a file in a mod. Let's call it "MMHL", standing for Merge Mod Hash List. MMHL is a list of file hashes. If MMHL is defined for a file, then this file is a merge between multiple similarly-named files that hash to these values.
For example:
mod1
foo/bar/baz.ini hashes to 0x123
mod2
foo/bar/baz.ini hashes to 0x234
mod3
foo/bar/baz.ini hashes to 0x345
mod4
foo/bar/baz.ini with MMHL=0x123,0x234,0x345; hashes to 0xabc
mod5
foo/bar/baz.ini hashes to 0xbcd
mod6
foo/bar/baz.ini with MMHL=0xabc,0xbcdmods 1-3 change the same file. mod4 is a merge of these files from mods 1-3. mod5 also changes this file. mod6 is a merge between the first merged file and the file from mod5.
The order in which previous mods are installed doesn't matter (and doesn't need to match the order of hashes in MMHL), since a file with MMHL is a full merge. As long as it is applied after the mods that it is supposed to be a merge of, everything's fine. MO can verify that this is the case by calculating hashes of the files that mod4 overwrites, and checking that files with these hashes come from the mods installed before mod4. If that is not the case, MO would show a "merge order conflict" flag.
If everything's fine, MO would show "merge component" flag for mods 1-5 and "merge target" flag for mods 4 and 6 (mod4 is both a merge of mods 1-3 and itself a merge component for mod6).
This works on a per-file basis. Files can be anything (text or binary). A mod might have a mix of merge-files and normal files. The overwrites/overwritten icons (lightning with "-" and/or "+" sign(s)) are not added if the only overwritten/overwriting files are merge-files or merge-components.
Optionally, it might be possible to make MO automatically rearrange mods in such a way that mods with merge-files are no top of any mods with merge-components. This operation might not necessarily be successful, since a mod might contain multiple different merge-files the correct order for which might not be possible to achieve without splitting the mod. I'm not sure whether an automatic mod-splitter for such cases is feasible or even desirable.
Doing partial merges (where a merge is defined for a sub-section of a file) seems impractical, as it would require MO to parse the file and extract, change and then join its parts.
To verify that merging is done correctly, base files must be present (i.e. if base game files are packed into an archive, then either MO has to have a capability to at least read files from that archive, or the base game files have to be extracted from that archive beforehand and presented to MO as a collection of loose files). If some of the files from MMHL are missing, MO can show a "merge with incomplete base files" flag for that mod (this is for information only - a merged file does not require other, previous versions, obviously; but their absence might indicate that some changes in this merged file come from uninstalled/disabled mods). If there are previous versions of a file not included in MMHL, then MO would show a "merge overwrites" flag, which means that this merge doesn't take into account some of the changes (probably coming from new, recently-added mods). If a merge-file is overwritten by a later file that is not a merge-file, that situation can be treated as a normal overwrite (though it would probably be nice to show a flag indicating that a merge-file is being overwritten - merge files SHOULD be at the top of any conflict, or at least be merge-components for other merge files that ARE at the top).
Creation of the merged files themselves is out of the scope of this proposal, though i would strongly recommend MO to learn to at least do simple 1+N-to-1 merges for text files (i.e. 1 base file and N different modded files merged into 1 merge-file that contains all changes; if N modded files change different, non-conflicting parts of the base file, this is trivial to do).
Creation of mods with merged files should be straightforward in MO. Any mod can have its files marked as merges (by adding MMHL to them) by activating an appropriate function (probably an item in the mod context menu). MO will look at each file in the mod, then look through all mods earlier in the install order, find conflicts, and add hashes from conflicted files to the MMHL for this file. MO cannot (and should not) verify that any file in this mod that is now being marked with MMHL actually contains changes from the earlier files (simple 1+N-to-1 merges can be verified; complex, non-trivial merges cannot, so there's no point in even trying). MO should probably warn about updating already-existing MMHL attrbutes (this is a valid operation when you update a merge-file with a new set of changes, but MO should ensure that this isn't done by mistake).
MMHL attribute must not be part of a merge-file (that is, it should not affect the hash value of a merge-file), since that file itself might need to be a merge-component for other, later-stage merge-files. How MMHL would be stored is an open question and the answer to it lies outside of my expertise, but i would guess that it can be in the same XML file where fomod info is stored.
Problems
Mod mods
Sometimes mods are mods for another mod. This is rather common for localized versions of a specific mod, though there are other instances of this e.g. 4K Notice Board Retexture for The Notice Board or the scenarios in #518. Managing these as separate files can lead to problems where the original mod gets loaded after mod mods if you forget to move one or just be annoying due to the clutter. It seems a common workflow to avoid these problems is to just merge the mods, but then only the original mod is preserved if it ever updates, and you have to reinstall / re-merge all of the mod mods.
Compatibility Patches
Sometimes mods will require compatibility patches to work with other mods. This results in the same problem as mod mods & can often lead to merging as well (necessitating a reinstall of patches whenever the original mod is updated).
Child Mods
In order to prevent the need for merging these mods, we could introduce the ability to manually make mods a child of another. A group of parent/children mods are treated as an inseparable group, so no matter where you move the parent mod in the load order, the child mods will retain their relative load order (though you can reorganize the load order among the children inside the parent mod). Parent mods are collapsible, so you don't need to have the children clutter the mod list.
Child mods would be purely an organizational tool that is just enforced via UI. As far as the underlying MO code is concerned, these are just separate, normal mods that the user just always happens to move as a group. This means there are no changes to the virtual file system / other functions of MO (aside from maybe mod list display features like filters), so everything else works out of the box (e.g. updating each child mod / preserving their mod ids / viewing them on the nexus).
We could also show mod backups as child mods so they don't clutter the mod list
Child mods are NOT a tool for organizing a bunch of separate mods (e.g. put all Armors under a child mod). This should still be done with categories / (collapsible) separators / etc