EXML patching, a new way to share mods?

[emoose]

Apologies for the length, I write too much sometimes...

spAnser suggested this to me which reminded me of some ideas I had before. Now that we're able to decompile everything to XML we could probably do proper mod patches.

What I'm thinking is that we could use RFC5261 "An Extensible Markup Language (XML) Patch Operations Framework Utilizing XML Path Language (XPath) Selectors", or maybe Microsofts own XML patching stuff built into .NET (which seems different to RFC5261), or some other XML patching frame work.

There's a lot of benefits I can see to releasing mods as patches rather than full MBIN files:

• Mods can work together on the same file: since the patches would only change certain parts of the file, multiple mods could edit the same file. Conflicts where multiple mods edit the same field could possibly be detected too as long as a record of patches applied to the file is kept.
• No copyrighted data included at all: right now all MBIN mods released are sharing the modded MBIN file, which is probably 99% copyrighted data, 1% modded data, and could be taken down at any time if HG wanted.
• Reduced size: patches only really contain the data that's changed, and a locator for that data, making them much smaller to download.
• Potentially works with game updates: right now, if the game updates an MBIN which is used for a popular mod, people who use that mod on the updated game would still be using the un-updated data included with the mod, which could be bad. With a patch system like this, mods would still likely work against game updates.
• Only contains actual mod data: I've seen a few mods released where the PAK files also contain original, unedited game MBINs. This relates with the issue above about using un-updated data with an updated game. By using patch files this could never happen, as mods would only contain patches that actually modify data.

Of course this would require some work from the developer of the mod manager, it'd need to have MBINCompiler built into it as well as the ability to unpak the original MBIN, and then repack the modified MBIN as a PAK file.

On the flip side, mod authors would no longer need to work with PAK tools to pak their mods - instead, they'd need to use MBINCompiler to create a patch file for their modded files, which is a decent trade-off since mod authors would likely already be familiar with MBINC. They could then distribute their mods using normal ZIP files, though users would need a mod manager installed to apply them.

The way I could see a mod author using it:

• Decompile MBIN to EXML
• Edit EXML file with mods
• Use MBINCompiler to create a patch file, eg. "MBINCompiler -p path/to/original.mbin path/to/modded.exml path/to/patch.pxml"
• Do the above for each MBIN being modified, then zip them together and release

And the way I could see a mod manager working with these files:

• Load mod zip file
• Extract patch files from zip to a ModFiles folder (which also contains previously loaded patch files from other installed mods)
• Load, but don't apply, each patch in ModFiles
• Check each patch against each other to detect any conflicts
• ---> If conflicts, notify the user and exit
• Delete _NMSMODS.pak file (stores the result of MBINs patched with all the patches inside PatchFiles)
• Create temp PatchResults folder
• Loop through each patch file:
• - Extract source MBIN from game PAKs, and decompile MBIN to EXML with MBINCompiler (or use EXML from PatchResults folder if it exists)
• - Patch EXML with patch file
• - Store result EXML in PatchResults folder
• Once all of the patch files have been processed, use MBINCompiler to compile each EXML in PatchResults folder to MBIN
• PAK all the MBINs into PCPACKS/_NMSMODS.pak
• Mods have been applied, user can launch the game and enjoy!

Only main issues I can see with this are:

• Users would be required to use a mod manager/tool to apply mods: IMO it isn't so bad considering the advantages it brings.
• Binary data wouldn't be supported, only data that can be decompiled to XML: best case scenario there'd be no need for binary data to be included in a mod, but of course textures aren't stored as MBIN, so naturally they'd need exceptions where the mod manager would copy them in. In the example above, textures could always be extracted to the ModFiles folder and then added to the resulting PAK file, this would allow conflicts to be detected too, but would then take up double the disk space.
• Templates need to be supported for the system to work: Right now all the main templates are supported, and the only ones missing are globals templates. If a mod edits a globals template that hasn't been mapped, the author should edit the skeletons provided in the MBINCompiler source, naming the field they're editing, and then decompile the global and edit the EXML instead. This action would not only help us, but help them too, by giving them an easier way to edit the data they've found.

Any thoughts?

[emoose]

The main post was getting a bit long, so I've split this up.

Some people might think "why not just make binary patches for the MBIN files?", that's one thing that could get around the copyright issue, but it wouldn't be able to allow mods to work together, or detect conflicts.

Mods might work together, but binary patches don't know the sizes of fields, they don't know the data they're patching, they only know that the data at offset x should be changed to this y bytes buffer.

As an example, what if an MBIN had a field at offset 4, an integer field that's 4 bytes, a mod then changes that integer, but in the file itself only one byte of the integer changed, say the byte at 6.

To the patcher, it'd only know that a byte at 6 changed - not that the integer at 4 changed - so when another mod changes the integer, which results in a change at byte 4 this time, the patcher wouldn't be able to detect this conflict.

It'd only know that the patches aren't changing the same bytes, not that they're changing the same fields, so the patcher would allow the two mods to work together, in the process creating it's own franken-mod monster where two mods have edited the same field, with the result that neither mods changes have been properly applied. Of course this is an uncommon scenario to happen, but it will happen eventually, given enough mods being installed.

With XML based patching this couldn't happen. The patcher is aware of each field being changed, there's no chance of only part of a field being changed, to the patcher the whole field has changed. If another mod changes that field, it'll be able to see this and alert the user to it.

[hhrhhr]

@emoose

No copyrighted data included at all

references to copyright in this case completely inappropriate. ...\No Man's Sky\EULA.txt: "You may not... in whole or in part reproduce, translate, reverse engineer, derive source code from, modify, adapt, merge, translate, disassemble, decompile, or create derivative works based on or of the Game...".

Mods can work together on the same file Reduced size

modders can use an ordinary diff+patch.

Only contains actual mod data

diff file much clearer and more useful than the xml-patch. for example, i want to change the price of a single resource. in the form of a xml-patch it will look like this:

<diff> 
  <replace sel="/Data/Property/Data[11]/Property[@name='BaseValue']/> 
    <Property name="BaseValue" value="600" />
  </replace> 
</diff> 

here is not visible old price and it is unclear at what exactly the resource (what kind of index 11?) i change the price. but if like this,

--- NMS_REALITY_GCSUBSTANCETABLE.old.exml   2016-08-25 17:36:37.519855100 +0300
+++ NMS_REALITY_GCSUBSTANCETABLE.new.exml       2016-08-31 09:31:31.253102900 +0300
@@ -107 +107 @@
      <Property name="Id" value="TECHRARE1" />
@@ -133 +133 @@
-      <Property name="BaseValue" value="60" />
+      <Property name="BaseValue" value="600" />

as well as immediately clear what was going on.

besides, i can use a lot of tools to produce "diff-code". but how and with what i can automatically compare two xml-file and get the RFC5261-like file differences?

[emoose]

I didn't really think of diff patching, but I could see some problems with it, eg. what if the locator that diff patching uses

<Property name="Id" value="TECHRARE1" />

is no longer in the file anymore, or has changed a lot since the patch was made? (thanks to updates, or other mods changing it), with XML patching which uses XPath as a locator this would still be an issue, but much less of one as the locator isn't as exact as the diff locator (eg just changing the value of that property would throw diff off, but with an XPath based on selecting by "name='Id'" the value of it wouldn't matter)

diff patching is more suitable for text data, like source code, but for structured data like XML there are better alternatives available to us. Readability of the patch data isn't really a concern, since the tool will be generating/reading the patch files for the user.

From the XML patch you showed above that seems pretty readable to me though.

here is not visible old price

The old price shouldn't be a concern, since that could have easily been changed by the time the patch is being applied (by updates or other mods), having that be a part of the patch file would make the patch more likely to fail if anything.

and it is unclear at what exactly the resource (what kind of index 11?) i change the price

Well the way I read it, it's changing the 12th Data element inside the Property element, and then searching for the Property inside that Data element by the name "BaseValue", once it's been found it replaces that Property element with the one inside the patch.

If we wanted these patches more readable (showing the original data being changed and what it's being changed to etc), it'd be trivial to implement a standard diff into the patching program, just for letting users see exactly what's changed, but while that's useful for users it's less useful for making a patch that can work well with other patches.

[hhrhhr]

what if the locator that diff patching uses ... is no longer in the file anymore, or has changed a lot since the patch was made?

if you add a new element to the structure or remove old one, then the index of the desired item can be changed. in that case xml-patch is also not acceptable: diff-patch did not work and can report the error, but xml-patch change is not what is needed.

[emoose]

Well I've decided that the best way to get the ball rolling is with a proof of concept. Over the next few days I'll be moving MBINC's main code into a class library, along with porting over spAnsers excellent PSARC packing/unpacking code to C#, which will allow us to handle PSARC/PAK files without needing to touch any copyrighted Sony tools. (MBINC won't be using this PAK code ofc, but the proof of concept will, along with any other C# programs written by others that wish to use the PSARC code)

This MBINC repo will still be used as the main repo, but MBINC will be relegated to a simple frontend for the code in the class library.

Meanwhile the proof of concept will show how patching + paking is the best way for mods, by allowing the features stated above (multiple mods editing the same file, game version independence, smaller file size..) in a simple to use app.

My hope is that through this proof of concept other mod managers might come around to using the same system for their mods.

(for anyone worried about the transition, it'll be done in a separate branch to the master of course, so any contributions people make won't go to waste. Keep making those pull requests :)

@hhrhhr

if you add a new element to the structure or remove old one, then the index of the desired item can be changed. in that case xml-patch is also not acceptable: diff-patch did not work and can report the error, but xml-patch change is not what is needed.

The XML patch would still have a better chance of success over the diff patch though, but error reporting is a good point.

Not sure how we could detect all the possible errors that might come up, maybe in the patches we can give the locator more specific information to locate the field that's being changed, and then if it isn't found we'll get our error, not sure how feasible that is to implement though.