Noggog
commented
2 years ago Hey! Sounds very similar to the thought processes I had when starting Mutagen. Way back in the day, I had planned to have conversion into json/xml format for the purposes of storing mods in a git repository be the introductory proof of concept project, before Synthesis and Analyzers cut in line and took over in the work queue.
I do think it's a worthwhile endeavor to get mods into source control. A big part of that would be two-way converting esps to a text format like json/xml/other. Being able to leverage Git would give developers a lot of power to experiment, branch, as well as potentially even merge conflicts with other developers in a way that I imagine is very tough right now with just storing/transferring zip files with the content.
Mutagen Fulfillment Musings
As I mentioned, I do/did plan on giving the concept a stab one of these days. As a short blurb on how I would go about it if the time came:
Mutagen exposes classes/interfaces like these:
public interface INpcGetter
{
ITranslatedStringGetter? Name { get; }
float Height { get; }
float Width { get; }
IVirtualMachineAdapterGetter? VirtualMachineAdapter { get; }
IObjectBoundsGetter ObjectBounds { get; }
// ...
}The parsing for how to expose that interface is handled for the user. They just get an object with strongly typed fields to use without any worry about how it's fulfilled. Notice, too, concepts like FULL being associated with Name is abstracted away as an esp format implementation detail.
Obviously we wouldn't want to hand-write the conversion from these interfaces to json, so either:
- Use normal C# serialization libraries like Json.Net that turn C# classes into json and vice versa. Classes already exist and have names, this just turns it into two-way json straight up. This could get a basic
esp<->jsonconverter going extremely quick, I think. Obviously might need some pruning and edge case handling, but. - Use reflection to write our own json serializer. We use C# and the defined classes to iterate all the fields in a mod object and decide how to write out json ourselves.
- Use the same code generation systems Mutagen uses for binary translation to make json translation alternatives. This would take the most work, but would be the fastest as it requires zero reflection, and also can leverage speed increase and memory reduction potential by being an overlay.
Some other items to consider that Mutagen brings to the table for a project like this:
- Strongly typed interfaces/classes that can be used in existing serialization libraries, as mentioned
- Already has 99%* coverage with Skyrim/Oblivion, and will have Fallout within the month, I expect. The systems are tested to pass through both ways with byte-by-byte accuracy.
- Handles and abstracts away translated string systems
- Equality/hash calls (needs a wee bit of correctness testing, still)
- "Translation Mask" concepts to selectively skip fields during serialization, equality, or copy calls. (docs not yet written)
- Handles FormID reindexing when masters change internally, powered by the FormKey systems
- FormID iteration and remapping tech. (no docs yet). Essentially can see all the FormIDs that exist within a mod, as well as give the system a
Map<FormKey, FormKey>and it'll replace them appropriately - FormID persistence systems developed as a way to track which keys are taken. Not sure if it would be needed, but something that might potentially come up
FormID Collisions
The talk so far has been about converting json<->esp, but I think there's a few more aspects to consider.
What happens when modder A adds 10 records, and modder B adds 10 records. They convert to json and commit into Git. They then push up and merge to master. Well, the merge might go cleanly according to Git as far as the text file goes (two sections of lines added without conflict), but now the esp will have duplicate FormIDs, as both sides probably allocated the same IDs to their new records.
As such, a tool that is empowering mod systems to be stored in Git might also want to offer a Merge helper tool. This would essentially notice that the mod on branch A and branch B have FormID conflicts. It could then load in either A or B, and remap the FormIDs to new values that haven't been taken. This might be powered by the mod loading/inspection/remapping/persistence tools mentioned earlier.
Probably a good deal of other similar "additional" checks/features that a tool like this might want to offer
Trailing Blabber
I'm excited someone is interested in these concepts! I think it would be a large boon to the community to get a tool like this on the ground
All I'll say on that front is that I did develop Mutagen with a lot of care towards not just making it contain features that suit my personal project needs, but attempting to make it a generic and accessible and powerful baseline for downstream projects of all kinds. The hope was that people could focus on cool downstream projects and make a quicker impact with a modern dev ecosystem without reinventing the wheel for the low level gritty details.
As you mentioned, serialization/processing isn't owned by anybody. There's already several that exist (xEdit/SkyProc/Esper/Im sure others). Mutagen is just another. If you love Swift and want to develop a tool on top of that, that's awesome as well. 8)
As far as my feedback on the end goal idea: I think it's super interesting, important, and likely doable one way or another
samdeane
Just wanted to point you at this project.
It's not directly related, and I'm using my own code to parse ESPs.
However the idea of a compiler/decompiler is really independent of the ESP backend, and is something that would benefit from a bit of discussion and peer review. Is it worth pursuing? Does the proposed format make sense?
The ultimate goal is that an entire mod project (esp records, resources, and scripts) could be contained in a structured source directory, and tools could automatically package them up as BSAs and ESPs, including compiling the scripts.