It's useful to think of me3 in terms of a regular software project, despite it not actually doing anything on its own or being a library to be consumed. Let's think of what it means to be "stable"
No change in behaviour between releases. The same mod should behave identically under any release of me3.
No changes in the exposed interface (APIs, binaries) without expressing that as a breaking change
Backwards compatibility with legacy work where possible
Up-to-date and correct documentation
Now, let's think of what it means to make these points "provable" in real concrete terms for me3.
No change in behaviour between releases
We need automated tests. Not just simple unit tests, but tests that launch the game and interact with it, proving that the framework is called in the way it was intended and produced the expected results.
A well-defined format for specifying mod profiles with schema evolution in mind.
Integration tests for popular mods, making sure that their setup still works between changes.
No changes in the exposed interface (APIs, binaries) without expressing that as a breaking change
We need to specify exactly what is expected to be delivered as part of a me3 release and verify that these artifacts are actually being produced by the packaging process.
Backwards compatibility with legacy work where possible
The existing config.toml from me2 has a lot of flaws, but there is a lot of existing work built on top of it that shouldn't be forced to push out updates to users or rely on an old version of me2.
Up-to-date and correct documentation
Much like rusts doctest, our documentation can be actual runnable examples of me3 mod profiles with their own basic smoke test expectations
Each of these are technical challenges on their own, but here are some proposed solutions:
Automated integration testing
We want this to happen in CI, not manually. The base requirements to do this are:
A Steam installation with the games under test installed and up-to-date.
Hardware good enough to spin up games multiple times during CI.
A single server isn't really going to be enough. We need to test this on various configurations with differing operating systems, installation methods, locales, etc.
With this we can at the very least script the interactions with me3 and Steam and do some basic interactions, however that quickly grows cumbersome and is too limiting to really do anything. Instead we can leverage the existing cargo test harness framework. The basic idea of this is the following:
Write tests that assume me3 is loaded into an instance of the game and provide it a mod profile.
Create a test harness that is able to inject itself into the game as a DLL.
Use that injected DLL to go through each test function, and call the respective export in the DLL.
Doing this would let us reuse everything that comes with tooling that already integrates with cargo test (reports, retrying, timing, etc.)
Well defined mod format
The current config.toml of me2 is very loosely defined and is used both for me2 configuration and mod configuration. This should be split into configuration and mod "profiles" so we can better decide which options should be overriden and which should be appended. Where this gets tricky is when we want to evolve the schema and replace existing features/options.
The kubernetes project has a good solution to this with its apiVersion property. See an example of how this would work in practice.
In the future we might want to make the external_dll options richer:
me3Version = "0.2.0"
[dlls.dll_1]
name = "1.dll"
init_function = "mod_init_func"
This approach allows code assistance in most editors with language server support to work as soon as you fill in me3Version and will give you all the versions to select from, also telling you if your file is valid or not.
Well-defined package outputs
The packaging of me2 was a bit fast and loose, requiring manual steps after building with cmake to package up the application in something users can consume. Instead, we should ship platform-specific packages that contain all dependencies and handle any additional install steps. Once we do this, we can also verify the packages work correctly when we release new packages.
Prior art for this involved a WiX toolset based MSI installer using CPack. Starting this work back up would be beneficial and provide a simple verifiable definition of what should be included in one of those packages.
E.g., for an installer we should be able to test that:
Launcher binaries were put in the correct place
DLLs were put in the correct place
Any system configuration (like PATH modifications) were done
It's useful to think of me3 in terms of a regular software project, despite it not actually doing anything on its own or being a library to be consumed. Let's think of what it means to be "stable"
Now, let's think of what it means to make these points "provable" in real concrete terms for me3.
config.tomlfrom me2 has a lot of flaws, but there is a lot of existing work built on top of it that shouldn't be forced to push out updates to users or rely on an old version of me2.Each of these are technical challenges on their own, but here are some proposed solutions:
Automated integration testing
We want this to happen in CI, not manually. The base requirements to do this are:
With this we can at the very least script the interactions with me3 and Steam and do some basic interactions, however that quickly grows cumbersome and is too limiting to really do anything. Instead we can leverage the existing cargo test harness framework. The basic idea of this is the following:
Doing this would let us reuse everything that comes with tooling that already integrates with
cargo test(reports, retrying, timing, etc.)Well defined mod format
The current
config.tomlof me2 is very loosely defined and is used both for me2 configuration and mod configuration. This should be split into configuration and mod "profiles" so we can better decide which options should be overriden and which should be appended. Where this gets tricky is when we want to evolve the schema and replace existing features/options.The kubernetes project has a good solution to this with its
apiVersionproperty. See an example of how this would work in practice.In the future we might want to make the
external_dlloptions richer:This approach allows code assistance in most editors with language server support to work as soon as you fill in
me3Versionand will give you all the versions to select from, also telling you if your file is valid or not.Well-defined package outputs
The packaging of me2 was a bit fast and loose, requiring manual steps after building with
cmaketo package up the application in something users can consume. Instead, we should ship platform-specific packages that contain all dependencies and handle any additional install steps. Once we do this, we can also verify the packages work correctly when we release new packages.Prior art for this involved a WiX toolset based MSI installer using CPack. Starting this work back up would be beneficial and provide a simple verifiable definition of what should be included in one of those packages.
E.g., for an installer we should be able to test that: