Open MythicManiac opened 3 years ago
Hi, a conversation already occured in a different place so i will do a recap of the ideas we shared and of the ideas we discussed but did not agree on yet. Anything written below is to be interpreted as an RFC draft and is subject to changes based on feedback.
The CLI (Command Line Interface) tool, that will be called CLI for the rest of this message for simplicity has one and simple goal: Allow the user to install, uninstall, download and manage the mods for many games. The first implementation of this CLI will be done to support the game Cyberpunk2077, but will eventually evolve to more than just this game.
REST API is the web service that listens for HTTP requests and manages the full mod database. The REST API shall also be mentionned as the Store
the CLI tool is the tool we are talking about
the GUI is something we have in mind, a graphical application that could sit on top of the CLI tool and could leverage it to make mod management easier for non-programmers. It would not implement anything and would just serve as an interface between the user and the complex commands the CLI will accept.
The words should and must will be used and are there to imply that during the implementation of a program the following information has to be exactly as stated in order for the program to be considered standard
First let's agree on what a CLI tool should do and should not do. A CLI tool is a tool that is often run through the command line to execute a piece of code. For example the following CLI tool cat
allows the user to run the cat
tool with a parameter to get the content of a file. The result of such a command would be cat my-file.txt
. The result we get out of the CLI tool (what is called stdout) is caught by the command line interface of the user and is displayed at the screen.
So basically, we have a program/binary (the CLI tool) that can accept many commands like install
, remove
, publish
, etc... (not real commands) that will output text of any form. This text can then be captured by anything that ran the program (our command line interface or even a GUI) and parse the result and act accordingly.
Now that we can agree on the idea of what a CLI should do, let's start talking about the basic commands the Games Package Manager cli should have to be considered a minimum viable product:
install
for downloading mods from the internet by contacting a Store. The choice of store could be offered to the user through an optional parameter if needed, something like install package-name --store new-store-address
.remove
to remove locally installed mods from the local repositoryinit
to help the user bootstrap a project for his future mods. It should by default act as an interactive command with prompts to fill the default values of the configuration file. Such a feature should be skippable through an optional parameter, something like init -y
to skip the prompts and use the default values everywhere.enable
to enable a mod that is in the local mod repository so the game will detect the new moddisable
to disable a mod that is currently enabled and is in the local mod repository so the game won't detect the mod anymorepublish
to bundle and publish a mod that follows the convention required by the store. For an easy setup the user should refer to the init
command. The publish
is expected to be run inside the root directory of the mod to publish, but this point is up to the person implementing the CLI of course as it could also use optional parameters to supply a path to a mod directory.The CLI tool has the responsibility to download, install and manage the mods without any user interaction (except from the commands, obviously).
The CLI tool should directly interact with the end-user filesystem to manage and install the mods.
The CLI tool should work entirely on the filesystem and the metadata files the mods have to manage the mods and should not use any database to store which mod is installed or not. The use of symlinks is recommended to avoid duplicated data on the disk of the end-user.
The CLI tool should only compare what's in the game directories and what's in the local mods repository and act accordingly in order to avoid any loss of track if the game install is modified by an external tool or by the user itself.
About multiple store: as said in the other thread, multiple store should be allowed a way or another. The method this will be done is unsure, but I will draft something about it (based on @Aelto comment in the other thread). Will assume as of now that this will work in a way similar to debian (multiple store that are seen like they are one, assuming that multiple store who have the same mod (identified by id) will result in a unique being selected based on version):
we would need to have command to manage those list of store. Suggested change:
install
will now search by default in all the allowed store, and install the more up-to-date version avalaible in any one store (where to look for those dependancy are unspecified, either all the trusted source or only the source wich host the selected mod). Only --store
will restrict the search to one remote.
add:
store add <store_id>
add the store list
list all the store installed.
store remove <store_id>
remove a trusted store. Keep mod installed via it. Allow removing the default(s) one.
there should be a/some default store trusted by the default.
unrelated: allow installing from local file via install ./mod.zip
or ~/mod.zip
or /path/mod.zip
or anything://path.zip
(the last one for windows, but could maybe also include https)
Slightly formatted excerpt of what I posted on Discord:
So I'm thinking we could make developers and users use the same interface with the tool
Thoughts?
@MythicManiac This seem a good idea, in particular when we want to be able to easily share mod definition. Some issue that could happen :
We need a better to specify dependancy. For example, if we want to install a mod from a local directory, the depency should point to it (the folder it has been unpacked to). This add the need for multiple dependancy source (but is otherwise a good idea).
If we are to continue the earlier idea on how we handle multiple sources for packages, this does not seem like the correct way to handle it, as we'd be hardcoding information about where a package should be obtained from to the dependency. Instead of doing that we could for example support defining a local package repository (which is just a directory with packages somewhere) and use packages from there as a higher priority, or simply have a parameter (e.g. --from-file
) when installing the packages. Thoughs?
Continuing, maybe we could take git-like approach, where you can define remotes (or local file usage) on a per-repository basis? this should allow for convenient local file usage if you want without requiring you to write it to the dependency file (that will be built into a package when publishing)
We should need more global state about application, mainly thinking for mod that replace/add file. Uninstallation should need to remove/restore them.
I think the current philosophy is that we do not want to store state (because it will get inconsistent), but what packages are installed should be obvious from looking at the files themselves. In reality this might be rather challenging, and I see two ways to go about it:
Not sure I would like to cd into my standard mod folder every time I want to just a simple mod. Can be circumented via a GUI that abstract this, or a CLI flag that specify some global profile (this is not too important, and can be thought about later)
Yeah agreed. We could have global contexts, or parametrize filepaths, or the default profile, etc. many solutions should exist to make the usage convenient on top of the base solution proposed here.
My current thoughts on the command structure:
Command | Description |
---|---|
init | Initialize a new project to the current workdir |
start | Start the project's game with the selected dependencies installed |
add | Add a dependency to the project |
remove | Remove a dependency from the project |
list | List the current project dependencies |
search | Search packages from the configured remotes |
build | Build the project into a package |
publish | Build the project into a package and publish it to a remote |
remote list | List remotes |
remote add | Add a remote |
remote remove | Remove a remote |
cache list | List packages in the local package cache |
cache add | Download and add a package to the local package cache |
cache remove | Delete a package from the local package cache |
This is still lacking several things, and assumes we'll take a lot of parameters to each of the commands for more fine-grained control if needed. Some considerations for missing features:
Also worth considering how do we configure what game is the project targeting or supposed to launch. We could have a gpm init --game cyberpunk2077
style approach for example.
Also worth considering how do we configure what game is the project targeting or supposed to launch. We could have a gpm init --game cyberpunk2077 style approach for example.
I suppose the packages will have a field to specify which game they work on. So if a package supports only 1 game it could use it as the default value. Otherwise your --game cyberpunk2077
idea sounds good
@Aelto I don't think it's a smart idea to require packages to define the game they support, although they could have a "preferred" game or a list of preferred games. Reasoning for this is that some packages might be applicable to multiple games (e.g. imagine you just distribute a 3d model). SOME WAY of figuring out whether or not a package is compatible with a certain game would be very nice, but I'm still unsure what that could be
Briefly read this over through a post in #wiki-updates on Discord and it's quite late in my area so I may not have the full picture but, nonetheless, here are my thoughts on this:
I don't think it's a smart idea to require packages to define the game they support, although they could have a "preferred" game or a list of preferred games.
@MythicManiac The way I see it, the only valid reason to have packages require explicit declaration of which game(s) they support would be if GPM would be used in the future to support more than 1 game at a time. Best way to do this would be to assign a game ID code to a game (eg. Cyberpunk = 1, Game2 = 2, Game3 = 3, 0 being "automatic", which would default it to Cyberpunk only for the time being, but could change in the future). It would then be up to the mod developer to include that flag in the mod's config/files, so that the mod manager could identify it when loading mods from multiple games.
I suppose the packages will have a field to specify which game they work on. So if a package supports only 1 game it could use it as the default value. Otherwise your
--game cyberpunk2077
idea sounds good
I would use this more as a "force the package manager to create cyberpunk2077-based bootstrap", which would be different from the default environment. However, if GPM will exclusively be used for CP2077, then there is no need to implement any of those flags, because it will load all the mods in the same way.
Thoughts?
@Cryotechnic About the first point, it may be a good idea to specify what the mod is compatible with with a list of String. For example (taking an exemple of a mod that work with both the withe 3 and cyberpunk 2077) [ "witcher3", "cyberpunk2077" ]. Maybe add group, that could be used as in ["openmw-engine"] actually mean ["morrowind-openmw", "morrowind-tes3mp", "othergame-openmw"] (tes3mp is a multiplayer form of openmw.)
Multi-game support is something I think we want to do given the quality of the tool we're building. I agree we need some kind of a game identification system, and like @marius851000 suggested also I'd personally rather use strings that are clear (e.g. cyberpunk2077
) than a cryptic numbering system.
Let's think about what's the game-specific information we need to track, because that's the primary relevant part. What comes to mind for me is at least:
So if we take a step back and look at this, what we really care about is the paths. IMO it would make sense to create discovery rules with a game identifier that autodetect these paths, but also leave it possible to manually configure them. This would make it so we don't need to "bind" a project/profile into a single game.
So how do packages get installed if they don't know what game they get installed to? That problem should be solved by the pluggable install strategies, which handle the installation given a package and a game install path. If we have game or engine specific install strategies, they could validate that the configured game path matches their expectations, and error if they're incompatible with it. Some strategies might opt in to do their "installation", which if applied to the wrong game, would just end up doing nothing. I'd feel like this is an important capability to have to create a good ecosystem, as longer term we could see packages that simply share models for example that get used across multiple games.
So to summarize my opinions:
The conclusion is that there's no technical reason we need to care about what game a package belongs to. The primary purpose of being able to relate a package to certain game or games is to make it easier to discover by the end users. This information however does not need to be baked in to the package (which is immutable), and could instead be configurable as dynamic metadata on the package repository website/API to enable appropriate search filtering.
Thoughts?
@MythicManiac @Cryotechnic I really think that the list of game should be a in the metadata, be it for discoverability (and warning the user, with a non-fatal message). It should be better if we want to just add an existing into a package simply (without having to fill supported games) (and then the remote read the list of supported game from the package).
@marius851000 the primary problem I see with that is that we'd have to pre-define each game to have a schema to validate against, or alternatively we'd just accept freeform strings, which would make it very close to the tags field.
Another problem is that now you're writing this information to something immutable, which means if you ever want to add a supported game, you have to bump your package version for practically no reason.
IMO it would be a good idea to separate dynamic metadata that might evolve over time from immutable metadata that we want to bake in, because immutable metadata changes will always require a package version bump.
So this really comes down to a design choice over anything else, we don't have a technical requirement for it but it could be useful. Can we come up with a couple of example scenarios where and how this metadata would be used to illustrate the need of the field (or lack of)?
I personally think that any information that end up on the user computer after downloading and closing the application should be stored in the mod file. That indeed mean we will need to update the file to add support for a new game even if it is just to update the list of supported game.
I see them as a list of reconized id (like [game1, game2]
, but they can also be [game1, engine1]
for matching the game engine. In this case, if either the game we install the mod into is game1
or user engine1
, no warning is displayed, otherwise, a warning is printed. For mod that use multi-engine file format (like standarized gltf), we can also use [format1, format2]
, and if the game is registered for at least one of those format (in the same way for the engine), no warning is displayed. We can end up with stuff like:
["engine-openmw", "format-dae"]
I personally think that any information that end up on the user computer after downloading and closing the application should be stored in the mod file.
I agree with this point, mod packages should self-contain all the required information for their operation. This is also partly why I don't think we should have a required game field in there that completely dictates it's operation, since:
Tags
field, aside for it's intended usage
3.b Have validation that will block creation of packages to games we don't have in our database yet (be it hardcoded into the CLI utility or hosted online somewhere)I see them as a list of reconized id (like [game1, game2], but they can also be [game1, engine1] for matching the game engine. In this case, if either the game we install the mod into is game1 or user engine1, no warning is displayed, otherwise, a warning is printed. For mod that use multi-engine file format (like standarized gltf), we can also use [format1, format2], and if the game is registered for at least one of those format (in the same way for the engine), no warning is displayed. We can end up with stuff like:
["engine-openmw", "format-dae"]
This seems like a overlap of responsibilities with the InstallStrategy
field. If we list supported engines, what's the actual difference between listing supported install strategies (which are bound to be engine-dependant for some, some might be generic)
Could we focus on what is the end goal of the would-be game field and explore all options that could be used? Right now there's a lot of points being made for having a game field, but a case hasn't been made for why it's needed and what it should be used for exactly, which makes proposing alternatives difficult. I feel like at the moment I don't have the required information to reject or approve of the idea; I can only point out constraints it implies.
The main use case I can see for them if for the user filtering them in the GUI. For example, we want to list all downloaded mod that can be enabled for a specific profile. For example, hiding mod for cyberpunk2077 while displaying those that can work for skyrim (assuming we want to mod skyrim).
Some elaboration of my current thoughts:
A game has:
cyberpunk2077
)The project
term is interchangeable with profile
, usually called profile
when talking about mod manager context, and project
when talking about development tool context.
A project is a working directory, under which all of the project's dependencies are stored in a configuration file. When the project is launched, GPM makes sure the all of the selected dependencies are appropriately installed in to the game
Each project is configured to target a specific Game, which in practice means they store the target game install directory and launch executable. The project configuration is stored in a configuration file within the project, e.g. akin to how git repositories retain configuration within the .git
folder.
Install strategy is a module, which given a mod package and a game install directory, will install the supplied mod package to the provided game install directory.
Install strategies should also provide a way to check if they are compatible with a specific game install directory. This means that the install strategy module is given a path (to the game directory), and it has to return whether or not it is applicable to that directory.
Given the above assumptions, we know that:
Using these design guarantees, we can follow the following operation model:
Each project is configured to target a specific Game, which in practice means they store the target game install directory and launch executable. The project configuration is stored in a configuration file within the project, e.g. akin to how git repositories retain configuration within the .git folder.
You should replace project with profile. It would be nice if only profile contain installation specific information, and to remove them when publishing/packaging the file.
about the filtering : If i understand well, we do something like:
supported_install_method
supported_install_method
in the install method.This can lead to issue for generic install method like VFS, I think (after all, all directory that contain a game should be a directory that can be patched via a VFS)
You should replace project with profile. It would be nice if only profile contain installation specific information, and to remove them when publishing/packaging the file.
This is actually why I drew an analogy to .git
, because the configurations under the .git
are local only, so I'd assume any such configuration (target game, paths, etc) would also be local-only in our case. Good for pointing that out, should have mentioned that in the description 👍
This can lead to issue for generic install method like VFS, I think (after all, all directory that contain a game should be a directory that can be patched via a VFS)
Yeah it might cause issues, hard to tell before we know what the VFS implementation ends up being like. I do think we have a fairly layered filesystem approach already which should be compatible with this plan, but let's see how will it turn out. Another thing we can do on the long term is have different options aside for just VFS, the important part is that the install strategies wouldn't know they run in a VFS, they just operate on files like any other tool. But maybe we should open a new issue about the VFS implementation details 😄
Anyway, on a conceptual level my main philosophy has been that install strategies should be the primary contract between a package and how it should be installed, and any compatibility checks would be performed by the install strategy. When a package developer chooses what they target with their package, it shouldn't be a specific game, but a specific install strategy. Install strategies get to define their own guarantees and limitations, and could target a specific game, an engine, or even multiple engines.
If the VFS prevents our current approach somehow, I still believe keeping the separation of responsibilities above is a good idea, and we should explore other ways to make it work.
This way we can nicely support generic packages as well as very tightly coupled game-specific packages, as it's all up to the install strategies. To draw an analogy, install strategies would work as almost the same way "build targets" do in a more traditional programming terminology.
Now, we need to know how to handle the fact that profile use only a single identifier is specified in the project file, yet we want the end user to specify a specific version. I propse:
Specifying the depency (that's similar to how rust does them:
[dependancies]
mod_any_version = {}
mod_some_version = {min_version="1.0.0"}
and in rust:
HashMap<String, DepencyParamater> // first string is id
struct DepencyParameter {
min_version = Option<String>,
max_version = Option<String>,
}
That sounds good to me, and something like that is probably necessary if we want to support version ranges (which I think is a good idea). So the lockfile would be used during runtime, but we probably want to build the ranges into the package for distribution, rather than the lockfile?
@MythicManiac yes, the range is included in the project file (and default to nothing). I'll start implementing this later today.
yes, the range is included in the project file (and default to nothing). I'll start implementing this later today.
I do think there needs to be some sort of "default", base version of a package that all mods need to have in order to be discoverable by GPM? This would be useful in the event of a game update, breaking some mods. We would add a patch to the GPM and bump the version up so that users would not see mods that are not compatible with the current version of the game.
Another option would be to specify which version of the game (the lowest version needed) the mod(s) need in order to run inside the lockfile/project file.
For something totally different : Is it still a good idea to use both json and toml for files ? We can use toml for every file, only keeping json for communication with the GUI (if the GUI does parse the output on stdin of the command line after adding the --json
flag instead of using interopability tool (like the C interability))
I would still keep JSON as the serialization format for metadata built in the package zip. Reason is that GPM is not the only tool that will read package information, and as discussed before, JSON is by far the easiest serialization format to be a consumer of.
We definitely could use TOML, but I'd like to hear what the major advantages of doing so are, instead of doing it just for the sake of it. IMO there are very clear advantages to using JSON as the build target format.
Okay. So I'll keep it so only human edited format are toml (and so keep the json for the lock file). (ps: with rust and serde, json and toml are as easy to use)
The Game Package Manager (or GPM) for short is a CLI tool which primary purposes are to:
This issue is meant to discuss the interface we wish to expose from this CLI