Manifest v2 idea gathering

[RiskOfThunderBot]

Thunderstore package manifest is getting a revamp, list suggestions on changes/ideas you'd like to see under this issue

---

Beep, boop, I'm a bot! This issue was created by @Mythic in #thunderstore.

[MythicManiac]

Section for "works with" mods, as defined in the issue #84

[mistername]

Client / host flag

[PlexusDuMenton]

ability to update read.me without having to increment version

[ontrigger]

package hash (excluding the readme and the manifest) also would be great if you could get the hash via the api

[MythicManiac]

@ontrigger I have a hard time seeing on how that can be made fillable to the manifest itself. Thunderstore itself actually tracks the SHA1 hashes of the zip files already, but the API doesn't return them at the moment. Would that be something you'd like to see?

If the idea is to have a hash for the entire file (which is a zip), without the manifest/readme (which are in the same zip), there's a bit of a logistics problem. You could of course have an index of all the files in the package and their hashes as a separate metadata file, but even then how would developers input this?

[kristiansja]

tag for plugin|framework|api|other|program

[varstyx]

A section for "incompatible with".

[ToyDragon]

A field for display name

[ontrigger]

@MythicManiac good point, i originally wanted this feature for my mod manager so that i could reliably (although very hacky) detect if a mod is not modified (can also check hashes against git) with malicious intent, but very soon stopped working on it due to exams and a few problems with the implementation.

However, if r2api loaded mods from the zip file, it would not only fix the hashing issue, but would also allow mods and mod managers to easily check the mod version/hash/other stuff and display it to the user without having to read the .dll file.

[varstyx]

@ontrigger R2API is not loading anything - BepInEx is.

[ontrigger]

@varstyx I know that, but bepinex was not created specifically for r2 and its creator might not want to change the mod loading method as per our guidelines. Therefore, it would be easier for r2api (which is made by r2 community members) to find .zip mods and load them by itself

[nihaals]

To not have a manifest file and instead have it part of the upload request, or to be able to remove specific files from the download zip, e.g. you could exclude all but the .dll so when you download the zip from the site, that's the only file you get

[varstyx]

@ontrigger R2API will not be loading any mods as it is a mod itself. You might want to look into writing a custom loader for BepInEx that supports loading from zips.

Also, to stay on topic, how about being able to leave out optional sections (dependencies, url, etc.) that are unused, instead of having to declare them as empty?

[ontrigger]

@varstyx i don't really care what will load the mods, r2api was just the most suitable example, as it is used by every single mod out there EDIT: although having to put a custom loader is a big nuisance for the end user and it would be great if the loader came in the same package as bepin and r2api

[MythicManiac]

@OrangutanGaming have you considered how 3rd party tools (such as mod managers) would interact with mod packages in that kind of a system? I at least can't immediately think of how that would work, given that mod packages aren't necessarily bound to Thunderstore or the Thunderstore API (e.g. if you're offline)

[bbepis]

• Semantic version ranges for dependencies (> 1.0.0, 2.0.0 to 3.0.0, etc.)
• Listing other mods that conflict, and listing mods that are recommended alongside the mod
• Ability to list game version(s) / range that the mod is compatible with, either as a regular dependency or a special field

Possibly unrelated but a standard on how packages should be structured out, or something in the manifest to specify that a folder inside the zip gets extracted to a special location (plugins folder, prepatchers folder etc.) so that automated mod managers can download and extract the zips without needing manual user input.

[ontrigger]

I know it's off topic, but i'd still like to know as to why it is not feasible to load from zips instead of extracted folders

[nihaals]

@MythicManiac but the manifest file is thunderstore specific. You may be defining a specification that anyone can use, but it is thunderstore that is defining it and there are already other requirements for thunderstore that other systems might not need. With other systems, not necessarily just mods, the loader tends to define a system that is easily computer readable. Having metadata in the code somewhere wouldn't really work as it would require either weird parsing or the tool to be made in C#. A better method could be some kind of comment on the first line of the file, which includes all the information. This stops confusion over which files people need to copy however wouldn't effectively work with .dlls. Another system could be to have the file in the root of the folder and force users to place their actual mod in some sort of subdirectory so users can simply copy that folder or they know to only copy that file.

The main problem that I think should be fixed is having so many files forced into your zip that the user also has to download when they do not need them and could lead to confusion.

[MythicManiac]

@OrangutanGaming The point of the manifest is to not be Thunderstore specific, and to be a standard that's shared across the entire tooling ecosystem. That is precisely why right now we have a public suggestions thread to discuss properties that should or should not exist on the mod packages.

There already are multiple 3rd party software that assume the manifest format to be as it currently is, and likely will be even more in the future. Keep in mind that this is the package format and the package metadata.

If what you really are worried about is the user experience when it comes to extracting zip files properly by hand, while a good discussion point, I believe that's not the kind of a problem we're trying to solve in this issue. Ideally users don't even have to see the zip files nor the mod files, as some tool will manage those.

[scottbot95]

At least for BepInEx mods, it might be nice to have some sort of mapping of files/folders in the .zip to the desired install locations relative to the Risk of Rain 2.exe or the BepInEx folder.

As an example, the R2API has a plugins and a monomod folders which are to be installed in their corresponding folder inside the BepInEx folder.

{
  ...
  "install": {
    "BepInEx": {
      "plugins":"plugins",
      "monomod":"monomod"
    }
  }
  ...
}

In the case of R2API the desired install path can be trivially inferred but there is no current standard enforcing that mod authors structure their packages in such a way that such deductions can be made.

[dustinlacewell]

Here is a stub of the class underlying the Disunity mod meta-data. It does not have an arbitrary mapping of where to place files from the mod archive into BepInEx. Disunity does not expose BepInEx to mod authors/users. Instead, mods simply facilitate utilizing those underlying features as formal semantics in the mod format:

public class ModInfo {

        private string _name;
        private bool _isEnabled; // should this mod be loaded?
        private ContentType _contentTypes; // what kind of content does this mod have?
        private string _author;
        private string _description;
        private string _version;
        private string _unityVersion; // what version of unity exported this mod?
        private string[] _preloadAssemblies; // what assemblies should be loaded during preload?
        private string[] _runtimeAssemblies; // what assemblies should be loaded during runtime?
        private string _runtimeClass; // what class should we call during runtime?
        private string _runtimeAssembly; // which runtime assembly contains the _runtimeClass?
        private string _preloadClass; // what class should we call during preload?
        private string _preloadAssembly; // which preload assembly contains the _preloadClass?
        private string[] _dependencies // what other mods does this mod depend on?
        private string[] _artifacts // what unmanaged files were also exported into the mod archive?
}

Now, not any serious deep thought when into what's here. It's mostly just representative of working towards a proof of concept that aligns with the exporter UX work. Soon, you'll be able to specify any number of startup classes for preload and runtime -- I just need to build the corresponding UI for that. And of course, we may decide that we like convention over configuration better and just load any ol class from the assemblies that meets some conditions.

Anyway, the JSON looks similar just with the naming convention from _runtimeAssemblies => RuntimeAssemblies, etc. The ContentTypes is an flag enum denoting the presence of each possible kind of data that can be included in the mod archive. PreloadAssemblies, RuntimeAssemblies and Artifacts are all relative paths into the mod archive. RuntimeClass, RuntimeAssembly, PreloadClass, and PreloadAssembly are names useful for interacting with the assembly loading APIs. Dependencies is a list of Thunderstore mod "full names" which include the author, mod name and version.

Missing here are information about the prefabs and scenes. The Assetbundles for those are currently stored in known locations within the archive and haven't required explicit denotation of their relative location. This is an example of convention over configuration I guess. When we load those assetbundles, we simply get a list of all the assets within them and provide that to mod classes for usage. But if we decide we like configuration over convention, then we could dump an accounting of the assets and scenes that were marked for export in the meta-data, including within which assetbundles those assets and scenes should be found in. This would open the door for multiple prefab and scene assetbundles too (a concern that Spacehamster has raised for games like Pathfinder which keep a single asset within a single assetbundle).

This would also allow us to do additional validation on mod loading. We can warn if there are prefabs that are in the metadata but missing from the assetbundles. That could be useful indeed. We are going to do the same thing for artifacts for example to make sure nothing that was intended for export is actually missing from the archive. If it's any mystery, I'm personally for configuration over convention.

[opl-]

Personally I'm still against having to configure paths to things that shouldn't require that. Something like @scottbot95 suggests in this comment seems very pointless as all of that could be represented by the directory structure either on disk or in the zip itself. Simply putting all the mod files in a data/ directory in the zip that represents and mirrors the game directory structure has several advantages:

• It simplifies the manifest file and makes it less error prone.
• Doesn't make the package format BepInEx dependent (as some solutions I've seen in the past did).
• Makes mods easier to install and uninstall by users manually as it doesn't require cross-referencing the manifest file.
  • Some users will likely insist on not using the existing tooling.
  • Installing and uninstalling mods when tooling fails becomes easier (See the modsaber.org incident. In short the owner of the website has shut down the website with no warning and essentially killed all tooling for at least several days).
• Allows greater flexibility for potential future tooling changes without requiring changes to the manifest format.

Some might see the issue of mod developers using invalid directory names for their mods in the BepInEx directory which could cause conflicts, but this could be relatively easily prevented by validating the package when it is uploaded to the Thunderstore or even by the mod installers. Package format flexibility is, in my opinion, more important than a minor inconvenience for the tooling developers.

Additionally, the manifest should contain a manifest version field for the sake of future proofing. New manifests could start with version 2, since the old ones (afaik) don't have any version specified. This would allow us to easily distinguish between old and new manifests and to relatively easily release updates in the future if needed.

[dustinlacewell]

As far as Disunity goes, the underlying BepInEx distribution wont even be installed into the target game directory -- instead the doorstop config will point back to a remote game-specific BepInEx distribution that is installed within the manager's data directory. Therefore having the mod format rely on the structure of the game's install directory wont do.

[mistername]

Add field for GUID

[MythicManiac]

@opl- Packages like that are technically of course possible, but they pose some challenges for mod managers (e.g. how does a mod manager know how to uninstall a package? can a mod manager tell what mods are installed just by looking at the file structure?)

It's the sort of package I would call unmanaged. I don't think managed mods should need to know where are they placed, rather, they could just define that they are managed mods and let the manager handle the rest (be it manual install or actually using a manager). This leads to us needing to define a standard way for managed mods to be installed (how & where).

Closest to managed mod installs currently is BepInEx plugins. Each mod can have their own subfolder matching the package name, thus avoiding conflicts. The only issue in the current system is how there's several different methods mods require for installation, rather than just a single one (e.g. monomod vs. bepinex plugin vs. extracting on top of the game folder).

I would like to see this standardized to the point that 99% of the mods can be managed mods (which might require other tools to also provide APIs for interacting with the rest of the game, providing filepaths to the mod's data directory for example). For the ever so rare mod that for whatever reason cannot be managed, support for "extract" mode install could be added. This could be used if you need to overwrite game files for example.

I would say Disunity is currently what we should focus on for standardizing the managed mod environment. If you can provide some scenarios for mods you think can not be a "managed mod" (so, has no control over where it is going to be placed on the filesystem), it would be really useful to take those into account at this point.

So please do share if you have any thoughts or concerns.

[opl-]

Honestly? Fair enough. Personally I find the idea of keeping the mods separate from the game directory somewhat odd, but I guess there's no hard reason not to do it. That being said, I feel like my list should still be taken into account. Don't overcomplicate the format or it will become a chore to use for the mod developers or worse – for the end users.

I also wanted to note that I've been out of the picture for a few weeks due to life, so I'm not entirely caught up on Disunity. I will be trying to learn more about it in the coming days.

[MythicManiac]

At the end of the day, managed mods should be less work for the developers than having them figure out extract locations themselves. One big issue with extracting to fixed locations is that it's difficult to change later on, which would mean modders actually have to know what's going on behind the scenes.

Exposing a good enough API for managed mods should allow them to not care how things work exactly in the background, while still being able to do whatever they wanted to before.

[opl-]

In the end it comes down to proper documentation. Standardized locations and config options will both be difficult to use if they require detective work to figure out.

Speaking of, a proper format specification would be another good thing to add to the list.

[MythicManiac]

Closing in favor of #93