[Draft] Package Manager

KAW0 commented 2 days ago

Tested versions

-

System information

-

Issue description

TLDR: Something like Unity Package Manager, but works properly.

Why?

Marketing potential: This would be a great marketing mechanism for the engine. If users browse libraries for Godot and see in the description that the library has support for Redot Package Manager, they will naturally ask themselves what Redot is.
Competitive advantage: Other engines have not delivered a fully functioning package manager in Godot. This would be a significant selling point for Redot.
Team collaboration: It simplifies working in large teams with multiple projects. Instead of developing them separately, you can assign a team to create a common codebase.
Modularity: It enhances modularity, e.g., someone could create a visual scripting library, and another developer could build a dialogue creation library on top of it, while both are compatible with a library for in-game events from yet another package.
Efficiency: It eliminates the need to manually install multiple packages. Instead, you can create a meta package that depends on several frequently used packages and install them all at once.

Why wasn’t it included in Godot?

This has already been proposed, but the creator of Godot was concerned about a situation similar to npm for NodeJS, where a single package can have hundreds of dependencies, and a mistake or deliberate sabotage in one dependency could disrupt the entire ecosystem. This is a valid concern, but there are good examples like the NuGet library for C#, which largely avoids this issue. Most packages have single-level dependencies, and many have none at all.

The reason is simple: when NodeJS was initially developed, it lacked a comprehensive standard library, so packages were created for even the simplest functions, often resulting in duplication. In contrast, C# has a rich standard library, allowing packages to be developed for specific tasks, with many of them compatible with each other through common interfaces. I believe the situation with Godot/Redot is similar the engine already has a robust codebase, so the risk of excessive fragmentation should be minimal.

Technology Options

Creating Our Own Solution in C++

This would likely be the best solution, but it would also involve the most work. Self-explanatory.

Existing C++ Solutions

There are already many package managers for C++, but I am unfamiliar with them, so I cannot comment in detail. This could be the best solution if we find one that meets our requirements.
Here is a roundup of C++ package managers.

NuGet

There is an existing solution based on NuGet, making integration easier. It supports passive delivery, meaning you don't necessarily need a dedicated server. Simply host the files on a server. It’s also compatible with C# packages out of the box. However, the libraries for NuGet are written in C#, so this would require permanent integration of .NET with the engine, which could be a significant drawback for some.

Reimplementing NuGet

Instead of using C# libraries, we could focus on the NuGet standard and implement it in C++. Whether this is better or worse than creating our own system is uncertain.

npm

Please don’t use npm as a base. Let’s don't repeat Unity’s mistake. This system is slow, and its application would take up as much space as the engine itself. Additionally, the authorization system didn’t work properly in Unity, although it’s unclear if that was due to npm or Unity. Using package managers written in other languages also complicates the engine-building process.

Example Package Manifest

This is an example of a file that would be located in the main directory of the package, defining what it is and what dependencies it has:

{
  "name": "packager#package0",
  "version": "1.0.0",
  "description": "Description",
  "icon": "http://repo.com/icons/package.png",
  "author": "packager",
  "license": "MIT",
  "dependencies": {
    "package0": ">=2.3.1",
    "package1": "https://github.com/packager/package1.git",
    "package2": "https://github.com/packager/package1.git?path=src/package/subdirectory",
    "package3": "https://github.com/packager/package1.git###24abc02",
    "package4": "Directory/Inside/Your/Project",
    "package5": "C:/Package/On/Local/Filesystem",
    "package6": "Nuget.package:4.3.0",
    "package7": "#CustomScope:4.2.1"
  },
  "packagerepositories": {
    "reponame": "https://packagerepo.com/allpackages.json",
    "anotherreponame": "https://anotherrepo.com/packages/allpackages.json"
  },
  "redotversion": "<4.4.0"
}

Official and Unofficial Repositories

If only the name and version are provided but no link:

The manager tries to resolve the package using custom scopes, if available.
The manager looks in the official repository (if such a repository exists).
The manager searches user-provided repositories.
The manager reads the `packagerepositories` field and prompts the user to add it to the project (a warning should be given since this could be risky).

Git Dependencies

This would be convenient because package creators wouldn’t need to host their packages separately. Instead, they could simply place the manifest file in a Git repository, which would be ideal for internal projects. This will require a `git.exe` file. The best solution would be for the editor to automatically detect if this file exists on the system. If not, the editor should display a prompt asking the user to provide a path if the project has Git dependencies.

NuGet

For C# users, many dependencies can be found on NuGet. It would be convenient for package creators to add dependencies from there something neither Godot nor Unity has managed to deliver. Of course optional if you don't want to use C#

Local Packages

This would involve using folders containing files. As long as they contain a manifest, they can be used as packages. This would be especially useful for package creators.

Custom Scopes

Instead of providing direct links to repositories, custom scopes could be declared in a separate project file. This would be useful when switching between the repository version and a local working copy, for example.

Credential Manager

One of Unity’s failures was not delivering a proper credential manager, which could be a significant selling point for Redot. A simple window where users can add repository links, login details, and passwords to access private repositories would be valuable, particularly for companies building their own ecosystems and for package creators who want to sell access to their repositories.

Inside a Redot Project

It might be worth adding a special path alongside `res://` and `user://`—for example, `packages://`.
Examples:

Plain: `packages://packager#package0/packagefile`
Specific version: `packages://packager#package0/%%1.1.3/packagefile`
Default version: `packages://packager#package0/%%current/packagefile`

Steps to reproduce

-

Minimal reproduction project (MRP)

-

thesayyn commented 2 days ago

FWIW, OCI Distribution spec is a perfect tool for this job, simply what homebrew uses to distribute all prebuilt artifacts.

You get free hosting if you use;

Github Container Registry
Dockerhub
Any other OCI compilant really

addmix commented 20 hours ago

What would be the benefit of a package manager, as opposed to only adding dependency checking, and allowing a package manager to be a separate module/program? It seems to me that a decent amount of end-user serviceability would be lost if there was a package manager possibly interfering with project files.

Maybe this could be factored in as an extension of the existing asset library functionality?

KAW0 commented 4 hours ago

I'm currently considering focusing on loading resources locally and from GIT repositories. @thesayyn As far as I know OCI is mainly designed for layering files and they do not have a system for resolving dependencies. @addmix The current AssetLib explorer is mainly file fetching, it does not include functions such as authentication, dependency resolution and places files in the resources folder so there is nothing that can be used here. I'm considering it as a module. A separate program would force the user to install something. This goes against the compactness of the engine. My idea is to separate packages from resource files. The package data would be physically in the .godot subfolder (the user would probably also be able to indicate another location) but to the user it would be visible from the FileSystem dock pic rel. Files in packages would be mostly readonly with additional support for changing them to working copies for package creators.

Redot-Engine / redot-engine