[Draft] Package Manager

KAW0 commented 1 month ago

Tested versions

-

System information

-

Issue description

TLDR: Something like Unity Package Manager, but works properly.

Why?

Marketing potential: This would be a great marketing mechanism for the engine. If users browse libraries for Godot and see in the description that the library has support for Redot Package Manager, they will naturally ask themselves what Redot is.
Competitive advantage: Other engines have not delivered a fully functioning package manager in Godot. This would be a significant selling point for Redot.
Team collaboration: It simplifies working in large teams with multiple projects. Instead of developing them separately, you can assign a team to create a common codebase.
Modularity: It enhances modularity, e.g., someone could create a visual scripting library, and another developer could build a dialogue creation library on top of it, while both are compatible with a library for in-game events from yet another package.
Efficiency: It eliminates the need to manually install multiple packages. Instead, you can create a meta package that depends on several frequently used packages and install them all at once.

Why wasn’t it included in Godot?

This has already been proposed, but the creator of Godot was concerned about a situation similar to npm for NodeJS, where a single package can have hundreds of dependencies, and a mistake or deliberate sabotage in one dependency could disrupt the entire ecosystem. This is a valid concern, but there are good examples like the NuGet library for C#, which largely avoids this issue. Most packages have single-level dependencies, and many have none at all.

The reason is simple: when NodeJS was initially developed, it lacked a comprehensive standard library, so packages were created for even the simplest functions, often resulting in duplication. In contrast, C# has a rich standard library, allowing packages to be developed for specific tasks, with many of them compatible with each other through common interfaces. I believe the situation with Godot/Redot is similar the engine already has a robust codebase, so the risk of excessive fragmentation should be minimal.

Technology Options

Creating Our Own Solution in C++

This would likely be the best solution, but it would also involve the most work. Self-explanatory.

Existing C++ Solutions

There are already many package managers for C++, but I am unfamiliar with them, so I cannot comment in detail. This could be the best solution if we find one that meets our requirements.
Here is a roundup of C++ package managers.

NuGet

There is an existing solution based on NuGet, making integration easier. It supports passive delivery, meaning you don't necessarily need a dedicated server. Simply host the files on a server. It’s also compatible with C# packages out of the box. However, the libraries for NuGet are written in C#, so this would require permanent integration of .NET with the engine, which could be a significant drawback for some.

Reimplementing NuGet

Instead of using C# libraries, we could focus on the NuGet standard and implement it in C++. Whether this is better or worse than creating our own system is uncertain.

npm

Please don’t use npm as a base. Let’s don't repeat Unity’s mistake. This system is slow, and its application would take up as much space as the engine itself. Additionally, the authorization system didn’t work properly in Unity, although it’s unclear if that was due to npm or Unity. Using package managers written in other languages also complicates the engine-building process.

Example Package Manifest

This is an example of a file that would be located in the main directory of the package, defining what it is and what dependencies it has:

{
  "name": "packager#package0",
  "version": "1.0.0",
  "description": "Description",
  "icon": "http://repo.com/icons/package.png",
  "author": "packager",
  "license": "MIT",
  "dependencies": {
    "package0": ">=2.3.1",
    "package1": "https://github.com/packager/package1.git",
    "package2": "https://github.com/packager/package1.git?path=src/package/subdirectory",
    "package3": "https://github.com/packager/package1.git###24abc02",
    "package4": "Directory/Inside/Your/Project",
    "package5": "C:/Package/On/Local/Filesystem",
    "package6": "Nuget.package:4.3.0",
    "package7": "#CustomScope:4.2.1"
  },
  "packagerepositories": {
    "reponame": "https://packagerepo.com/allpackages.json",
    "anotherreponame": "https://anotherrepo.com/packages/allpackages.json"
  },
  "redotversion": "<4.4.0"
}

Official and Unofficial Repositories

If only the name and version are provided but no link:

The manager tries to resolve the package using custom scopes, if available.
The manager looks in the official repository (if such a repository exists).
The manager searches user-provided repositories.
The manager reads the `packagerepositories` field and prompts the user to add it to the project (a warning should be given since this could be risky).

Git Dependencies

This would be convenient because package creators wouldn’t need to host their packages separately. Instead, they could simply place the manifest file in a Git repository, which would be ideal for internal projects. This will require a `git.exe` file. The best solution would be for the editor to automatically detect if this file exists on the system. If not, the editor should display a prompt asking the user to provide a path if the project has Git dependencies.

NuGet

For C# users, many dependencies can be found on NuGet. It would be convenient for package creators to add dependencies from there something neither Godot nor Unity has managed to deliver. Of course optional if you don't want to use C#

Local Packages

This would involve using folders containing files. As long as they contain a manifest, they can be used as packages. This would be especially useful for package creators.

Custom Scopes

Instead of providing direct links to repositories, custom scopes could be declared in a separate project file. This would be useful when switching between the repository version and a local working copy, for example.

Credential Manager

One of Unity’s failures was not delivering a proper credential manager, which could be a significant selling point for Redot. A simple window where users can add repository links, login details, and passwords to access private repositories would be valuable, particularly for companies building their own ecosystems and for package creators who want to sell access to their repositories.

Inside a Redot Project

It might be worth adding a special path alongside `res://` and `user://`—for example, `packages://`.
Examples:

Plain: `packages://packager#package0/packagefile`
Specific version: `packages://packager#package0/%%1.1.3/packagefile`
Default version: `packages://packager#package0/%%current/packagefile`

Steps to reproduce

-

Minimal reproduction project (MRP)

-

thesayyn commented 1 month ago

FWIW, OCI Distribution spec is a perfect tool for this job, simply what homebrew uses to distribute all prebuilt artifacts.

You get free hosting if you use;

Github Container Registry
Dockerhub
Any other OCI compilant really

addmix commented 1 month ago

What would be the benefit of a package manager, as opposed to only adding dependency checking, and allowing a package manager to be a separate module/program? It seems to me that a decent amount of end-user serviceability would be lost if there was a package manager possibly interfering with project files.

Maybe this could be factored in as an extension of the existing asset library functionality?

KAW0 commented 1 month ago

I'm currently considering focusing on loading resources locally and from GIT repositories. @thesayyn As far as I know OCI is mainly designed for layering files and they do not have a system for resolving dependencies. @addmix The current AssetLib explorer is mainly file fetching, it does not include functions such as authentication, dependency resolution and places files in the resources folder so there is nothing that can be used here. I'm considering it as a module. A separate program would force the user to install something. This goes against the compactness of the engine. My idea is to separate packages from resource files. The package data would be physically in the .godot subfolder (the user would probably also be able to indicate another location) but to the user it would be visible from the FileSystem dock pic rel. Files in packages would be mostly readonly with additional support for changing them to working copies for package creators.

terefang commented 1 month ago

bad question:

why not leverage on some standard c-libraries like LibArchive and LibOpkg ?

https://git.yoctoproject.org/opkg/tree/libopkg/opkg.h

KAW0 commented 1 month ago

@terefang I didn't know these libraries, but they won't be useful to us here. LibArchive is a library for compression and archiving, which is beyond its scope. The engine already includes libraries for these purposes. LibOpkg has an incompatible license.

terefang commented 1 month ago

you are looking for MIT or 0BSD license ?

KAW0 commented 1 month ago

you are looking for MIT or 0BSD license ?

As far I know everything that allow closing source should work if Redot want (i sure they do) to keep upstream policy.

nkarl commented 1 month ago

I see that NixOS is in the list of C++ implementations, but I'd like to recommend Nix over NixOS. They are not the same thing, and Nix (and recently flakes) are way more versatile than NixOS.

You can read my exposition at my blog here.

Frontrider commented 1 month ago

I'd ask the following question before deciding on a format: what is it that we want to be in a module?

What I think is:

If it's a native extension, then it may not even have any dependencies whatsoever, just a single pre-compiled binary.
If it's a native extension with a dependency on another native extension (interfaces via the class registry for example), then that must be handled on this side. I expect this to be extremely rare.
if it's a gdscript addon, then this system pretty much has to act as the package manager for GDScript. This also goes for you if the addon has scenes.
if it contains an external language with it's own package manager (eg.: nuget), then I think the dependent code should come from that package manager, because whatever tools you'll use with that will understand it better. (a c# addon should pull in dependent code via that package manager). This may not need to be handled at all, if the addon itself just has the correct manifests. (eg package json, no idea what c# uses because I gave up when I tried, requirements.txt for python, pom file for jvm etc)
If it's an external language, but the dependency is another gdextension via the class registry then it should be handled here. (reasonably this won't ever go cross language, the dependent extension can be expected to be native c/c++/rust/zig etc)

I do not believe that we'll have very deep dependency trees with compiled addons, especially as the engine has a very good stdlib.

NOT pivoting to a specific language other than native code via GDExtension is a very powerful idea/tool for the engine.

KAW0 commented 1 month ago

@Frontrider This project is only about the method of delivering files. If Redot somehow supports loading modules from files, then this project will support delivering these files, but loading them is out of scope. My idea for delivering nuget packages is exactly as you write. Godot/Redot already supports nuget packages indirectly in such a way that by adding the package name and version to the .csproj file, msbiuld downloads the appropriate package. The problem is that if the Assetlib package contains a dependency on nuget, Godot/Redot will not automatically add it, which I intend to introduce here. It would be only one-sided. I don't see a situation where a package from nuget would depend on a package for Redot. I also plan to add dependencies on modules inside the engine. For example, a package related to navigation should require the engine to have a navigation module. As for external languages, the best solution would probably be to add an API

Frontrider commented 1 month ago

As for external languages, the best solution would probably be to add an API

Honestly, if I go off on how the c#/jvm modules are planned to work in the future just setting a dependency on the runtime is enough. (which becomes an addon/module depending on a second one)

The developer should be able to understand the rest from there. Does cause some issues with javascript (yarn+npm) and jvm (maven/gradle/ivy), but those should just be solved on that runtime's side.

I also plan to add dependencies on modules inside the engine. For example, a package related to navigation should require the engine to have a navigation module.

That actually does sound like something that Juan wanted but never happened.

KAW0 commented 1 month ago

According to what I saw on https://proposals.redotengine.org/ this is currently the most requested feature. I'm not announcing anything, but I just want to say: I'm cooking

SkogiB commented 1 month ago

According to what I saw on https://proposals.redotengine.org/ this is currently the most requested feature. I'm not announcing anything, but I just want to say: I'm cooking

YAAAA BUDDY

michaldev commented 3 weeks ago

In my opinion, Flutter has a great package manager, and the package format is okay.

name: GameName
environment:
  sdk: '>=3.2.6 <4.0.0'

dependencies:
  flutter:
    sdk: flutter
  equatable: any
  vibration: ^1.8.4

dev_dependencies:
  flutter_test:
    sdk: flutter
  flutter_lints: ^2.0.0

Frontrider commented 3 weeks ago

In my opinion, Flutter has a great package manager, and the package format is okay.

name: GameName
environment:
  sdk: '>=3.2.6 <4.0.0'

dependencies:
  flutter:
    sdk: flutter
  equatable: any
  vibration: ^1.8.4

dev_dependencies:
  flutter_test:
    sdk: flutter
  flutter_lints: ^2.0.0

The dev dependencies is an interesting one, as that is effectively defining something as "editor only". But, that also touches the way the final package is assembled, you have to exclude those and it takes some work to get the engine to do that (you can filter folders in an export plugin). There is also a third case it explicitly does not cover. A plugin with a new node+it's editing tools. That can't be made out of 2 modules you need to install separately.

Also, I'd avoid nodejs like version "modifiers" as in my experience those cause more issues then their worth. Keep it simple and predictable, it will be used by less experienced people and not having it does not actively hurt the more experienced ones. I'd also integrate it into the project file+ the editor's gui instead of a text file.

tindrew commented 2 weeks ago

According to what I saw on https://proposals.redotengine.org/ this is currently the most requested feature. I'm not announcing anything, but I just want to say: I'm cooking

Hey just wanted to touch base, is this still in the works?

KAW0 commented 2 weeks ago

@tindrew It is. I will DM you on Discord

Redot-Engine / redot-proposals