New Add-On (sub-project) system

[reduz]

After discussing with @bojidar-bg I think we reached some general idea on how the add-on system could be improved. Currently, the add-on system has many limitations:

• Requieres unique and absolute paths
• Makes it difficult to test self-contained
• Causes problems with duplicated information in files such as project.godot, README.md, .gitignore,etc.

Most of the feedback for the current system centers on that it should be great if it could be possible to:

• Just write the add-on as a godot project
• Do not include the root directory
• Be able to install it anywhere in the project and likewise remove it
• Be able to simply check out an addon using git to a sub-directory of an existing project and keep it up to date this way

So, we've been discussing this for a while and reached the following proposal, which I believe allows to implement this in the simplest way.

Writing an add-on will work as follows. It will not be an add-on in itself, but a sub-project:

1. Create a new project, this will be the add-on.
2. Make the add-on as if it was a project in itself.
3. Resources (And this includes scripts) will have a new function named get_project_path(). If you are writing a script in C#/GDNative you can query for the base path. When running the project as sub-project, it will be res://. When running in the game it will be the path to the subproject.
4. For GDScript this process can be transparent, so nothing needs to be done, scripts will just work. Just use res:// normally with extends, load and preload.
5. In GDScript, in the case your scripts needs to access a resource from outside the subproject path, then just use ResourceLoader.load() manually.

Unde the hood, sub-projects can go anywhere within the main project and it will work the folowing way:

1. From the editor, sub-projects will probably be scanned automatically, but you will have to toggle them on/off yourself similar to now. This ensures that all sub-projects go to a list.
2. This list is passed to ResourceLoader. Resourceloader can easily detect when a resource being loaded is sub-project. This information will be passed to each loader, and loaders will be able to remap res:// of it's sub-resources to the full path. Most likely, only resource loader/save binary/text need this addition, the other loaders don't support subresources anyway so they are fine as-is.
3. Settings added in project.godot of a subproject will be editable in ProjectSettings for the project. If the setting is edited, though, it will be re-owned to the main project and the setting in the sub-project will no longer be used (in other words, main project.godot always has priority over sub ones, this makes sense as you may want to modify a sub-project setting, change input mappings definied in a sub-project, etc.).
4. Some settings will adapt fine to this (such as input), but others we need to change how they are saved, so they use individual properties. This is mainly the case of translations and translation remaps, which go into a single array now and will need to be converted to individual properties.
5. Settings that contain paths, will be converted automatically by brute force when loading a sub-project.godot. If a strings begins with "res://" it will be assumed to be a path and will be converted.

So this is pretty much it. This approach is the simplest we could come up with, should require little work from the codebase, and should make much, much simpler the process of creating addons (which are simply sub-projects now). Feedback welcome!

[reduz]

Modified the text above, renamed addons to sub-projects, which makes more sense.

[Zylann]

I am writing my plugin along with 3 other ones that extend my workflow:

• HTerrain plugin
• Demo for the terrain plugin (separated to keep the core plugin clean)
• Channel packer plugin (soon to be made a separate plugin. Not a direct dependency, but helps a lot)
• Project statistics (I use it locally for my own fun)

Each are in a separate git repo, but I copy them in the HTerrain project because I need them for testing a real environment. I don't do the opposite because those other plugins change way less frequently. What I do at the moment is, I write my plugin normally, but I gitignore all the others. Having them in the project allows me to test it waaay more conveniently (I can't afford recreating a whole terrain everytime I test). Also, I may still use an addons folder to install plugins anyways because I find it cleaner.

• What would the new system imply for this workflow?
• How would plugins detect their extensions, if any?

[toger5]

Questions:

• this one is dumb, but addons are equivalent to plugins, right?
• will it be possible to also have a global Addon path shared by all godot projects on a system. (to put editor oriented addons like the sketch fab implementation, or a clock in the editor for example) Right now I find myself to install a couple of addons all the time and add them to the gitignore which is just ... cumbersome.

Thoughts:

• I'm not sure about having to decide if you want to create an Addon or a project when creating the project. I really love the current option to just create simple addons/plugins on the fly when working on a godot project. I also really like it when there is no additional Gui to setup a project like a lot of IDE's do. I prefer if the IDE design is flexible enough to make these decision on the go. Like godot does at the moment.
• I think it would be create if right click in the fileEditor would give you the option to create an godot.addon file. than you immediately can work on it. Also the project.godot file is kind of important, since it gives you also the option to enable/disable other addons. Which can again be very useful when creating addons. (maybe I want to use the git Gui when working on a addon)

[henriiquecampos]

Sorry if this may sound "dumb" but I'm very inclined to ask: can this be used for, as @Zylann seems to be, somekind of dependency system? E.g.:

I'm currently creating a Platform Template for my future projects, and I'm likewise making updates in the Actor Plugin from many other projects, currently I have to update the Platform Actor Plugin manually and likewise updating all projects that shares this dependency. And there are many other "modules" for animations, particle packs etc...

Can we break the dependencies into git projects and "patch"(?) every other project to use the up to date version while keeping the project.godot override the settings from the sub-projects/modules?

[swarnimarun]

Awesome idea, I was having some trouble with how to maintain an asset. If it's also a project but this solves everything.

@toger5 I don't think that we will have to select anything. Just create as if it was a project. No need for creating addon file i believe. Please correct me if I am wrong.

I want to see a way to keep tabs on the plugin.cfg file. I really tend to forget about it. Probably it should be editable and visible in editor. Probably a template plugin can be generated too something to start of having basic features.

And as the other guys said will there be support for a dependency system?

[Zylann]

@swarnimarun plugin.cfg is needed to define plugin-specific script stuff though, and is currently the only thing that allows to differentiate an "end-project" and a plugin.

[swarnimarun]

@Zylann Atleast it can be visible and editable in Editor.

[LeonardMeagher2]

Does your main scene now have to extend an editor plugin?

[Zylann]

@LeonardMeagher2 plugins don't need to be a scene, so I'd say no

[LeonardMeagher2]

@Zylann I guess I'm confused, what's the difference between an add-on and a plugin? If they got rid of the "plugin.cfg" it'd have to be possible to make a plugin in the editor also, and since add-ons do need a main scene I imagine your main scene for a plugin would need to be EditorPlugin.

I imagine if this helps organization or use, it'd be the method used for any add-on including plugins (which I'm not sure there is a difference)

[swarnimarun]

@LeonardMeagher2 They are not removing plugin.cfg. That was me being silly. And want to clarify this that I want a way to be able to check on my plugin.cfg removing them is not my intention.

And, addons and plugins are the same. If addons written with C++ are added via GDNative and there are considered different. And modules are just a different topic.

[Bauxitedev]

A dependency system would be really great. I'm running into the same issues as other people here, when I e.g. want to use my motion blur shader in one of my games, and I fix a bug in the shader, I have to go update it manually in all my games. Additionally, it seems addons cannot be easily put into another folder for organizational purposes.

Heck we could even have a full-blown package manager inside Godot. It would integrate really well with the asset library - since we already have a versioning system for assets. The only thing we would need is a dependency system.

Unity also has a package manager. You can scroll through packages and install them with a single click. When packages in your project can be updated, the package manager will show a notification and you can update them with a single click as well.

[reduz]

@Zylann, @Bauxitedev:

I suggest we do the following.

1) Let's keep the current system, and make it more like a package manager, if you select to install it it will go and install a package. If dependencies exist, it will install them first. We can check for updates every time the project is run. The limitations will be the usual, you can't choose where to install it (always project wide on the same place) and you can't have anything in the project root nor global settings. Or maybe you can but they are merged when installed. 2) Let's implement the proposed system above, these are assets, you can choose where you want to put them, and they will contain their own project.godot. Assets may depend on packages, but it will be up to you to publish the asset without the package (so the editor installs the package before downloading the asset) 3) When submitting to the asset library, you will need to pick whether you want to submit an asset, a package and specify the dependencies.

Would that work?

[Zireael07]

Having two systems in place (asset and package) seems confusing IMO.

[reduz]

@Zireael07 It's not really two systems imo. Think that the diversity of stuff you can download from the asset library is huge. Programmers may be more interested in doing a set of reusable classes, which need to be installed on a fixed place, while you may just want to download a pack of textures, shaders, materials, animations, etc. and put them on a specific directory in your project, while you expect them to keep working. The system should be as simple as possible, but also as flexible as possible to support most use cases.

[Zylann]

I'd just like to:

• Develop plugins by versionning only the addon part people can install: this would allow devs to have more plugins and test stuff installed on their local repo if they want, and they can use git submodules as well for them. If a plugin was developped like a project, that would mean I need to add the other plugins and assets to some folder which I gitignore. If addons are subprojects, that can make them standalone, but for a developer with a setup like mine it doesn't mean much because I already develop all my addons as part of a local parent project, for the sake of testing with other addons as I explained earlier.
• Install plugins under a common folder: it keeps a clean convention, separating what is yours and what is third-party, and makes it easier to detect optional dependencies by path. If people choose wherever they want to install anything, it would need a registry of some sort to ask where a plugin is from script using an ID. I understand it makes a bit less sense for demo projects though (as long as you don't intend to install them in your existing project! In which case the addons/ folder would be relevant).
• A dependency system should still work as long as Godot knows where plugins are. Assets can depend on plugins, but plugins can also depend on other plugins (optionally or not). If people can install things wherever they like, how would a plugin lookup things like a base class? How do you extends "thing.gd" if thing.gd is located in another plugin but you don't know where the user puts it? (same for inherited scenes). In my engine I was using identifiers instead, which looked like: extends "other_plugin:thing.gd", where the prefix is used to resolve the path to the location of another subproject.

[vnen]

AIUI there are 3 types of things provided by the asset library:

1. Templates, which are just regular projects that serves as a base for new projects.
2. Plugins, which are tools for the editor (including importers).
3. Assets, which are stuff that can be reused between projects (can be art or code).

For templates, the current system works well, because you just need to download it somewhere and start editing. You don't care about updates, because it's quite easy to break things or override the work you did. Dependencies on assets and plugins can still be managed by Godot like it would manage in any project.

Plugins can be put in a fixed location, because they are usually self contained. They might depend on another plugins, so we can either use the fixed path or make some special code for other plugins. The problem is one different plugins depend on different versions of the same plugin: should Godot install both or only the most recent? This is the most problematic part of a dependency-resolution system.

Assets can be put anywhere, so their path must not be absolute. If assets can depend on others, it's quite easy to become a mess. I can see assets as "sub-projects", but I'm not sure how paths could be properly resolved or how the many project.godot can be merged without conflicts.

[AlansCodeLog]

Okay, I'm liking the idea of plugins as projects, but a bit confused about the plugin vs asset distinction. If I understood correctly: Both will be project plugins? One will have a fixed install? (addons/?) like now but they won't be able to put anything in root? But the others won't, and they can be moved around by the user?

Regarding plugin versions, maybe plugins could specify a range their dependencies (plugin A requires ^0.0.1, and plugin B requires ^0.1.0 = 0.1.0 - to anything below 1.0.0 is installed), so only major version changes would cause a problem (plugin A requires ^0.0.1, and plugin B requires ^1.0.0), and in that case, could Godot support dependency nesting? Otherwise just throw an error/warning?

Edit: Also will regular projects be able to depend on plugins as well? It would be nice as I mentioned in godotengine/godot#18131 to have project templates as a workaround to the lack of a global plugin system where the project requires x plugins and they're automatically downloaded and installed if missing.

[mhilbrunner]

Maybe relevant: https://github.com/godotengine/godot/issues/17092

[BastiaanOlij]

Sounds really good, love the general approach.

One beef I had with the current system, which is why I have two repos for each GDNative project, doesn't seem solved by this so I'm just flagging it if anyone has any ideas. This may be limited to how a sub-project is made available on the asset library then the actual structure of the sub-project itself.

For GDNative projects you have the C(++) code that builds the module, and then the module itself which would be really nice to deploy as a sub-project. Obviously I can keep working the way I do now, having the C(++) code in a "source" repo, and the compiled version with all its support and configuration files in an "asset" repo on github.

My ideal scenario would be to have one repo but the root of the sub-project being a folder inside of that repo. Incidentally that is how my source repos currently are stored, with a folder called "Demo" containing the files for the asset that I simply copy over to the "asset" repo when deploying it.

[nobuyukinyuu]

There are many use cases I can think of where you'd want an extension to the editor to persist between projects. The API allows us to modify some global settings, so I wonder what taking this direction means for that notion. I would say that changing the name to subprojects makes it clear that these exist to extend the tools available to your project, and not be seen as integrated as an actual extension to the editor itself.

The current system is mainly only cumbersome to git repos in my view because in order to deploy it easily to end-users with the editor's AssetLib integration, your repo must have a structure which is not conducive to working with how most addons are actively developed (as extensions to an existing project). Multiple addons can't easily be in the same necessary folder structure while each having their own repo -- especially on Windows where symlinks aren't always obvious -- and the parent project needs to be put into .gitignore for the addon repo if you want to deploy directly from there to remote, which would also interfere with the parent project's repo, if it exists. I'm not understanding yet exactly how the new system addresses this, but I have a vague idea. It seems "okay".

But, I'm for having a fixed location option available. I'm sure sub-projects are great, but I'm also sure there are probably some improvements that could be made to the system that would put a lot of the same power into the hands of people a lot faster. The way it's structured now heavily implies that the majority of addons were expected to be designed for use in monolithic projects already, ironically. Just that they were ones which would've had limited use on an editor-wide scale. Addons in a fixed editor-wide location can cede priority to one installed local to a given project.

P.S. Git submodules aren't necessarily for everyone...

[reduz]

@nobuyukinyuu I agree, global editor extensions would be very useful. I wanted to wait until the system is more or less stable to work on that (or for someone to make a proposal). The problem though, is that many times what it seems like it should be global is not, as example a Tiled or PSD importer should be in the project, so it will work for other users cloning it.

@Zylann I am not really sure a package management system should be compatible with cloning git. If you manage packages, you generally need the list of files you installed to be able to update it, which is not the case of git, but I'm still sure we should be able to find a way to do a simple workaround for your use case.

@vnen Still, from this thread, it seems to be that package management is a lot more important than I thought it was, and it makes a lot more sense now

[API-Beast]

I don't think adding dependencies is a good idea. Addons should be self-contained if possible, and even in situations where it is not feasible it shouldn't be encouraged. Else you end up with situations like NPM where each package depends on at least 10 other packages going into hundreds of packages in some cases.

This can get out of hand really quickly.

That said it should be possible to have installed addons interact with each other, maybe just a few functions for querying what other addons are active and what their base path is?

[Zylann]

@API-Beast how do I solve this in a user-friendly way, then https://github.com/Zylann/godot_hterrain_demo/issues/1 The demo is 60 Mo or more in the future, I don't want to force people to download and get it when installing my plugin (and I don't want it in my Git repo either, but that's Git-specific). Also, while it sounds possible to have plugin extensions without support from the engine, that's additional work for all plugin maintainers. The NPM case sounds like it's an issue about how devs make packages with it, not the feature itself.

[API-Beast]

@Zylann You could bootstrap a script file that checks for dependencies and provides the user with the option to install the dependencies for them. It's not functionality that needs to be implemented in the engine itself. If necessary some API functions could be added to simplify the process of writing such a script.

[Zylann]

@API-Beast well yeah, that's exactly what I mean by additional work for all maintainers.

[toger5]

@reduz yea I agree that having local addons has the huge advantage when working in a team. and it is surprising how many things are needed for everyone working on the projects (your importer example is good!) It still would be great to have a global addon dir. Than there should be a button in the addon list which allows you to make it local, (copying the files inside the current project dir. so git can pick it up). so you first can activate/deactivate addons without installing them in the current project and for the ones you really need can be copied with one click. and a git commit!

[API-Beast]

After giving it a bit of thought I have come to the conclusion that this proposal is a bit too complex. Godot always had the advantage that it keeps things simple by working very close to the file system, changing how "res://" paths work would work against it.

I would instead suggest to:

1. Implement relative paths for sub resources. Any plugin would work just fine regardless of what directory it is in as long as all paths are relative. To ease the transition a few context menu entries could be added to convert all paths in a resource or scene from and to relative paths. Everybody should be familiar with relative paths, while only very few people understand changing roots.

2. Add a new resource type with the extension ".plugin", this would replace the old "plugin.cfg" files. Godot would scan all .plugin resources during the auto import step and add them to the list of available plugins. As such they no longer depend being placed in a specific sub directory and can be placed anywhere in the file tree. Using built-in scripts this would even allow for micro-plugins that are just a single file.

[toger5]

@API-Beast makes sense to me. But I still don't completely understand reduz's proposal so I might overlook something. Would you than propose something like: plugin://path/to/file to use the relative path? I think it would be coherent since res:// basically is nothing else than a placeholder to the project path, plugin that would be a placeholder to the plugin path.

this also would be nice so you could still use the res:// folder when writing your plugin. for example you could create a output folder inside res:// to store files generated by your plugin...

[Zylann]

@API-Beast the problem with relative paths is that if you move a .tscn somewhere else, if it references resources that are somewhere else in the plugin it will break anyways. Also, being tighted to death to the file system isn't that good of an advantage to me, I always thought the ultimate solution is godotengine/godot#15673, but that's not going any time soon...

Also I just thought about it, if I make a demo project depending on a plugin, there is no easy way to make Godot download the dependency using scripts, because the demo project doesn't contains a plugin script. It probably doesn't even contain code. So dependencies are something Godot should handle, at API level towards the assetlib, in order to decide what to download.

Having a res://-like pointing to a project's root still makes sense in sub-project context because the meaning of it doesn't really change, and a plugin cannot assume how the project it is installed in will look like. The confusion might come from how it gets used, for example if you really intend to access a path from the real root of the user's project, you are starting to make assumptions (it's possible the path you want doesn't exist). So in the context of sub-projects you would need something explicit like rootres:// to do that (but usage would be script only). Which is what Reduz proposed by using ResourceLoader directly. However, that would not work if you have, say, a TSCN in a plugin which extends another file from another sub-project, because there is no way to explicitely resolve the path. So to me, only a few things can make references work in a plugin-based environment: conventions, more complex path system, or GUIDs.

[AlansCodeLog]

I don't think adding dependencies is a good idea. Addons should be self-contained if possible, and even in situations where it is not feasible it shouldn't be encouraged. Else you end up with situations like NPM where each package depends on at least 10 other packages going into hundreds of packages in some cases.

That said it should be possible to have installed addons interact with each other, maybe just a few functions for querying what other addons are active and what their base path is?

Okay, but what happens when the addon you want to interact with is a different version than what you need?

The NPM case sounds like it's an issue about how devs make packages with it, not the feature itself.

I don't know... I think what causes huge node_modules is the fact one can be so modular and dependencies will inevitably clash if you're depending on someone else's work. And in those cases 99% of the time it's really nice that everything just works and npm automatically dedupes everything it can.

imo Godot encourages modularity and it's inevitable people will want to split their plugins up, so some sort of dependency manager is important, and if so it will have to deal with this in some way. You can try to encourage people to write plugins that don't depend on each other, but if it's allowed they inevitably will, and dependencies will inevitably clash.

[samvila]

@reduz One of the biggest issues I have with the actual way of Godot handling addons is the fact that there's no way (at least as far as I know) to have specific addons enabled by default. For example in Blender once you enable an addon it will stay enable for any session unless you manually disable the addon or remove it. This is very convenient when you are planning to use the same addon for every new project.

Example of how Blender handles the situation:

https://www.youtube.com/watch?v=0v9JOEaVJu4

[swarnimarun]

I think it would be better to have an addon manager with the sub-project system over a dependency system if they can't be implemented together(or are too troublesome) cause the sub-project system is probably going to solve all major problems with the current system other than dependencies.

And also dependencies add an unnecessary levels of complexity to the whole addon system which will only cause problems the more it grows up.

[Ranoller]

I feel dumb because i can´t understand this proposal... if plugin is a separate project, how do you test in your current project?... If plugin system is going to be reworked i would ask to make a workaround to the mandatory close/open project every time a plugin script with dock dependency is saved.

I´m agree with @toger5 ... i make plugins in my current project to project-specific workflows, so i like the fact that i can edit project-scripts / plugin-scripts and project-scenes / plugin-scenes in the same project... Can we maintain this "one editor instance" edit?

Other think i want to ask is to test extensively performance in new system. For now, gdscript in parse/manipulate text files in project (ej, searching strings, converting to StringArrays, etc...) have a semi-decent performance but very variable (up to 8 seconds in a plugin that counts lines of code, size of files, make a tree with ordered results and allow to open directly resources, etc....for a project with thousand files...). I can work with this performance (that requires heavy "inlined" gdscript and a lot of micro-optimizations like don´t use functions, constants or api calls in sub-for/while loops (not allways feasible), don´t use line-comments, etc...). If new system doesn´t add new internal calls to the compiled script that kik down performance it will be fine, because for now, to certains operations, gdscript performance is not downgradable, or the utility becomes unuseful (a.k.a -> you can´t wait 60 seconds in a entire project - text parse)

[AlansCodeLog]

I feel dumb because i can´t understand this proposal... if plugin is a separate project, how do you test in your current project?...

No, I'm a bit confused too and I was also wondering about this. I'm assuming you'll see them? Right now if you nest a project inside a project Godot just ignores that entire folder, I assume it's blacklisted somewhere. It would also be nice to be able to click a project.godot/addon.godot and have it launch a separate instance (would be super useful for demo projects).

[MarianoGnu]

I was thinking, whou could some properties be shared beatween main project and Addon? for example when you make the Addon project (asuming it could be used elsewhere) you dont know the final game resolution will be, so making a GUI that fits the final resolution could be difficult, or maybe your addon project expects to have defined specific inputs, when the scriptso n the addon is executed it read's the addon's input map or the main project's? in the first case then you would need something like SubProjectSettings.get("subProjectName","key") for global properties, configurations and presets

[PaulHMason]

Just something to add (it might have already been mentioned, but I missed it) relating to adding custom nodes via an addon. If you create, say, a custom button (MyButton) that extends button, when you use the custom button node, the script for MyButton is attached to the node instance, which isn't very useful because you expose the custom functionality implementation. It would be better to not do that and allow the user to attach a script the usual way and extend MyButton (not have the existing script that extends Button attached by default).

Edit: I see there's some other discussion and @reduz is implementing something to address this, possibly in 3.2. Also, there is a workaround using script inheritance (a bit clunky, but it works).

Incidentally, my interest in this is to use Godot for more complex UI development (not games) and provide a reusable UI widget library.

[lvottero]

What I would like to see is just a way to link a tscn file across multiple projects. For example a player camera can evolve outside different tests. There's many assets that could evolve outside many different test projects. It would be nice to have one directory to store many different assets and then be able to instance them as scene children. Right now as far as I can tell this visually can only be done with scenes that exist in the project. Copying a scene to every project defeats a little the point of instancing (or referencing) I'm coming from a Maya background. Its nice being able to have files anywhere on the file system and be able to reference them into the Maya scene. It makes it easy to setup a common repo for reusable assets.

[toger5]

@lvottero I think the idea that a project folder contained everything needed to work on the project is a well suited system for a game engine. In a asset creation program like Maya it's a different story. But for a game engine you want one folder/git project to be executable on every machine. (you should not need to redirect some linked files to different directories.) That is also the problem when moving tscn files from one project to another... There always are so many broken deps. The workflow for that should definitely improve.

[MarianoGnu]

I belive something like this could be resolved using a Resorce wich loads a actual resource from the filesystem, and copies it on export time. Of course dependencies can easily break, but that's out of scope. But another way to face the problem is allow mounting virtual folders on the project pointing to github repositories (or maybe any static location on the interter using http/ftp?) and download files by demand for the editor and download the whole folder on export time to be included on the pkg. the only restrictions here would be:

1. You need internet access (unless the files are catched)
2. Files are ReadOnly, wich suposedly doesn't matter for plugins, but sometimes you want to adapt stuff to your project

[willnationsdev]

@MarianoGnu

I belive something like this could be resolved using a Resorce wich loads a actual resource from the filesystem, and copies it on export time.

This already happens with a normal resource when a game is exported though. When you export, the project folder's files/resources are bundled into the .pkg compressed file that is shipped with the engine executable. The binary copy of the resources are contained in there.

mounting virtual folders...pointing to github...download files by demand for the editor and download the whole folder on export time

I'm not sure what you mean here. The current system downloads the files on demand for the editor. And then those files (whether you've edited them or not) are in the project folder and are thus included at export time in the final .pkg file. And if GitHub exposes some sort of API to mirror a repository onto one's computer in real-time (like, auto git-pulling on a read-only folder or something??), then yeah, you could do that, but that seems like a waste of time when someone can just actually use git and manually manage things. Plus it's more restrictive than necessary (read-only?) and there are other potential sources besides GitHub.

You need internet access

We want to avoid situations where the editor doesn't function properly without Internet access.

Files are ReadOnly, wich suposedly doesn't matter for plugins, but sometimes you want to adapt stuff to your project

I don't think anyone is suggesting that plugins be read-only. More like people have suggested that if a plugin's files have been edited, then the editor should record that. If someone were to then do an "auto-update-plugins" operation, any package manager shouldn't auto-update those plugin directories as they've been changed from their source. I think that was mentioned at some point (though nothing's been decided yet).

Of course dependencies can easily break, but that's out of scope.

The point being debated is that the breaking of dependencies is not out of the scope of this Issue's concerns. We have to decide whether to incorporate a package manager for handling Godot's asset library and plugin ecosystem and, if so, how it should manage dependencies.

[MarianoGnu]

This already happens with a normal resource when a game is exported though. When you export, the project folder's files/resources are bundled into the .pkg compressed file that is shipped with the engine executable. The binary copy of the resources are contained in there.

yes, the point was that these files in particular are outside of the project folder

[samdze]

I was thinking about this whole proposal and thought how this could integrate with the new mode of registering custom types. (with class_name inside scripts)

I'd completely eliminate the old add_custom_type() and remove_custom_type() functionality and only allow the new mode of types registration.

Plugins could be created this way:

1. Create a new project or open an existing one.
2. Create your classes, resources and file you wish to integrate in your plugin.
3. Create an EditorPlugin script that will handle enabling/disabling etc.
4. Use a new export plugin window, very much like an export project window, in which you export your project as a plugin, you set the plugin name, description, choose the EditorPlugin script, and which resources to include. You could even create and save several presets to export different plugins within a single project more conveniently and you could also export plugins already imported as plugins.
5. The exported folder would include the plugin with all the chosen assets and a plugin.godot (or something like godot.plugin, whathever) that will include all the custom types defined in it, together with other settings. Godot will read it on import and recognize that the custom types included in the folder are to be enabled only on plugin activation (because they are listed inside the plugin.godot) and even dynamically add a namespace to them based on the plugin name. (or better, require it mandatorily on plugin export)

In the Plugins tab in Project Settings you could then see the enabled plugins and the custom types they define, together with (other feature) individual specific plugin settings?

Plugins would became narrowed down versions of Godot projects (as they don't need a lot of the settings of a normal Godot project), completely self contained and with automatic custom types registration defined previously with the new Script Classes system.

Regarding dependencies, one solution I though that combines with the proposal above could be this:

1. Dependencies are set in plugin export presets in a simulated/real environment. Remember, a Godot project can contain several exportable/imported plugins, even ones downloaded from the AssetLib. (in this case they could include data required to redownload them via AssetLib in their godot.plugin?) This is a matter of selecting the plugins already imported into the current project that you want to add as dependencies, with version numbers, etc. Then the exported plugin will have those dipendencies inside its godot.plugin with the data required to download them from the AssetLib. Simple and clean.
2. An imported plugin which defines dependencies could show a warning inside the Plugins tab in Project Settings. There the user can manually trigger the download of the required dependencies. (this could be a feature of the specific plugin settings mentioned above)
3. Based on Project Setting configuration, imported plugins could also download all their desired dependencies automatically.

[Jeremi360]

I found this when I just start to work on GodotPackageManager addon :/

[eon-s]

@jebedaia360 your addon can be a base to continue this discussion and for whatever decision is made about this stalled proposal (which may be closed soon).

[aaronfranke]

I think it would also help to have asset library assets extract to a preset path, see #554

[Vivraan]

It would really help if symbolic links in Windows are detected properly.

[me2beats]

I would keep the current addon system, but add a dependency system. A dependency could be any addon (asset / plugin) that meets the following requirements:

• all addon files are in the addon folder (if I understand correctly, this is called self-contained)
• it is forbidden to have multiple plugins in one addon
• the addon has version information, as well as a unique identifier by which it can be downloaded from Asset Lib.
• it is possible that the addon also has info about the platforms it can be installed on and the supported versions of Godot.
• addon has a list of dependencies

All this info could be in a config file. When installing addon X, Godot

1. Checks first if the requirements are met
2. then reads the config file
3. makes sure that all listed dependencies are installed (and activated if the dependency is a plugin), or installs and activates them.
4. installs and activates addon X.

This would

• resolve dependency issues (hopefully)
• in the future, allow to create a manager of installed add-ons, which will let one to update add-ons (manually/automatically), remove them, view their dependencies, etc.

As for subprojects, I think this is an interesting idea and I also think it would be wise to implement this as an extra feature at first time, since there are always hidden pitfalls, while the current addon system is generally working and tested.

Also, there seems to be a lack of examples of the structure of the subproject and its use, API (at least in the form of pseudocode).

[Xrayez]

I feel dumb because i can´t understand this proposal... if plugin is a separate project, how do you test in your current project?...

No, I'm a bit confused too and I was also wondering about this. I'm assuming you'll see them? Right now if you nest a project inside a project Godot just ignores that entire folder, I assume it's blacklisted somewhere. It would also be nice to be able to click a project.godot/addon.godot and have it launch a separate instance (would be super useful for demo projects).

The directories which contain other (nested) Godot projects are skipped during filesystem scan entirely, the behavior is the same as for having .gdignore within a particular folder.

Here's a patch which allows you to display nested projects:

diff --git a/editor/editor_file_system.cpp b/editor/editor_file_system.cpp
index 78fd88e50..627c4ea7d 100644
--- a/editor/editor_file_system.cpp
+++ b/editor/editor_file_system.cpp
@@ -684,8 +684,6 @@ void EditorFileSystem::_scan_new_dir(EditorFileSystemDirectory *p_dir, DirAccess
            if (f.begins_with(".")) // Ignore special and . / ..
                continue;

-           if (FileAccess::exists(cd.plus_file(f).plus_file("project.godot"))) // skip if another project inside this
-               continue;
            if (FileAccess::exists(cd.plus_file(f).plus_file(".gdignore"))) // skip if another project inside this
                continue;

I also think that's the first step to start implementing this proposal... 🙂

As for me, I've recently stumbled upon limitation of maintaining self-contained parts of a game project. I've started to create various "sandbox" projects just because it allows me to better concentrate on features which are relevant at the moment, and it's much quicker to launch Godot (3-6 seconds compared to 10-20 seconds...) Then, I'd just copy and paste a working implementation to a main project once completed, but then yeah the only thing which prevents this kind of workflow is exactly having to update resource paths within scenes manually.

Something tells me that this part of the problem should be straightforward to resolve (if we put plugins/addons concerns aside, of course).

[Xrayez]

Implementation for subproject paths resolution

I've come up with proof-of-concept implementation for resolving subproject paths in my fork: https://github.com/godotengine/godot/compare/3.2...Xrayez:subproject-paths (for Godot 3.2, can also make a draft PR for discussion purposes and easier review).

The way it's done:

1. Scan the editor file system and seek project.godot (see above post). Base paths are passed to ResourceLoader with add_subproject_path() method.
2. The list is also saved to ProjectSettings, so it works outside the editor and once exported (just like having global classes written). This is where we can reuse addons functionality, but that's actually not what I needed myself so I decided to write a separate list of subprojects which are not necessarily addons (which also makes the feature backward-compatible with 3.2).
3. When attempting to load non-existing resource, ResourceLoader tries to load the best matched resource in one of the subprojects by remapping paths until the resource is found in one of the subprojects. Note that the process is different from path_remap because the task here is to "convience" the ResourceLoader into believing that this is the resource it's been looking for.
4. ResourceSaver is not touched at all. The only thing which I had to change is ResourceFormatSaverTextInstance::save so that the previously loaded resource from the subproject gets written relative to itself, so no diff should appear in the subproject. Seems to work with binary formats too.
5. Had to fix the way GDScript preloads resources to use ResourceLoader:exists() over FileAccess:exists(), because FileAccess is not aware about subprojects.

All resources, including those in subprojects, get imported into the main project, and can be referenced just like any other resource. Normally, you shouldn't try to use subproject local paths in the main project.

This approach seems to resolve a lot of limitations without much changes. But this universal solution has some caveats:

1. If two resources in separate subprojects have the same path, then this may lead to collision.
2. If the main project has a resource path equivalent to one in the subproject, load() will only return the resource from the main project, again leading to errors.

I think those issues can be resolved as denoted in the proposal text:

2. This list is passed to ResourceLoader. Resourceloader can easily detect when a resource being loaded is sub-project. This information will be passed to each loader, and loaders will be able to remap res:// of it's sub-resources to the full path. Most likely, only resource loader/save binary/text need this addition, the other loaders don't support subresources anyway so they are fine as-is.

All those caveats can be resolved if you make sure that you have unique subproject paths to avoid those path collisions (just like people do with addons).

I've likely over-engineered something in the process, so feedback welcome! This is the first time I'm modifying different core parts of the engine at the same time...

Also, there are some issues with renaming/fixing dependencies, not sure how to solve yet, probably just have to skip subprojects.