godotengine / godot

Godot Engine – Multi-platform 2D and 3D game engine
https://godotengine.org
MIT License
91.35k stars 21.25k forks source link

Problems with preventing "engine bloat" and the AssetLibrary #19486

Closed karroffel closed 4 years ago

karroffel commented 6 years ago

Sorry for the very long post. I'm not mad if you don't want to read the whole thing, so here's a short version of it.

This is a mashup or many different issues. I didn't know where a comment like this would have been appropriate, so here it is in its own issue.

TL;DR

Using assets should feel like using core-functionality. Currently it doesn't. Proposed fixes:


Related issues:

19178

17092

15661

13187

10635

7402

6277

5947


Details here ⇓

Godot is amazingly powerful for the size of the binary. Unity and Unreal are bloated in comparison, but they do offer more QoL functionality out of the box. The situation with Godot is that there are some PRs that add good and useful new nodes which are rejected since they are "easy to code in GDScript and don't need to bloat the engine". This is a very valid and reasonable reaction - if Godot should stay a < 40 Mo download then adding new Nodes and features needs to be justified. While I do think this is a nice approach that ensures a pretty clean core, it has some problems. # Criticism and problem description ## 1. Functionality in core is official and undergoes quality control If there's an `InterpolatedCamera` Node in Godot, I assume that's the interpolated camera most people use, I assume that it's the most developed and "optimized for the engine" one since it's the one that's **part of the engine**. It has something *official* about it. The good thing about a "bloated engine" with many features is that those features are official and most likely the most widely used ones, more tutorials and documentation about them etc. They have been tested in many projects and there's a single place to discuss how to improve them. I, as a game developer, prefer official nodes over custom ones written by *Some Person*. Of course this approach has downsides, like huge download sizes, longer loading times and other nasty things. *BUT* I think in terms of user friendliness, *official* features are preferred over custom addons. Which one of the 4 `InterpolatedCamera` assets should I use? How do they differ? Is the documentation good enough? Where's the issue tracker? etc etc... ## 2. Downloaded assets from the AssetLibrary are just copy pasted *into* the project There is no difference between an Asset from the AssetLibrary and some code/scenes from the "main project". However, there is a difference between "official" Nodes and Nodes imported from the AssetLibrary. If I wanted to add some custom behavior to one of my `InterpolatedCamera`s, I could add a script on it and add my functionality - all cool and nice! If I wanted to add some custom behavior to one of my `AssetInterpolatedCamera`s, I could add a script on it and - oh. Attaching a script to a custom Node actually **replaces** the script. The very same script that is the reason I downloaded the asset. This is an editor problem *and* a core problem at the same time. Problems: 1. Script inheritance isn't *suggested* when attaching a script - override is the default for those cases. 2. Script inheritance doesn't work cross-language. A project that uses C# can't extend the camera with C#, while this would be possible with an official node. Fixes: 1. Make the editor fill in the path of the already existing script as the base type so script inheritance is the default. But this doens't solve the problem when you want to clear *your custom script for that one camera* as it would just remove all scripts entirely. 2. Make script inheritance part of the Script API. Currently the scripting languages all do script inheritance on their own - they don't know how to deal with other languages. My suggestion is to add a `set_base_script` that accepts *every* type of script. Usually the scripting language implementations just use the Script API on the base class anyway, so this could be generalized. The problems of "copy-pasting assets into the project gets amplified by the fact that all scripts are only defined by their path. In the camera case, I could do `var camera = InterpolatedCamera.new()` with an official Node, but I would basically have to do `var camera = preload("res://addons/com.name.interpolated_camera/scripts/interpolated_camera.gd").new()`. This is more than suboptimal. I know that reduz opinion is that paths are easier to use than class names, but I know for a fact that multiple users would prefer *an optional alternative system* that uses namespaces and class names instead. Especially in the context of downloaded assets, this would be a huge improvement. # Proposed solution Downloading assets and using them should feel like they are part of the engine. If they engine is kept small and meant to be extended with assets, that workflow has to be made as seamless as possible. The basic goal is to make using assets feel like using core-features. My proposed solution is a mix of a few existing discussions, but I think this is the place where all of them would shine together. ## 1. Make the AssetLibrary more package-manager like - Having a `dependencies.json` could be good enough. Using the AssetLibrary UI in the editor would just fill in the json, the download and installation happens at another time. An asset can itself have dependencies. - Assets are saved with version associated, all stored in `.local/godot/assets` or something like that. They are no longer part of the project itself but available system wide. - At export the needed files can be copied into the `.pck`. - Have a way to actually copy the files into the project in case modification at the source level is desired. Only then the files should be brought into the project folder itself. "Cloned" assets could live in a hidden folder like `.custom`. ## 2. Make using downloaded scripts easier - add a ScriptDB kind of system that can be used to reference classes without knowing the exact path - add support for cross-language script inheritance # Conclusion I know this is a very long post, but I hope to spark some more discussion on the AssetLibrary and also usability side of things. I think making using assets as smooth as possible is important. While things like proper versioning and allowing sub-projects will help, I think it's not enough to shift the focus to a more AssetLibrary focussed use of the engine.
toger5 commented 6 years ago

Really like your post.

EDIT: Damn it overlooked this paragraph

Have a way to actually copy the files into the project in case modification at the source level is desired. Only then the files should be brought into the project folder itself. "Cloned" assets could live in a hidden folder like .custom.

So you basically have covered the concerns I had. (sorry) I just go more in depth how I imagine the workflow + the behaviour.

POST: One workflow is overlooked IMO though. Using assets as just a resources or a first building stones. The current really transparent and simple approach also lets you just download a couple of assets. delete some files. Than keep some png's and a script which has some of the functionality you want in it. Currently it is possible to just work on them and modify.

If assets would be hidden in a .local/godot/assets folder this functionality would be kind of lost.

Proposal

Assets should still be visible and listed in the in editor file browser. Independent if they are inside the .local/godot/assets folder or the project folder itself. So you can browse the files and understand how it works in case there is no proper documentation, or if you want to learn from it. Every Folder which is 'an Asset' should also be marked as one with some kind of icon so you have an overview what is an asset and where the actual files are stored. There should also be a editor window which lists all assets. (in a table)

As soon as you want to edit/save/update a file from an Asset, you get prompted with two options

Save in .local/godot/assets

either update the asset in the .local/godot/assets which puts it into a custom user version. It the package-manager is git based, the custom user version could be one additional commit. which gets commit --amended whenever you save. (3.2.5/user for example) and is than available in all your other projects as well. (when updating the version in those other projects to you user version) why is this useful? (example) A theme from the asset store is almost perfect, but you hate the font type... change the font type save it as a new asset version. Revisit all your other projects and change the asset version to the /user. Now you are happy. Maybe there also could be one button to publish the user version of the asset. So in the asset store other ppl have the option to also choose a user version with minor tweaks. Just as an intermediate to make a pr to the asset git repo. (or the author of the asset deicides to cherry pick the changes form a /user version to a new version on master.)

Make local to project

copies the files from the .local/godot/assets folder into the current project... removes any special asset behaviour (like version...) and now you can mess with it however you want to. (also the folder loses it's custom appearance as an asset.)

This allows the asset store to also cover the use case of templates. And is a perfect choice for almost any art related asset. Since you can work on the assets to make them as you need them to be. You can always just activate a bunch of already downloaded assets (lets say some basic prototype sprites) make them local and you have them imported.

random thought

This might even be a nice solution for your own local asset/resource lib/collection. A folder in .local/godot/assets lets you add it's content to any Godot project with a nice editor ui and without searching for it in your OS file manager.

mhilbrunner commented 6 years ago

I agree that both the AssetLib as well as the asset lib integration (both in the editor and with GDScript) need love if we keep pushing it as an alternative to including things in the core engine.

However, I personally disagree with how heavily this should be a plugin is pushed.

The first question we ask ourselves on a new PR shouldn't be could this be a plugin? but rather is this going to be used enough to be in core?.

There are problems with having things in plugins:

Examples:

Edit: To be clear, I'm all for avoiding bloat. But this is a balance, and I want to voice support for not having everything in plugins. Very game specific or not broadly used stuff, proprietary plugins, big dependencies, yes please, put those on the Asset Lib!

dsaltares commented 6 years ago

@karroffel This is an excellent post and I mostly agree with everything you said. I would, however, like to comment on this point:

Assets are saved with version associated, all stored in .local/godot/assets or something like that. They are no longer part of the project itself but available system wide.

This is similar to how Python pip system works. People then realised that the latest version of a package screwed the other projects you have in your system. virtualenv solves that problem. In Javascript's npm ecosystem, modules are installed locally to a project by default and system wide is an opt-in feature (npm install --global). You then check in your package-lock.json to source control to make sure everyone in the team uses the same version of every npm module and avoid conflicts.

System wide packages are good as they reduce bloat on people's systems. Version pinning is good because it helps the team keep a consistent set of versions and can help you identify a full collection of versions that work well together.

If we go the system wide route, I would suggest keeping version pinning. For example:

.local/godot/assets/fsm/1.0
.local/godot/assets/fsm/1.1

Project A could use fsm-1.0 whilst Project B uses fsm-1.1. Project C comes along and could re-use either. Users should be able to specify specific versions of plugins in their dependencies.json files or minimum patch, minor or major versions.

Once dependencies are downloaded, a dependencies-lock.json file could be generated and checked into source control.

karroffel commented 6 years ago

@mhilbrunner I definitely agree, I think the "this should be a plugin" mentality is pushed a bit too hard, exactly those situations sparked those thoughts. I think more common things like the SpringArm are worthy to be in the engine, as well as the InterpolatedCamera, which Juan wants to be removed, is insanely useful and cool! Even thought those things are only a few lines of code, they take a while to get right, so "this is too small" is also a bad justification for pushing towards the AssetLibrary.

My thoughts were mostly centered around "Well, the project lead said it's that way, so what can be done to make the new situation better?"

I would like if the heavy pushing of features as plugins would be lowered a bit, but either way I think the changes presented would help a lot!


@saltares Yeah that was my thought. I am a big fan of cargo for Rust. It operates pretty much the same. Libraries are installed in a system wide folder in all the versions that were ever needed locally. The description is in one file and a specific configuration is in a .lock file, it works pretty great!

I'm glad this sparked some discussinon already :blush:

reduz commented 6 years ago

Here's my view to the above points:

Pushing stuff out of the engine

Pushing stuff outside the engine is IMO the way to go. There are too many platforms where deploy size matterns (mobile, html5) and bloat is a big concern for a project the size of Godot, so we can't just go happy include everything.

Philosophy to me should be that something should be part of the engine only if ALL the criteria below are met:

  1. It's used relatively often
  2. It's difficult to do yourself
  3. There is only one way to do this feature

This rules out spring arm, terrain and I hope it can help to, in the future, remove features such as InterpolatedCamera, Vehicle, etc. that I believe should be outside the engine.

The main concern about this is that, if something is not official it's not going to be maintained. This is where you guys are wrong.

No one said these things should not be official, we can have an official repository for them and they can be first party extensions. Demos are maintained this way and they are up to date, so i don't see why it wouldn't work for an official extension repo.

Extensions should feel like part of the engine

Regarding these points:

I'm very strongly against this. There is no practical benefit at all to doing this, and the implementation cost is very expensive. Sorry guys, I know it feels like a nice feature to have but weighting cost/benefit/risk to it is on the very negative side.

I also think it's positive that custom nodes or resources should appear as-if they were NOT core is a good thing for end users. Users are not stupid and this information should not be hidden for them.

If a new node is added from an extension, and is done with a script, this should be even shown in the create dialog.

This needs to definitely be done and would be a very welcome addition.

My feeling about this is that it can either work fantastic, or result in a complete mess and add-ons that break without an obvious reason. Looking at how other engines work with this, it does not seem like having dependency management to me is so important.

This is something we discussed before, but the conclusion we reached was that it should mostly be script language dependent (and not provided by the engine itself as a core ScriptDB), with a small usability enhancement in the new script dialog in case you want to use a global class.

PLyczkowski commented 6 years ago

My feeling about this is that it can either work fantastic, or result in a complete mess and add-ons that break without an obvious reason.

I guess the add-ons should never update automatically in our case, but instead there should maybe be a list of installed ones with version listed and manual updating, so the user can request update and check right away if anything got broken.

LikeLakers2 commented 6 years ago

This comment is separated into collapsed sections due to the large length of it.

Preamble I came from Game Maker Studio to Godot because, despite having paid for GMS, I still felt I couldn't implement all I want without having to resort to some *really* hacky code or coding in C++. When I came to Godot, not only did I come to find that it was much easier to create new functionality (I've only found *one* thing so far that I could not easily code in GDscript, which is getting spectrum data for creating audio-reactive experiences), I was incredibly impressed at how little I actually had to code to get 95% of my ideas working. Want a camera? Godot's got a camera that can follow a player, with little to no code from you. Want to make an animation in Godot? Not only can you make character animations, that same system can (almost) replace that cut-scene system you worked so hard on, only needing maybe five or so lines of code from you. The editor has even been used to make *other editors* like RPG in a Box. It's fantastic just how much you can create in Godot.

Now with that out of the way, hopefully anybody reading this can understand what I'm thinking coming into this as I try to respond to some of reduz's points regarding pushing things out of the engine. (This won't be replying to the "Extensions should feel like part of the engine" section as I have no thoughts on that currently.)

Removing stuff from the engine, and Guidelines >Pushing stuff outside the engine is IMO the way to go. There are too many platforms where deploy size matterns (mobile, html5) and bloat is a big concern for a project the size of Godot, so we can't just go happy include everything. I can agree that bloat is a concern, but when I see what you're wanting to remove so far, I ask myself if it's really even worth it to have that stuff removed. Even if you remove nodes like Vehicle, InterpolatedCamera, etc... How much of a file-size difference will that really make? Maybe half a megabyte? Maybe a couple of megabytes? Even if it saves several megabytes worth, what's the difference between a 35MB download and a 45MB download anyways? The game assets will quickly fill that 10MB void regardless! Unless you plan on removing half the engine, I don't see the point. If you really are concerned about saving as much space as possible, might I suggest making it easier for the end user to remove the nodes they don't need from the engine instead? Let the people who are going to use the engine decide whether they're okay with the bloat -- don't decide for them. Now of course we can't just include every single suggestion given to us... >1. It's used relatively often >2. It's difficult to do yourself >3. There is only one way to do this feature ...and these guidelines are a great starting point for scrutinizing what should be allowed in the engine. There is no doubt that all three points are already being considered for every single feature request that comes through the issue tracker. >something should be part of the engine only if ALL the criteria below are met But c'mon. The fate of a feature or node should *not* rely that heavily on being a core feature. Your suggested guidelines are great, don't get me wrong, but we shouldn't be using them as a checklist. That's what YoYo Games does when you suggest a feature on their forums, and is another part of why I was pushed away from their engine -- can it be made using an extension? Yes? Then our response is make an extension. Guess how many feature requests have been denied because they could be made into extensions instead, regardless of how reasonable it would be for it to be in the engine, of how easy in comparison it would be to implement, or of how the community feels about that feature. What I'm saying is that you can definitely keep those guidelines in mind. You can consider them every time a feature is suggested. But whether *all* of them apply should NOT be the deciding factor in whether a feature gets in. Weigh the idea against those guidelines, consider how important each of those guidelines are and apply weight to those factors... Just because an idea doesn't seem like it's used too often doesn't automatically make it a bad idea (keep in mind you might not know the true scale of how it's used, even with several years of experience). Just because an idea is easy to implement ourselves doesn't automatically mean it's a bad idea. And just because we can implement an idea in more than one way doesn't mean it's a bad idea! Not to mention that maybe someone's suggesting idea X because they think other people would benefit from it. Remember that this is an engine that bills itself as being beginner-friendly, and every time a beginner has to code something non-specific to their game that's not part of the engine, or go look in some asset library to get the code they need, or go some long route to get what they want out of the engine, it could push them a little further away towards another engine, or sometimes even away from game development as a whole. **tl;dr**: Guidelines are great when they're used as guidelines and not checklists.

An official repository for first party extensions >The main concern about this is that, if something is not official it's not going to be maintained. This is where you guys are wrong. > >No one said these things should not be official, we can have an official repository for them and they can be first party extensions. Demos are maintained this way and they are up to date, so i don't see why it wouldn't work for an official extension repo. I personally disagree with this idea. Alongside with what I've said above (that removing it from the vanilla engine won't save you much on space), I have to ask if we're going to convert them all to GDscript, or if we're going to require them to re-compile the engine just to get those nodes. - If we keep them as C++ and require a re-compile... why though? There's no good reason we should ever be required to recompile the engine unless we're changing a core part of it, or adding our own functionality. That's just additional time taken away from developing, because we're waiting for the compilation process to finish, alongside requiring the average user to install other tools they might not even use ever again just to get some first-party nodes. Now *that* sounds like bloat! - If we convert them to GDscript, then that's going to come with all the limitations of GDscript. Limitations like the potentially overhead of additional child nodes (especially if you're asking the user to import this first-party node as a scene), having to use paths rather than class names to refer to their classes (though you addressed this, it seems, but I feel I need to bring it up), lower performance (regardless of how little the difference might be, it's still lower), etc.. It's a bit more ideal than C++ because you've gone from re-compiling to drag-n-drop, but... Either option also means the files for these nodes will have to be imported into the engine with every new user (for C++) or into the project folder with every new project (for GDscript) that wants to use them. I don't know about everyone else, but that sounds like a hassle to me, and sounds like it'd make your first guideline ("It's used relatively often") very debatable. I'll try to explain why I think this. By requiring additional effort to use those nodes, I'm fairly sure the number of usages of this first-party solution will drop, and this first-party solution really will no longer be used "relatively often" where it may have been before. That, in turn, will slowly increase the number of alternatives that recreate the same features when people stop learning there's a first-party solution -- when I'm fairly sure we'd prefer existing versions (in this case, our own) to be improved, rather than have a new one created with feature X added, a new one with Feature Y, and a new one with both of those features but not as optimized, and so on... It's creating this market for alternatives when people don't know there's a first-party solution. It's creating a *mess*. Additionally, you say they'll stay maintained. That's great! But who's going to maintain them? What are our guidelines going to be for adding new things to this repository? And are we going to go to the effort of ensuring this repository of first-party extensions will always work, whether with a stable release or with master? Basically, is it worth the effort of what is effectively a separate project?

Summary What I'm trying to get at here is that I don't feel, with what your message has told me, that the time's been taken to consider whether those are really the right ways to handle things. Honestly, and this is more than likely the annoyance within me speaking (so please forgive me as this will probably sound rude), it sounds more like "what reduz wants" rather than "what reduz thinks is best for the engine", with little to no consideration for "what the community wants". Sounds a lot like YoYo Games and Game Maker Studio all over again, at least to me.
reduz commented 6 years ago

by the way, I much prefer to implement multiple script per node support rather than cross language inheritance. The former is just a couple lines of code..

On Sun, 10 Jun 2018 at 13:51 PLyczkowski notifications@github.com wrote:

My feeling about this is that it can either work fantastic, or result in a complete mess and add-ons that break without an obvious reason.

I guess the add-ons should never update automatically in our case, but instead there should maybe be a list of installed ones with version listed and manual updating, so the user can request update and check right away if anything got broken.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/godotengine/godot/issues/19486#issuecomment-396063688, or mute the thread https://github.com/notifications/unsubscribe-auth/AF-Z21jtcnUhRyhdyEEnaM_hfTACsxOiks5t7U6HgaJpZM4UhqDC .

Zireael07 commented 6 years ago

Multiple script per node has been attempted a couple of times and always failing.

reduz commented 6 years ago

what was attempted is the multiscript node, but it was a hassle

On Sun, 10 Jun 2018 at 15:01 Zireael07 notifications@github.com wrote:

Multiple script per node has been attempted a couple of times and always failing.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/godotengine/godot/issues/19486#issuecomment-396068742, or mute the thread https://github.com/notifications/unsubscribe-auth/AF-Z26u8Rqc1yL-BGX16CAx_iWsYEo1oks5t7V8CgaJpZM4UhqDC .

vnen commented 6 years ago

I understand that assets should not look the same as the built-in stuff. But I also think they should be at a higher-class than user-made scripts. They should have a clear separation so it's easy to replace/upgrade assets without having to mine them out of your project.

Also, adding a layer for assets and let them integrate better with the engine to extend it is great way to keep the core small and clean, while easily provide new functionality via the asset library. If assets can register new types not only for nodes/resources, but also classes to be extended, in a way that is globally accessible (i.e. not path dependent), it would be a great benefit for frameworks to build on top of the engine.

Frameworks like Escoria could benefit a lot from it, and there could be other frameworks for things like RPGs, platformers, visual novels, etc.

All I'm saying here is quite abstract, but having a clear separation of what's built-in, what's third-party asset, and what is user code (however this separation is presented) will make the "keep the core small and provide extensions" go forward much easily.

neikeq commented 6 years ago

Cross-language inheritance sounds like a horrible idea to me.

willnationsdev commented 6 years ago

In the work I've been doing with the TypeDB in the EditorData, the I've been keeping a namespaced-typename to filepath HashMap that gets converted to a Dictionary in the ProjectSettings upon modification. In my branch, the GDScript parser and compiler are using this data to create global names for types. At this time, I've got custom type scripts showing up. The scripts can access and inherit them by name. What's more, this data is accessible to anything that can see ProjectSettings. So a CSharpScript seen by the Editor can generate a mapping that is entered into the ProjectSettings Dictionary, allowing types declared in C# to be accessible to GDScript. There's probably also a way to wrap the ProjectSettings Dictionary data in a C# Object that returns Script objects when accessed, that way C# would be able to access GDScript types by name too.

This technique keeps the shared namespaces out of core and purely within the Editor context, whereby languages that opt-in can prepare their own means of accessing the data.


I do like @karroffel's suggestion for being able to have cross-language script inheritance though (don't even know if that would be possible). Somehow automatically duplicate methods that delegate calls to the base script and also define all the same properties/signals/constants, etc.

I too would like a more streamlined access to content in the AssetLibrary, with the ability to keep some plugins for the editor overall and other plugins for just a single project.

The TypeDB system I'm making will also give editor support for...

  1. Having all of one's custom scripts show up in the CreateDialog
  2. Have extending a custom type script automatically make you extend that script when adding a script to the node
  3. Have removing a script that extends a custom type script reinstate the custom type script after removal (so you can't remove a custom script - you must use the "Change Type" command in the Editor instead).

Also, all of the interface changes I'm making would keep the custom type scripts clearly understood as being scripts as well as keeping the script in question accessible. Users will always know what is a script and allow them to get to the script, but it still makes them feel like in-engine objects to an extent. Even within GDScript, the names are not globally accessed, as if they were a core type, but rather extracted from a global Dictionary called Scripts. So far I have it working with custom type scripts, but I would like to eventually expand to having the Dictionary register custom type scenes and any typical script or scene available in the project too, likely via the Import system if the script language doesn't have a simple means of acquiring the namespaced typename.

QbieShay commented 6 years ago

As I have been part of the thinking process of this proposal , i would like to point out few things:

My vision of Godot

TLDR: I imagine Godot as a tool that can be highly extended with assets and plugins in a very easy, modular and beginner friendly way.

The motive behind the rejection of my SpringArm pr are more than reasonable, and that's why @karroffel and I discussed about the direction that the engine could take to keep being lightweight and, at the same time, being user friendly.

What I'd love to see is a core engine clean, small and fast and a well integrated ecosystem of addons and plugins. As ArchLinux has more than one package repository, I think it would be great if Godot had an "official"(mantained by Godot) repository for core functionality (and Nodes that are now core) and another repository for community. In the Godot core "extension repository" I would put all the nodes that are a pain to code but not enough used to be shipped with the vanilla version of the Engine. Side note, what it's often not taken into account is that some features might be easy to code but might also require a lot of research before laying down these few lines.

I imagine users being able to download plugins and assets that feel like they are "extending" the engine itself: depending on what you are downloading, you have Godot-fps or Godot-platformer or Godot-racing.

Godot could be the first engine to feel really Unix-like and modular, surpassing even Unity with its editor extensions. Many of you know how good it feels when your Linux system is not "that distro", it's your very own and customized system with your very personal functionalities.

Advantages

TLDR: this approach would allow Godot management to manage separately hardcore technical core features and more user-friendly plugins without worrying too much about bloating the engine

What is good about this proposal is also the separation of concern and the handling of the contributor. I feel that, for now, it's not clear what is okay to be in the core and what is not. There is no clear rule nor statistical data to speak about. Some might say a certain feature is widely used and some other think it's not.

Dividing Godot in different "rings" gives the opportunity for a more fine grained management of PRs and features, and can give more time to @reduz and other very technical developers to focus on hardcore technical stuff, while another team would manage the extension ring, taking more into account user feedback and usability rather than having to face engine bloating, since the "extension ring" would be bloat on demand.

About multilanguage inheritance

As Godot is growing, the asset store can easily become a flaming mess. The strict division between scripting language can literally destroy any attempt that people make to sell assets and grow an ecosystem on the asset store, killing many opportunity for freelance toolmakers to make and sell tools for Godot. Language bridges need to be more complete if we want any chanche to develop a lively ecosystem. For now cross-language inheritance sounds great to me, but i am not technical enough to say anything on this matter. I just want to point out a problem that will likely present itself soon, expecially since there is such a huge community around C#.

Note about usability

I've been using Godot for almost a year now, and lately i feel like it happens that people that decide about features are not using the engine for anything bigger than small demos. @LikeLakers2 pointed out a problem that i feel is shared by some of the users that are not deep into the developing core team.

swarnimarun commented 6 years ago

I think that what is considered bloat is a trivial topic in the greater scheme of things, it's not like the decision we make now is what we will need to abide by for the whole life of Godot. Being open-source allows us to make changes whenever we like the only problem is that the more we wait the more heavy the decision of change is going to be.

Now that's also a discussion for another day. Right now in the present the facts are,

  1. Godot is a compact, fast and portable game engine. And I like it alot but,....
  2. Godot's executable is around 45-60MB depending on various factors. But the export template can be as large as 300MB. Now that's a big blow to the first point.
  3. Godot has lot's of amazing features that make it a treat for the beginners like Camera follow, Parallax BG and Vehicles, this is a big plus over other similar game engines. As it makes creating games much simpler in Godot. And makes Godot a home for lots of newbie devs, who don't have the skills to write their own implementations for such tools.

Now what will happen if we remove features that some people consider bloat. Well the engine will surely be more compact but is that required? I don't see a problem with myself downloading a 100MB Engine if it fulfils my needs better. But what about exporting, won't a 100MB engine export bigger games?

Probably, yes, but in a world where desktops don't seem to use anything less than 256BG SSD + 1TB HDD and even Phones have space starting from a minimum of 16GB all of which which is more than likely to increase, would it be that BAD? I mean to say we don't need to remove anything yet, we just need to start managing a few things better and that should be it. Features should have some sort of importance list to be decided if they are supposed to be in the core or not but that doesn't mean that it becomes a law but rather a rule of thumb.

And for the nit-pickers we could remove some extremely unused tools but that's it. Features don't bloat an engine unless we are careless. And even all the bloat in Unity has not stopped people from using it, has it. And I absolutely don't mean we should not care about bloat but being careful and being paranoid are different matters all-together.

As for multi-language inheritance, I don't really see a use for it here. Sure we should be able to interact with scripts of multiple languages but inheritance! That's likely more work than I could imagine with benefits that just don't seem to be worth it. Whereas, multi-script might be just the solution for lot's of problems from, addons creation to better modularity. Although that is my perspective.

nobuyukinyuu commented 6 years ago

There's something to be said about limiting the number of node and resource types that come in the box. By squirreling away any lesser-used assets to an external download, you make it 1. a lot easier for new people to get a grip on the system just by "Messing around" without being overwhelmed by the breadth of features (decision paralysis), and 2. don't end up creating as much legacy baggage if it turns out some of the more narrow use-case-scenario assets end up having a workflow that isn't as good as something that comes afterwards with a different API.

When the latter happens with built-in's, you're stuck with them. When it happens with downloadable assets, the most popular ones tend to become de-facto standard over time, but don't have their fate permanently entwined with that of Godot. That means they can fade into obscurity without having to worry as much as breaking expected core compatibility.

neikeq commented 6 years ago

The cross-language inheritance point needs some explanation. Not only I consider it to be a bad idea, but I don't even see how it is remotely possible.

  1. Godot's executable is around 45-60MB depending on various factors. But the export template can be as large as 300MB. Now that's a big blow to the first point.

300MB the export templates? Do you mean with assets?

karroffel commented 6 years ago

@neikeq the thing is that if your whole project is in C# but you are using a custom node downloaded from the AssetLibrary then that Node is just a regular node, most likely with just a GDScript attached.

If you want to "extend" the node, you can't! I guess you can add a child node to add new functionality, but that way it's not possible to override things.


Implementation

Currently the scripting languages use a really similar pattern for inherited calls which could be generalized.

https://github.com/godotengine/godot/blob/76875ba145c5745033d0a1c9fda2f4349e2509b3/modules/gdnative/nativescript/nativescript.cpp#L696-L712

https://github.com/godotengine/godot/blob/76875ba145c5745033d0a1c9fda2f4349e2509b3/modules/gdscript/gdscript.cpp#L638-L651

https://github.com/godotengine/godot/blob/76875ba145c5745033d0a1c9fda2f4349e2509b3/modules/mono/csharp_script.cpp#L1175-L1191

The pattern here is

Class *c = bottom;
while (c) {
    if (c->has_method(m)) {
        c->call(m);
        break;
    }
    c = c->base;
}

So it starts at the bottom of the script inheritance tree and then tries to call the upper levels if there's no such method found.

If each script had a Ref<Script> parent_script instead of the language-specific inheritance structure, this could be generalized to

if (this_class->has_method(m)) {
    this_class->call(m);
} else {
    // method not declared here, go to base class
    parent_script->call(m);
}

This might not be a good idea after all, but not having the option to override stuff cross-language might be a problem over time if we push the AssetLib more.

swarnimarun commented 6 years ago

@neikeq See this, image

akien-mga commented 6 years ago

@swarnimarun That's export templates for all platforms, all supported arches, debug and release, their size is completely irrelevant to the "bloat" of the engine. We can decide to support only Linux X11 64-bit release and this file will be 20 MB...

swarnimarun commented 6 years ago

@akien-mga I was not intending to call it, bloat. But a new user will download it full-size they won't be experienced enough to know that they can build separately or download for a specific platform.

How is it irrelevant when this is what we provide to beginner users?

Zireael07 commented 6 years ago

@akien-mga: Export templates size matters since your argument for removing nodes was

deploy size matterns (mobile, html5)

akien-mga commented 6 years ago

@swarnimarun @Zireael07 Yes export templates size matter individually, but my point is that measuring the size of a bundle of templates doesn't give us any indication. The size of one template (e.g. the wasm release template, 3.4 MB zipped, or the Android release APK, 23 MB zipped with armv7 and x86 libs) is much more relevant than "downloading everything needed to develop takes 350 MB" (+ maybe 10 GB for Visual Studio if you need to build something...). What matters if what size a minimal game will take on platforms where size matters (web, mobile).

QbieShay commented 6 years ago

Although i think that deploy size is really important, i also think that what needs to be taken into account is also developer time. For example, Unreal has literally any feature you could be looking for, but the fact that there are so many makes the development process highly inefficient because every time you need to do something with it, you first need to figure out which builti-in functionality you need to use and then learn how to properly use it.

This is something i really love about Godot. You can code your feature straight away and then someone will come and tell you "yo, did you know that we had this built-in thing you could use?"

The problem when an engine has too many features all together is that developing with that engine becomes a huge tiring experience. Yes, it's nice to have all the features you need but it's a gigantic pain to go through all of them to figure out what you actually need. That's bloat. And that's why we are proposing to have on-demand well integrated features.

Ranoller commented 6 years ago

add proper dependency manangement with system wide installs of assets. Move addons out of the project itself with a way to bring them back for modification.

This is the same that losing portability. My plugins are part of my project, i need that in whatever computer i´m working and i need the concrete modification of every plugin for every project... I don´t see any benefict in have my plugins stored in .local/whatever. This is kind of obfuscation my projects.

Actual system of res://addons is good... free for the user and portable around computers. If there is no complains about this system, is necesary to change that? There are a lot of other things that users really complain...

QbieShay commented 6 years ago

@Ranoller you would be able at any moment to copy the plugin into your game folder. Anyway, in that case your project.godot would list your plugin as a dependency and it would be downloaded to your new computer with the correct version

Ranoller commented 6 years ago

Downloaded? My plugins are inside the project that is inside the pendrive, and are part of the git repository ... I don't understand proposal... Plugins change the same that other code... I change more code from plugins that gamecode, i don't wan't to number plugins by version and work outside from the project... Every one line code change will need to manually change every plugin in every computer?... Really plugins feels like part of the project... Give plugins out from git repository is nonsense to me, but i really think that i do not understand something about that...

karroffel commented 6 years ago

@Ranoller the idea is to move those plugins outside of the project that you don't need to modify. For example, the InterpolatedCamera will just be a node you want to use as is, not change the code of it.

The same could be the case with editor-only plugins like TODO lists or git integration etc. Those are things you want to use but not change.

This is a lot like a package manager and libraries work in every language that's not C and C++. You have a file that specifies all the dependencies and then the package manager will fetch those dependencies. This happens even when you change PC - the package manager will download any not-yet-existing packages.

When you want to modify something, then you bring the code into the project. Then it's in your git and the project with all its source code etc. But that's not always the case.

My proposal is to have things outside the project by default until you actually need to modify something.

Currently many assets are mostly templates that you do need to modify, having such a system in place might change that so things are more general purpose via APIs, not via source modification.

neikeq commented 6 years ago

@karroffel But how is cross-language inheritance going to work in each language? It may be easy for GDScript, which is tightly coupled with the engine, but it's a completely different story for general purpose languages like C# or other languages added on top of GDNative. I seriously can't see how this could be achieved, even less how it could not do more bad than good.

neikeq commented 6 years ago

IMO it's not a good idea to modify the content of an addon your project depends on. If any modification is needed, just copy the content outside the addon folder and do your changes there.

EDIT: or "fork" the addon 😜

QbieShay commented 6 years ago

@neikeq What you modify is inside your project. When you want to modify an addon it is copied inside of your project and only locally modified.

PLyczkowski commented 6 years ago

@karroffel An idea of a project plugin and an editor plugin may be needed to avoid confusion, with clear separation of the two.

neikeq commented 6 years ago

@QbieShay What if there is an update that modifies the files you modified?

karroffel commented 6 years ago

@neikeq Script inheritance isn't THE SOLUTION to this problem, it would just be one. There's currently not a good way to override functionality of a node with a script attached when you want to use a different programming language.

If we had an option to override methods in the Script API then that would be good too! (basically externally defining new methods and overriding existing methods).


@PLyczkowski that's a suggestion that would be useful too, but what I propose is not limited to that - and for a reason.

Let's take the InterpolatedCamera example again. I will never touch the source code for it, I will only use it. Having the source in my project would be wasted. Also most projects I make are all 3D, so I would probably have that code copied in all my projects.

On the other hand, a git integration might be a super useful to add in custom behavior that's hard to express with git hooks, so customization there is desired.

There's no black/white line when something should be inside the project and when outside, editor or not.

karroffel commented 6 years ago

@neikeq

What if there is an update that modifies the files you modified?

In that case you would lose versioning information, so you can't update easily anymore. You should use a tool like git for those cases then.

The idea is

Basically they become just regular files once you copy them into your project.

Ace-Dragon commented 6 years ago

Reading this, I can say that avoiding bloat and minimizing the file size of games is a valid one. Allow me to throw a few ideas in here though.

My 2 cents anyway.

willnationsdev commented 6 years ago

@Ace-Dragon

  1. I think your first point would be really cool to have as an engine-wide feature. If you're exported game doesn't use a Node or if a Node is not used by any of the Nodes you are using in your project, it'd be cool if it were possible to automatically exclude them from the final binary. Outside of that, you currently just have to rely on switches and recompile the engine. :-/

  2. This is basically what the Asset Library already is. The issue is that it doesn't have any concept of dependencies, directly drops things into your project directory, and doesn't have any means of maintaining a given set of plugins system-wide for every project you make.

  3. Custom nodes are always explicitly marked, so no need to worry about that (they have a script icon next to them because you make custom nodes using a script to extend it). Because they are scripts, they don't ever get shipped with the engine. They would be part of a plugin of some sort, so if anything, you might have something more like my godot-next repo that is a "node extensions" repository for collecting stuff. It would just be a more "official" capacity more than anything (that repo could also use a lot of work, lol).

Ranoller commented 6 years ago

Ok, slowly but catched... If i maintain the addons inside the project i could work in the plugin code like in another scripts without losing git integration with current project and without losing portability... But if i want to convert any of the plugins in a tool for other developers or in a general tool for all my projects i share it in a form of external package that can be actualiced with the external git repository of the tool and changed to other people with the asset library... If it is, for me is ok... If other developer uses the tool, and need their personal version, he/she need to store inside the project and in this case the plugin doesn't actualice (don't lose their own work)... It is?

willnationsdev commented 6 years ago

@Ranoller Yeah, I think you've gotten the gist of his proposal.

reduz commented 6 years ago

Some feedback on what you guys are discussing

1) If this was an actual problem that needed solving, I much prefer supporting multiple scripts per node (which is just a couple lines of code), rather than cross language inheritance (which is madness). I still don't think this is a problem that needs solving.

2) I don't like the idea of a package manager and system wide install of add-ons. I think it's a terrible idea and nothing meaningful at all is gained from this. I've seen how traumatic is even for Unity developers to update a version of an asset in large projects and see everything break and spend days fixing it.

3) I don't like the idea of package management itself. I've seen Unity users hacking away what they get from the asset store to no end the vast majority of the time. Too often they just use it as a base, then work on it and customize it to their own uses, because the asset you get is only 60/70% of the way where you want it to be.. or they just wanted to see how it's done and then do their own thing later.

I think nothing really is gained in-real-life from the proposals being discussed, so not planning to add much to this discussion myself.

Zylann commented 6 years ago

How much engine size can we reduce if we allow people to export without any 3D stuff in a simple way? Many games (half? more than half?) don't require 3D features but they still sit there in the same binary. I know it can theoretically be recompiled yourself with a macro on every platform though, with a bit of investment... Also, following this idea, if modules were just "internal" dynamic libraries that are part of the official engine's distribs, adding more features to the engine would be less of a problem in the sense that they can be opted out real easy (just that), however using dynamic libraries like this could be a bit constraining for some platforms (mobiles and HTML5 mainly). So for modules I would understand that GDNative is preferred (oh wait... it has the same problems^^).

@reduz how would multiple scripts work? And how would it contribute to engine extension? Did you explain your idea in an existing issue? (asking because it's probably too much to explain it in this thread)

starry-abyss commented 6 years ago

add proper dependency manangement with system wide installs of assets. Move addons out of the project itself with a way to bring them back for modification

No. I'd like them in the project directory for robustness's sake. But maybe with a way to update picked ones manually with package manager, and maybe with system-wide caching of them.

As for bloat concerns, I'd appreciate more control over modules instead, and this was already voted on January 2018 Patreon poll:

Roadmap Priority: More options to opt-out compiling parts of the engine to create smaller binary sizes [ Expected for 3.1 ]

rminderhoud commented 6 years ago

It's hard to comment on this proposal since it touches a variety of things and people are commenting on different aspects and raising different issues.

Overall I think the proposal for the "dependency manager" sounds good on paper but I think the implementation will not net any benefits that simply vendoring addons in the project folder won't bring. It will require incorporating a complex mechanism for dependency management to solve 1 case of "easier to install". This project already has 3000+ issues open with a small team of contributors. Introducing something as complex as dependency management is a footgun in my opinion.

I agree with @reduz that the core should be minimalist. In my opinion anything and everything that cannot be classified as "core" should be incorporated independently. As a user I expect to Godot to be a multi-platform game engine. If I want the godot game framework (vehicles, interpolated cameras, terrain, player controllers. etc.) it should be available as a first-class add-on (or module?) independent of core. I think improving the addon story helps with this but a full-on dependency manager and automated addon manager seems to be pushing it for low net benefit.

My final comment would be to be wary about catering to "beginners" and forsaking "power" users. Godot has been getting the attention of more advanced users due to its powerful features (modules, gdnative, gdscript, editor, etc.) and flexibility. It's these kind of users who put godot to the test, push its boundaries, explore its api and often contribute back to the project. These are the same users who can create symlinks to a directory in their file system to re-use their addons across multiple projects.

vnen commented 6 years ago

If the idea is to remove higher-level nodes from the core, then those nodes should be readily available and easy to find for download. Ideally we could maintain them as "official", so they have better visibility (just like the in-engine nodes have). Then official assets could be used in demos and tutorials without much problem.

The advantage of outsourcing those is that they can be improved and released without having to wait for a new engine version. So bugs could be fixed right away, since for an extension is quite simpler to pack up a new release.

I guess in the end package management would be quite a hassle most of the time. Even in the case of a new version of a custom node you added to your project, updating it might cause many other issues (even with semver, since you might have worked around a bug that is now fixed).

OTOH for some editor plugins it might make sense to have then out of the project and have them updated often. My Tiled importer plugin is an example. You most likely don't want it shipped with the game and want to have the latest version which supports more features and have bugfixes.

This applies even more to other editor extensions, like git managers and time trackers. You might not even care if your team mates have the time tracker plugin or not, it's not related to the project. Yet, you have to put in your addons folder and enable in project settings, then be careful to not commit it with the rest of the game. Those are probably the minority of the plugins, but still having system-wide plugins that are not tied to the project have a real use case.


I believe the middle-ground would be to create a package manager as an external tool, not have it integrated in Godot. It doesn't even need to be official (though it could be "endorsed"). If some changes are needed in the base engine to support this (such as having global plugins), then it wouldn't be much of a problem to add, especially if they are editor-only.

Then those who think it's beneficial can use it. If it becomes widespread and everybody is using it, we will have both the usage statistics and feedback on usability to decide whether and how to integrate it fully in the base engine.


Now, cross-script inheritance for me is a no-no (if that's even possible). Multiscript is kind of weird, but it would effectively replace cross-script inheritance in a cleaner way. Though it raises a few concerns, like: how to call the function of another script in the same object? I guess that's another discussion.

Referencing scripts by name OTOH sounds really nice. Ideally, all resources could be referenced by name so paths are not a problem to solve if you decide to move things around.

QbieShay commented 6 years ago

@vnen the only problem i still see is the incopatibility between addons and projects. How would you address that problem? Because if there is an official extenion lib maintained by Godot, either every node is shipped in every possible language, or i see cross language inheritance the only way that we can keep user friendliness.

Zylann commented 6 years ago

@vnen if you reference all assets by name then it will be a conflicting mess. That's more suited for scripts since only programmers have to come up with unique names for their classes to type and use them (or more rarely, resources). For the other stuff, if we get rid of paths, it would need this https://github.com/godotengine/godot/issues/15673, unless you want also artists, designers etc to have to name all their assets uniquely (which still comes with the issue of renaming).

willnationsdev commented 6 years ago

@Zylann @vnen The TypeDB I'm writing should allow people to optionally register a name for any asset, theoretically. I'm currently only using it for scripts, and plan to also do it for custom type scenes, but you could also modify it to work with resources. The logic currently just has a HashMap each for scripts and scenes, but maybe I could generalize the logic to handle any kind of asset, all in categorized HashMaps.


I can respect reduz's concerns with the system-wide stuff, and those are certainly concerns with plugins that would modify the engine's offerings, but when it comes to features for the Editor specifically, like vnen is describing, I can definitely see that being useful in a relatively safe way. If people are adding plugins to grant the Editor more features, those aren't things they will typically crack open and modify themselves. They'll just use it as-is, or if they do make modifications, it'll be in order to PR those changes to improve the tools probably. You aren't adapting them to a game after all. As such, if it's for a plugin like this, I think we should have a means of easily granting those features to the engine when you start a new project (somehow).


One possible method that could make things easier without needing a dependency manager would be to simply not share dependencies between plugins and instead have them duplicated within the project, that way versioning is a non-issue. It would make an individual project bigger potentially, but if it's for content that is game-relevant and people will be tinkering with it all anyway, as reduz says, then they can just move the dependencies. What we would need in that case is a way for plugins to declare WHAT external-plugin-dependencies they have and give users the option of customizing where they look for that dependent plugin. That way, if they DO choose to move a plugin and have other plugins look for it in the same location, they can customize that. But then, by default, everything would just work out of the box by managing its own dependencies. This case would probably also be easier to handle with the whole plugins-as-sub-projects proposal (#19178).

This would keep us from having global dependencies that require a dependency manager at all (reduz's issue number 3). And we could couple it with a feature that allows users to create a favorited list of assets from the Asset Library that they can choose to automatically install into their projects when a new project is created (so everything is still stored locally, but it happens automatically and they don't have to manually go out, find all their preferred plugins and manually download and install them). This would fix the usability issue that @karroffel wants to deal with in regards to easily extending the engine with identical assets when creating a new project.

For more editor-specific plugins that wouldn't need to be duplicated, perhaps people could just flag those "favorite" plugins and they would have symlinks made for them automatically, storing the plugins in a user folder of some sort. So, maybe have "Editor Favorites" and "Project Favorites" for "I want this in every project, store it in a shared location and make a symlink" and "I want this in every project, duplicate it", respectively.

Combine that with the TypeDB stuff for naming scripts / customizing custom types, and I believe it would properly address all of the mentioned problems except for the cross-language script-inheritance / multiscript issue which is something else entirely.

starry-abyss commented 6 years ago

So, maybe have "Editor Favorites" and "Project Favorites" for "I want this in every project, store it in a shared location and make a symlink" and "I want this in every project, duplicate it", respectively.

Maybe then it's more convenient to have "bundles" of plugins - user-created officially approved + user-created local. So instead of "Editor Favorites" and "Project Favorites" we have "First Person Shooter", "Voxel Block Game", or just "My stuff". Pretty much like e.g. Unity starter assets, but no need to bloat them with textures etc, different types of Godot mascot faces should be enough :D

This also works as a package manager, where package manager now is the human curating plugin bundles.

willnationsdev commented 6 years ago

@starry-abyss The plugins that people make now already have the potential do be "bundles" of other plugins (the user or plugin developer can just move them in themselves and merge them together). The goal here is to have a functional difference between the categories where the editor actually performs usability-improving operations on behalf of the user so the user doesn't have to do things manually.

To this extent, we already have the "humans curating plugin bundles" feature. The problem is with automating tedious tasks while keeping it customizable with efficient productivity in all use cases.

vnen commented 6 years ago

@Zylann

if you reference all assets by name then it will be a conflicting mess. That's more suited for scripts since only programmers have to come up with unique names for their classes to type and use them (or more rarely, resources).

Is it so much of a big deal to properly namespace stuff? But it could be UIDs, it doesn't matter much to me as long as it's not path dependent. Even the path is just like a name, since resources are remapped to their imported counterpart...


@QbieShay

the only problem i still see is the incopatibility between addons and projects. How would you address that problem?

If this is really a problem, then multi-script is likely the simplest solution. It was removed mostly because of the discussion in #8546, but I think this can be solved without much effort.