Track resources and references by GUID instead of res path internally

[Flavelius]

Describe the project you are working on

RPG-like multiplayer

Describe the problem or limitation you are having in your project

It's basically a copy of this issue: https://github.com/godotengine/godot/issues/15673 and everything discussed in it. The gist is that it happens quite frequently that renaming something (nodes/resources) breaks existing scenes. It seems the chances are higher the more deep nested scenes exist in the project. It also breaks by default, from personal experience, if c# scripts are renamed outside the project (in IDEs).

Describe the feature / enhancement and how it helps to overcome the problem or limitation

If resources where tracked by guid or a similar mechanism that uses a database-like structure for assets in the backend, the actual path to a resource would only be stored once and remapping/fixing/changing/editing a reference wouldn't require a slow crawl over the whole project structure with the mentioned potential misses/partial adjustments, making the base more robust while not making it less vcs friendly.

The database-like storage would also allow for storing metadata alongside, like import configurations; or maybe even contain information that can aid in fixing lost inner references by having pre-post comparable data, or may allow for quicker lookup of certain data without requiring to load the actual asset.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Basically what the linked issue mentions

If this enhancement will not be used often, can it be worked around with a few lines of script?

I can't think of a simple workaround that would not require immense exposure of engine internals.

Is there a reason why this should be core and not an add-on in the asset library?

It can make working with nested structures more robust for everyone.

[Flavelius]

Maybe even something like the .meta files unity uses could help with a lot of the problems godot is facing. Like saving user property changes to it, for example the loop toggle in imported animations, which means the original file can remain untouched, but at runtime the respective changes are applied. And if an asset operation is not done in the editor IDE plugins can aid in adjusting the meta's name like it's common for unity.

[dalexeev]

Perhaps we just need to replace numeric IDs with strings and provide a convenient UI that will list all external/internal scene/resource resources. The user will be able to rename the automatically generated names to more meaningful ones.

Before:

[gd_scene load_steps=3 format=2]

[ext_resource path="res://Control.gd" type="Script" id=1]

[sub_resource type="Theme" id=1]
Label/colors/font_color = Color( 1, 1, 1, 1 )

[node name="Control" type="Control"]
theme = SubResource( 1 )
script = ExtResource( 1 )

After:

[gd_scene load_steps=3 format=2]

[ext_resource path="res://Control.gd" type="Script" id="Script1"]

[sub_resource type="Theme" id="Theme1"]
Label/colors/font_color = Color( 1, 1, 1, 1 )

[node name="Control" type="Control"]
theme = SubResource( "Theme1" )
script = ExtResource( "Script1" )

Also, we should probably combine the IDs of external resources and subresources into one namespace:

[gd_scene load_steps=3 format=2]

[resource path="res://Control.gd" type="Script" id="Script1"]

[resource type="Theme" id="Theme1"]
Label/colors/font_color = Color( 1, 1, 1, 1 )

[node name="Control" type="Control"]
theme = Resource( "Theme1" )
script = Resource( "Script1" )

[Zireael07]

... and renaming automatic names will likely result in spurious version history diffs, like already happens when adding/moving resources... :/

[dalexeev]

... and renaming automatic names will likely result in spurious version history diffs, like already happens when adding/moving resources... :/

If you rename the resource before committing, I think there should be no problems. When given meaningful names, the likelihood that another developer will add another resource with the same name in the same scene is rather small. And, at any rate, when resolving conflicts, string identifiers are better understood than numeric IDs or GUIDs.

[Zireael07]

You have a point when it comes to conflicts :+1:

[dalexeev]

You have a point when it comes to conflicts +1

I was referring to conflicts that occur for other reasons. Introducing string IDs undoubtedly reduces the occurrence of false conflicts compared to numeric IDs.

Previously: 2 developers in 2 branches added 2 different resources with the same ID to one scene (for example, 5). There will be a conflict when merging these branches.

Now: 2 developers in 2 branches added 2 different resources with different IDs to one scene (for example, "LoadingScript" and "ButtonsStyleBox"). In this case, there will be no conflict (if different parts of the scene file are changed). And even if both developers are too lazy to rename the added resources, a merge conflict will not necessarily occur ("Script1" and "StyleBox1" are different IDs).

[Flavelius]

As far as i understand, the linked pull request doesn't actually address this proposal, it it only replaces sequential numbers with random strings, nothing more, so every file with external resource still has hardcoded strings to filepaths. Or do i misunderstand it, and it really implements some kind of asset-database?

[Calinou]

As far as i understand, the linked pull request doesn't actually address this proposal, it it only replaces sequential numbers with random strings, nothing more, so every file with external resource still has hardcoded strings to filepaths. Or do i misunderstand it, and it really implements some kind of asset-database?

Indeed, file paths are still used internally. I edited the linked pull request's post to not reference this proposal anymore. (GitHub will still display the reference here despite this.)

[Flavelius]

I'm not sure if should post this as a bug report as i think it qualifies as both part of this improvement proposal and an issue: In my project which contains more than a few hundred small files (icons, sounds, textures), renaming assets or -just creating empty folders- often takes around 10-15 seconds per action, where the editor is 'scanning' the filesystem. An indexing asset database like structure to quickly query for linked assets would likely immensely accelerate that instead of needing to traverse the whole folder structure; and probably not initiating a scan for newly created folders would help too.

[Calinou]

In my project which contains more than a few hundred small files (icons, sounds, textures), renaming assets or -just creating empty folders- often takes around 10-15 seconds per action, where the editor is 'scanning' the filesystem. An indexing asset database like structure to quickly query for linked assets would likely immensely accelerate that instead of needing to traverse the whole folder structure; and probably not initiating a scan for newly created folders would help too.

This is already being tracked in https://github.com/godotengine/godot-proposals/issues/280.

[ghostbutter-games]

@Calinou @YuriSizov So I just opened issue #4598 about this and it was immediately closed as "Not planned", which I find really irritating.

Coming here, I also realize that this has been an issue to many people and always been closed pretty much straight away without a lot of discussion happening.

I am a professional developer, having released multiple commercial titles and am currently making a living off my games. I have been using Unity for more than 7 years and am really looking forward to switching to Godot from here on out.

However: This makes me really confused as to where the core developers are standing with regards to making this a real - non-toy engine that can and should be used by professionals.

Being able to rename script files, move script files and rename symbols from an IDE should be one of the most important, basic things in the toolkit of any developer using Godot - yet this is currently simply impossible.

This is a huge show stopper problem for me.

I ran into this even when making a small jam game with Godot and GDScript, but this becomes an immense issue when working on larger, complex games with potentially hundreds of script files & custom classes.

As of right now, this makes the process of developing non-trivial games using Godot a really brittle, fragile process and feels like I could break the project integrity at any time if I don't pay 100% attention to what I am doing in the IDE.

I do realize that this is not trivial to implement, but I really urge the core developers to make this a higher priority for Godot 4.0.

(Please let me know if this is – again – the wrong or duplicate issue and where I should have this discussion instead)

[YuriSizov]

So I just opened issue https://github.com/godotengine/godot-proposals/issues/4598 about this and it was immediately closed as "Not planned", which I find really irritating.

It was closed as a duplicate, not as "not planned". That's just GitHub's general label for issues that were closed but not fulfilled. Sorry that it made you feel irritated, but software management tools are stupid like that at times and can't cover all user needs. We try to minimize the number of open proposals because having 2500 of them makes it very hard to focus development, and things that can be consolidated get consolidated.

As I've explained in the linked issue, we also use the issues for actual suggestions on how to fix problems, with implementation detail or at least mockups and overall plan. You only posted a complaint, which is non-actionable. It's fine to discuss your ideas for a solution, if you have any, here, as it is a good idea to open a discussion thread in the discussions section of this repository.

In general, giving a thumbs up to a proposal that you agree with is also good to give it more weight. It helps us to decide what users expect the most from the engine.

Currently, this is not a very supported proposal, but IMO it is still likely to be implemented at some point. We are trying to narrow the scope of 4.0 at this point, but it's not ruled out from future releases, and can even be a part of 4.1 even if it breaks compatibility to some degree.

Do note, however, that support for external tools, while essential for C# and other third-party languages, and often preferred for the first-party GDScript, is not the core of Godot editor experience. We don't have many opportunities to measure public opinion other than discussions such as this (and reactions to the OP of such discussions), and our annual community poll. We have limited resources, and we try to make sure that we can make the experience good where it most counts for the existing community.

If you have a know-how and can work on this feature, we'll appreciate it. If you can provide a more detailed plan for your alternative proposal, you can describe it there and we can reopen it, in case you don't consider this one a solution that fits you.

[ghostbutter-games]

So I just opened issue #4598 about this and it was immediately closed as "Not planned", which I find really irritating.

It was closed as a duplicate, not as "not planned". That's just GitHub's general label for issues that were closed but not fulfilled. Sorry that it made you feel irritated, but software management tools are stupid like that at times and can't cover all user needs. We try to minimize the number of open proposals because having 2500 of them makes it very hard to focus development, and things that can be consolidated get consolidated.

As I've explained in the linked issue, we also use the issues for actual suggestions on how to fix problems, with implementation detail or at least mockups and overall plan. You only posted a complaint, which is non-actionable. It's fine to discuss your ideas for a solution, if you have any, here, as it is a good idea to open a discussion thread in the discussions section of this repository.

In general, giving a thumbs up to a proposal that you agree with is also good to give it more weight. It helps us to decide what users expect the most from the engine.

Currently, this is not a very supported proposal, but IMO it is still likely to be implemented at some point. We are trying to narrow the scope of 4.0 at this point, but it's not ruled from future releases, and can even be a part of 4.1 even if it breaks compatibility to some degree.

Do note, however, that support for external tools, while essential for C# and other third-party languages, and often preferred for the first-party GDScript, is not the core of Godot editor experience. We don't have many opportunities to measure public opinion other than discussions such as this (and reactions to the OP of such discussions), and our annual community poll. We have limited resources, and we try to make sure that we can make the experience good where it most counts for the existing community.

If you have a know-how and can work on this feature, we'll appreciate it. If you can provide a more detailed plan for your alternative proposal, you can describe it there and we can reopen it, in case you don't consider this one a solution that fits you.

Thanks for your explanation, I appreciate it! I think it's really strange that Github adds "Closed as not planned" - it signaled to me that this was already decided that it would never happen and it would not be open for discussion or something.

I posted a "complaint" because I am not skilled at engine development and have not the faintest idea on how to practically fix this. I am, however, very experienced with actually making games and know for sure that this is a really huge issue that will break my project at some point and prevent me from shipping games in Godot.

I have read through all the linked, simliar issues and have not seen a workable solution to this yet - maybe I did not understand something, I don't know.

I know that GDScript is the primary language - but this also affects GDScript and is in fact the place where I encountered that issue: I could not rename/refactor symbols in GDScript from within the editor scripting interface, so I tried doing it with VSCode, and everything broke. It's just a really jarring experience right now and fixing this would hugely help the UX for both new and old developers.

I am aware of the annual community poll and if I remember correctly, in 2021 about 20% of developers did use C# as their main language - presumably with an external editor. That percentage will probably rise even more with 4.0 as the plan seems to be to merge both Mono and Non-Mono editor builds into one build - making C# even more accessible.

I think fixing something that affects at least 20% of your users in a significant way is worth doing.

Anyways - sorry if I come across slightly harsh - I am just in a bit of a panic because for me this issue is so huge that it would make me reconsider using Godot at all if it was not fixed.

I hope this issue can get some traction and the necessary core dev support - unfortunately I am just a (mainly C#) game dev and can not contribute anything meaningful to the C++ core as of yet - I can just say with the utmost urgency that this is the most problematic thing about writing gameplay code in Godot right now (to me).

[Calinou]

I think it's really strange that Github adds "Closed as not planned" - it signaled to me that this was already decided that it would never happen and it would not be open for discussion or something.

For reference, the close button in the top-left corner of the comment box looks like this:

It's only visible to those with permissions to close issues on a repository, and unfortunately, the close notifications don't use the same wording.

[KoBeWi]

This proposal is mostly implemented btw. Almost all Godot resources now have an associated UID and scene stores reference to resources both by UID and by path. If something is invalid, Godot will automatically try to resolve the depenedency.

The only resource that doesn't support this are scripts, because they are plain text without defined structure, unlike tscn/tres files. For GDScript I remember there was an idea to add an annotation that defines script's UID. It could also be saved as a comment in the first line. For C# we could do something similar.

[ghostbutter-games]

This proposal is mostly implemented btw. Almost all Godot resources now have an associated UID and scene stores reference to resources both by UID and by path. If something is invalid, Godot will automatically try to resolve the depenedency.

The only resource that doesn't support this are scripts, because they are plain text without defined structure, unlike tscn/tres files. For GDScript I remember there was an idea to add an annotation that defines script's UID. It could also be saved as a comment in the first line. For C# we could do something similar.

That's great to hear - I was not aware that UIDs were already implemented in the master branch. Maybe @neikeq has an idea on what could work with C# scripts? I think a single file tracking all scripts and UIDs would be preferable to having the UID in plain text in every script file, but I don't know if that would be feasible.

I also remember that a couple of years ago, renaming/moving C# files within e.g. Rider or Visual Studio would break things in Unity as well - but somehow they managed to fix that. Not sure if that was from the Unity side or the Rider side, though.

I am definitely also on the side of keeping things as simple as possible, so I don't think a language server would be necessary, as that seems like a huge pain to maintain. (I think that possibility was talked about in another issue about this.)

Anyways, I really appreciate the discussion around this as this is currently probably my # 1 most-wanted feature.

[Flavelius]

Interesting, i just had to look it up and it seems there was actually work being done in the right direction. 4.0 stable cannot come soon enough (way too buggy for me currently unfortunately) https://github.com/godotengine/godot/pull/50786

[Byteron]

while essential for C# and other third-party languages

You offer a Godot Mono download as official build, and plan to fully integrate C# into Godot Classic, so there is only one binary, for 4.0. How is C# a third party language, and not first-party?

[YuriSizov]

@Byteron That's just semantics, don't take them close to heart.

It's not a part of the core, and it's not a fixed part of the distributed package. It is officially supported, but it is a third-party language. It was only added the way it was because there was interest from Microsoft and because some people really really wanted it migrating from other tools, and language bindings in Godot weren't a thing yet. If it was done today, it would've been done as a language binding, similar to Rust or Nim. In fact, I believe that the plan for 4.0 is to provide it as a GDExtension. You won't need a separate build of the engine, but it would still be a pluggable part.

[ghostbutter-games]

@YuriSizov We are getting a bit off-topic now, but I find some things about this confusing:

It's not a part of the core, and it's not a fixed part of the distributed package.

GDScript is also not a part of the core, isn't it? It's a module, same as the Mono/C# module.

@neikeq confirmed in #2333 that "There will only be one editor binary, which will include C# support (some dependencies may not be included by default, but the editor will download them transparently). No more Classic vs C# versions of Godot."

I believe that the plan for 4.0 is to provide it as a GDExtension.

Again, quoting from neikeq, the plan for 4.0 seems to be to keep it as it was, then maybe provide a GDExtension for 4.1, but it's not clear if this will happen at all yet.

If it was done today, it would've been done as a language binding, similar to Rust or Nim.

I find that really problematic because that would lead to an absurd amount of language fragmentation among Godot developers - even more than currently exists. I would not want that to happen, I really like that currently C# is for example officially documented alongside GDScript in the Godot Docs and the fact that a large share of devs (about 20% or so) does use C#: There is tons of benefits to this, a lot of community support, a lot of tutorials and very good usability in general.

This is absolutely not the case with (current) GDNative extensions such as Godot-Rust (the best maintained, but still lacking) or other ones. The quality of these bindings varies very much and most of them I would say are practically unusable for commercial projects.

I really really want there to be one non-domain-specific language that is well supported by the core team, that lets me build complex, ambitious games with industry-standard tooling. I do not care that much if it's C#, Java, Rust or any other mature, high-performance language. But that choice is already C# - why not go all out and make it a first-class citizen for Godot, even more than it currently is?

I have the sneaking suspicion that some parts of the Godot core team view the "special treatment" that C# got in the past as a mistake and would like to correct this by making every language other than GDScript "equal".

As I already said: I do not care in particular about C# but me and most other non-hobbyist gamedevs really need a language - any language - with those characteristics (statically typed, huge library ecosystem, great tooling, high performance etc.) and GDScript really does not cut it, as much as I love hacking away with it.

And we need official support for that language, coming from the core devs and not relegating that to 3rd party binding libraries that may or may not work.

Sorry if this went a bit off-topic but I think it's a very important discussion that needs to be had.

[YuriSizov]

I really really want there to be one non-domain-specific language that is well supported by the core team, that lets me build complex, ambitious games with industry-standard tooling. I do not care that much if it's C#, Java, Rust or any other mature, high-performance language. But that choice is already C# - why not go all out and make it a first-class citizen for Godot, even more than it currently is?

C# is not provided because it is in any way better than GDScript. Our core focus and goal is to have performant GDScript, we believe that a DSL is the best option for the engine, otherwise we wouldn't spend time designing and improving it, having a dedicated team for that feature. GDScript is by no means a toy scripting language that is unusable for larger scale projects.

For parts of your game that require raw performance you can always use C++ and modules. They aren't hard to make, especially if you are already in the business of optimizing performance and understanding where it comes from, for your project and in general.

And we need official support for that language, coming from the core devs and not relegating that to 3rd party binding libraries that may or may not work.

In an open source project like Godot "official" is not really a meaningful qualification. We only have one person working on C#/Mono/.NET, and while MS was sponsoring this work at some point, it doesn't anymore. We could just as well welcome Godot Rust under the organization umbrella and provide it as an endorsed package, if things worked out differently. But it would still depend on willing maintainers for the feature to exist.

In other words, developers and support come from outside in, we don't have hidden human resources that we can allocate, and we prefer not to push people to work on things they don't want to work on. Maintainers are there to organize things, evaluate contributions and provide long term commitment. 🙃

C# support, while beneficial to some users, is not something that would put the next version of Godot on hold. If neikeq doesn't have time or desire to work on it for 4.0, we'd still release 4.0, without Mono/.NET. It looks like he is committed to make it in time, but we don't put any pressure on him, thank him for his benevolence, and let him work in peace (and I think vnen is helping with some bugs too). But it won't be a showstopper if he was unavailable. That's what I mean by it not being a part of core.

---

This is indeed very much off the topic for the current thread. Feel free to use social platforms if you want to discuss this, or general channels on the RocketChat.

[ghostbutter-games]

I am not saying that GDScript is unusable for larger projects – but there are and always will be real downsides to choosing it and most studios with more than a few developers would very likely want to choose something different: Strong static typing, IDE support and a mature library ecosystem are hugely important. GDScript will never have this.

C++ modules are not an alternative to GDScript, the workflow is completely different and basically completely bypasses the editor. It's good for very specific, usually performance-oriented problems but not for writing general gameplay code and working with Nodes, which is what most people would want and do with C#.

I find it really harsh to say that you would really think about releasing Godot 4.0 without Mono/.NET support. I hope that's not the general opinion of the rest of the core team (neikeq obviously excluded): Doing so would cull >20% off Godot's current user base, most likely the part that's most invested in making ambitious games with long development times. Many of those people could then not keep on working with the latest version of Godot, would be (rightfully) frustrated and would most likely leave Godot altogether - I know I would.

This is really about a few things: Trust, and stability. If I start using Godot now for my next multi-year project, I need to know that I can rely on C# support to always be there, and for it to be robust and allow me to ship my game. This is the opposite of robustness, this is unreliable and unstable.

I will not engage this topic further in this issue, again sorry for going off-topic too much. If you want to keep on discussing this, my DMs on Twitter are open.

[Zireael07]

How do you know C# users are "the most invested in making ambitious games"???

[ghostbutter-games]

How do you know C# users are "the most invested in making ambitious games"???

I regret that wording, I should have probably said "complex".

You would generally not use C# unless you knew you needed it - meaning you're not a beginner and have probably had some experience with GDScript already. C# gives you, as an example, static typing, which translates to more control over larger codebases. You would usually have larger codebases in more complex games.

It's the same reason why almost all experienced JavaScript developers nowadays prefer to use TypeScript: It's just way easier to navigate a vast, complex codebase with it.

[Zireael07]

@Kopkako: I happen to be working in Python with a large codebase that's been growing for 10 years or so. Complex, large projects are possible in languages w/o static typing.

[Byteron]

@Kopkako: I happen to be working in Python with a large codebase that's been growing for 10 years or so. Complex, large projects are possible in languages w/o static typing.

possible doesn't mean pleasant.

It is totally possible to walk 10 miles on hour hands while blind-folded. Doesn't mean it's practical in any way.