Implement custom resource path support

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

There are several cases where the resource file paths offered by Godot are not flexible enough, and community has complained about this.

• A way to have editor-only assets (for global plugins).
• A way to stream assets from the web or other locations.
• A way to just create custom resource paths for users to interact with sandboxed game data.
• A more flexible way to have moddable content without compromising the game filesystem.

Godot currently hardcodes res:// and user://, but users often require more flexibility here.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

This feature would allow to add custom resource path providers to the filesystem. Users would need to pass a class and some configuration information. Probably an API like this:

# Create the game:// path using an existing filesystem path
FileAccess.add_resource_path("game://","/home/user/Documents/somepath") 
# Create the assets:// path using an URL
FileAccess.add_resource_path_url("assets://","https://mygame.com/assets")
# Create the database:// path using a custom class that inherits FileAccess and DirAccess.
FileAccess.add_resource_path_class("database://","CustomFileAccessClass","CustomDirAccessClass",userdata)

This would allow, as an example, to have global editor plugins. As the editor could register an editor:// path where the global plugins are saved.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Currently the res:// and user:// paths are hardcoded, and a lot of places in the codebase check for "res://' manualy to validate resource files. This would need to be cleaned up to look up into a FileAccess function.

Implementing this is not specifically hard, since what determines the base path and the File/DirAccess to use is mostly on open().

If this enhancement will not be used often, can it be worked around with a few lines of script?

N/A

Is there a reason why this should be core and not an add-on in the asset library?

N/A

[mrpedrobraga]

On my RPG framework StarEngine, I implemented a custom scripting language, StarScript, which I wanted to be able to quickly refer to resources without having to load them as export variables.

Paths are clunky and can change, so I created a class ResourceMap which stores, in a dictionary, instances of ResourceProxy which contains the path to resources and a reference if it's loaded.

Then the resources can be loaded from StarScript with load MUS/Mysterious Forest or load SSH/lines_school.

Would this system if implemented make this addon redundant?

And if a custom path root is under res:// will godot be able to track Move actions on resources that are assigned to export variables on scenes?

[RedMser]

@reduz Some thoughts:

• This needs to be compatible with ProjectSettings.load_resource_pack. It looks to me like it fits with the other APIs you proposed, so maybe move/rename it to FileAccess.add_resource_pack("mod://", "/path/to/pck", replace_files, offset)
• Would be nice to allow "mounting" to a sub-folder. So instead of mapping a folder to assets:// I could create a virtual sub-folder like assets://models/ and load resources from that. Virtualizing the file system would make isolating mods from each other a lot easier. I know this is possible via your custom class approach, but I feel this would become a common requirement.
• To go hand-in-hand with the first point, it would be necessary to allow specifying a replace_files boolean which would work similar to how it does for pck file loading.
• Unload functions would be important as well, I can imagine. See https://github.com/godotengine/godot-proposals/issues/2689
• Regular filesystem access could be moved to a file:// protocol. Not for security, but for consistency, and to avoid bad practices like begins_with("res://").

Maybe the overlap I'm seeing between resource packs and resource file systems isn't correct. But as they both are basically just a source of resource files, the two should not end up as different, incompatible systems.

Ideas for the future:

• Possibly allow specifying these additional file systems in ProjectSettings, so that they can also appear in the editor filesystem viewer? Or maybe it must be done using an EditorPlugin.
• Update the editor filesystem viewer while debugging a game, so you can inspect which files are currently loaded in which namespaces? Would be especially great for debugging custom FileAccess/DirAccess APIs, to inspect them and try methods like copy or delete.

---

@mrpedrobraga No need to store the loaded reference, since Godot already caches resources by default. Multiple loads won't make performance worse from my understanding. But you could probably do it using this proposed system, by creating a virtual filesystem like star://MUS/Mysterious Forest which aliases other paths and passes them to load as-is.

[sairam4123]

I had suggested this kind of Custom Resource Path support in the discussions. I had kind of tinkered with the idea of custom resource paths which does the save, load and overwrite operations.

[mrpedrobraga]

+1 for mounting points specifically.

It seems like a good way to swap a large amount of resource pointers without loading a whole .pck on every switch.

I'd imagine calling load again would get the new version while the old version is still loaded whenever until they're disposed of (reloaded or deleted).

[Frontrider]

Is it the right thing to do here to use the "protocol" of the path for what I could consider "namespaces"? My only concern here is how easy it is to get 2 things to clash, and how much work is needed on the user (here the developer using the addon/library) to prevent those, and how brittle the code may become from it?

We need this idea, the only thing I'll question is if this exact way is the right one. For example what happens if a 2 packs I pull in declare 2 file accesses with the same name?

[bitbrain]

I can imagine this working with something like Steam Cloud. I could easily read/write files using the steamcloud:// path. This makes me wonder how this proposal solves the issue of authentication: not every custom resource path might be available by default but needs to be made accessible through authentication.

[Frontrider]

I can imagine this working with something like Steam Cloud. I could easily read/write files using the steamcloud:// path. This makes me wonder how this proposal solves the issue of authentication: not every custom resource path might be available by default but needs to be made accessible through authentication.

What I was thinking is to map the resources to a rest api/s3 bucket and load assets from there.

[Sooshiko]

I love the idea but is there a way to set the custom paths in ProjectSettings? I'd like them to be available at startup so we can use them in ProjectSettings such as Log Path. I always wanted to output my logs to a folder next to the executable and I can't find a way to do it currently.

[me2beats]

I love the idea but is there a way to set the custom paths in ProjectSettings?

Or in EditorSettings

For global plugins imo it would be better to use editor settings folder for editor plugins data

[AlfishSoftware]

Just an observation. I think it's better to avoid using :// double slash for those, because people are very likely to incorrectly assume these are compatible with URI syntax, but they aren't.

Godot's existing res:// and user:// paths cannot be considered a form of URI because the URI syntax makes it so the part after // and before the next / is the "authority" (i.e. the part with the host, port, etc). That's why file URIs in the local system are either file:... file:/... or file:///... (0, 1 or 3 slashes). The "path" only starts after a single / or after a //<authority...>/ where authority is optional. So, if you don't use authority you should either have 0 or 1 or 3+ slashes, not 2.

It would have been better if it had been res:/// and user:/// and/or res:/ and user:/, but those are already done now. But these new ones don't need to make the same "mistake".

[rsubtil]

Just an observation. I think it's better to avoid using :// double slash for those, because people are very likely to incorrectly assume these are compatible with URI syntax, but they aren't.

I don't think the existing system has ever meant to imply it follows the URI specification. The ://, from my experience, is only meant to separate the source (res or user) from the file path, and the concept of authority is not used. While it looks similar to an URI, it doesn't share any of that extra behavior expected from a URI.

I agree that if it were res:/ or res:/// then the possibility of users confusing this as following the URI spec would be lower, but since it is res://, having custom resource paths deviate from this would only generate more confusion.

[Frontrider]

It would have been better if it had been res:/// and user:/// and/or res:/ and user:/, but those are already done now. But these new ones don't need to make the same "mistake".

Save this for Godot 5? I don't know when/if that would happen, but it may be a good thing to remember.

I agree that if it were res:/ or res:/// then the possibility of users confusing this as following the URI spec would be lower, but since it is res://, having custom resource paths deviate from this would only generate more confusion.

Yes, now it was done like this, it has to match. It being normal URI spec would indeed be better, but if we get it in Godot 4 then it has to follow this spec.

[Zireael07]

@Frontrider Godot 5 is years down the line.

[Frontrider]

@Frontrider Godot 5 is years down the line.

I know, mainly just noted that it is that territory.

[AlfishSoftware]

Agree that this seems like a major change, and to keep a "proper URI compatibility" for res and user in mind for Godot 5, whenever that happens, is a good idea.

On the other hand, I'm pretty sure res:/// is forward-compatible, it already works currently (since consecutive / slashes in path are just merged) so if the Godot team wanted, they could start, little by little, "pushing" that form even now, in preparation for a future definitive change. That's why I suggested it could start here. Maybe these paths could even also work as :// for the moment, but if the API is prepared and documentation always refers to :///, then you would be "future-compatible" in a better way IMO.

In any case, I was concerned because of the examples given for the API; it should not include slashes:

FileAccess.add_resource_path("game","/home/user/Documents/somepath")
# note "game" instead of "game://"; the :// or whatever amount of slashes should be implied

Still, to be honest, for custom resource paths that I define myself, I would expect a well-defined and well-known standard of URI compatibility, so the examples above would make much more sense like this:

game:///home/user/Documents/somepath
assets:https://my-game.example.com/assets
database://user@my-server.example.com/schema1/MyTable/?date_since=2023-08&category_id=4#page1

Not just for the slashes/authority part, but the whole thing, including query parameters and fragments, all of those matching URI could be useful for users' arbitrary definitions.

[eight-ways]

The res:/// format feels like a workaround. Indeed, the amount of code changes and the impact on existing code may be minimal, but there is no doubt that it will become a technical debt in the future. I feel it would be more beneficial to adhere to a format that can satisfy the URI specifications.

[eight-ways]

I believe the core of this discussion centers on how to avoid collisions within the scheme.

Firstly, there are three combinations of problems:

Engine vs Engine: This issue can be easily resolved through communication or coordination.

Engine vs Plugin: This might be rephrased as, "does the engine interfere with existing plugins when new features are added?" One suggestion could be to set a prefix for the user plugin's scheme, for example,

ex-assets://
plugin-assets://

By enforcing such prefixes, it’s possible to separate engine functions from user extensions, which should prevent any hindrance to future engine development.

Plugin vs Plugin: This largely falls under the user's responsibility, hence it may not be a concern for the engine developers. However, if needed, having a config file where each plugin's scheme is listed could help avoid the introduction of conflicting plugins.

I consider this issue a high priority as the current res:// format is not suited for large-scale development.

[Frontrider]

The res:/// format feels like a workaround. Indeed, the amount of code changes and the impact on existing code may be minimal, but there is no doubt that it will become a technical debt in the future. I feel it would be more beneficial to adhere to a format that can satisfy the URI specifications.

This is actually good, but since Godot 4 is out the way it is now it should not be changed/broken.

For Godot 5, make it happen.

[dalexeev]

See also:

• godotengine/godot#22825
• godotengine/godot#22950

This is an interesting idea. But we should think about making sure that custom protocols don't break/shadow engine or other commonly used protocols. In the code base I found mentions of the following protocols:

• Godot-specific: res, user, uid, local, gdscript.
• Network: http, https, ftp, tcp, ws, ssh.
• Applications: file, mailto, itms-apps, trash.

[willnationsdev]

Perhaps it would be a good idea to have the engine and plugins define up-front all of the different protocols that they use? For example, via a read-only ProjectSettings property and the plugin.cfg file; that way, users have a centralized place in which to look to see which protocols/terms are "taken" for their project.

Back when I had more time, I was working on something akin to bitbrain's Pandora project (a Resource-driven "RPG database"), but ran into problems in early days: I wanted a centralized pipeline for processing over whatever users were saving into it w/o resorting to a separate singleton (just good 'ol ResourceSaver), and the only way to do that was to have a custom protocol...and thus, ran into this feature being needed. Definitely a critical component for larger-scale projects that intend to stream external data effectively and provide stable support for modding. I'm glad to see associated work seems to be moving along well (specifically godotengine/godot#61286 and the issue it partially resolves #2689).

[willnationsdev]

Another relevant comment from a different proposal that suggests inclusion of a pack protocol as a replacement for res:// for intra-PCK inter-resource references as a means of differentiating from resources located in the external game project.

[nordyuki]

I think the things discussed here are sounding increasingly similar to a Virtual File System. I'm currently using one (https://github.com/xoofx/zio/) in my project so that we can reference external files using virtual paths (like /gamedata/foo/bar.cfg) instead of doing complex path calculations, and a similar function should be also needed by many other people too.

As for the protocols, res://foo.tres just sounds like /res/foo.tres in disguise. It's not much difference between registering a new protocol and mounting a custom-implemented VFS into the parent VFS (apart from not allowing a couple of characters in the path string). So if we want to dream big, we can just implement the current res://..., user://... and other stuff all as VFSes, and ditch the protocol thing altogether to favor virtual paths without protocol prefixes. [^1] Yay, we now have a unified interface on registering protocols, mounting arbitrary packs onto arbitrary directories, and overlaying multiple packs together. (what is this, plan9?)

What's even better on using VFSes is that every VFS implementation is a working file system on its own, so you can imagine use cases like giving different VFS to different plugins or subprojects, so that every plugin and loaded package can work on their favorite file system mounting scheme, without worrying about breaking other parts of the program. You can give every pack their own mounted VFS, so inter-pack and intra-pack references are nothing more than absolute and relative paths, you get to control which folders can be seen from where, and additionally we get free sandboxing for packs.

VFS isn't without problems. Aside from needing to rewrite a large portion of file managing code, call paths might become significantly longer with many virtual calls in them, and performance might degrade when too many VFS layers are overlaid together. You might also lose a global file system shared by everyone if sandboxing becomes a thing. It's just my two cents, and please take it with some salt.

Disclaimer: I'm still new to Godot (I fled from Unity in August), and haven't looked into Godot's code yet. All my knowledge on how Godot's file system is implemented comes from skimming through issues like this one. I may be wrong in some places, and if that's the case, I would appreciate a kind correction.

[^1]: A compatibility layer can be added to preserve backward compatibility.

[Frontrider]

I think the things discussed here are sounding increasingly similar to a Virtual File System. I'm currently using one (https://github.com/xoofx/zio/) in my project so that we can reference external files using virtual paths (like /gamedata/foo/bar.cfg) instead of doing complex path calculations, and a similar function should be also needed by many other people too.

That is the better version it. That also solves the problem of "what if the devs need local assets, while the finished product uses remote ones". You can control it from 1 location in the code.

[Spartan322]

As for the protocols, res://foo.tres just sounds like /res/foo.tres in disguise.

The whole <scheme>:// is already closer to the standard URI structure, its like a URI, not just a VFS, and you can use a URI to reference a VFS but you can't reference every type of URI using a VFS, this is especially incorrect if we look into the concept of remote assets which you can actually reference in a URI. You can also forbid access by scheme easily whereas you can't always have an inherent idea what the VFS is doing or points to. Godot's scheme doesn't currently support an authority element of the URI standard, so its not as comprehensive as a standard's compliant URI, but its still more closely related to a URI.

https://wikipedia.org/en/Uniform_Resource_Identifier

[fire]

https://github.com/godotengine/godot/blob/6543495b49613d20f7e32f2b9d38e4a2f1d06db1/core/io/file_access.h#L45-L46 FileAccess is already a virtual file system and especially if it's FileAccessMemory or a FileAccessPack.

[Frontrider]

Then we have a second reason for why it might be a good idea to do it through that.

I like that idea for one reason. You could mount dev resources from res or an external folder/pck, while in production you can mount say web resources without touching anything in your code except 1 place where you toggle it. Instead of having 1 if for each resource you need to access.

[nordyuki]

The whole <scheme>:// is already closer to the standard URI structure, its like a URI, not just a VFS, and you can use a URI to reference a VFS but you can't reference every type of URI using a VFS, this is especially incorrect if we look into the concept of remote assets which you can actually reference in a URI.

Yeah, I'm more from a current state perspective, that res:// and user:// look more like mounted directories instead of full-fledged URIs. I'm also kind of afraid of people abusing URI protocols [^uriprotocol] or path structures [^uripath], while in VFS the interface is more restrictive and provides a more uniform way to represent resource structures.

[^uriprotocol]: e.g. creating protocols with colliding hardcoded names everywhere since they're not namespaced, or creating redundant protocols that can be replaced with regular paths. [^uripath]: Since in URI, additional to path, you also have authority, query and fragment to complicate things up.

Custom URI protocols might be beneficial compared to pure paths on things that aren't naturally directory-structured. For example, things like /code/dotnet/by-fqn/MyGame.Effects.MyEffectNode would look like a strange query crammed into directory structure, and remind me of /proc/42/fdinfo/4 or /dev/disk/by-id/scsi-1234abcdef. On the other hand, dotnet:fqn:MyGame.Effects.MyEffectNode does look more natural to human eyes (event though the underlying structures are pretty much the same).

godotengine/godot@6543495/core/io/file_access.h#L45-L46 FileAccess is already a virtual file system and especially if it's FileAccessMemory or a FileAccessPack.

Then we have an even better reason for that! In this case, it won't be that much work (compared to what I expected) to write a DirAccess that mounts/overlays other DirAccesses onto given paths. And then we force relative local paths in standalone pack files, and that should be a minimal working example on mountable packs that won't mess up the file system.

[Spartan322]

FileAccess is only kind of a VFS specifically because res and user are hardcoded into it, (and even then its not really FileAccess that's doing that) if resource paths that aren't local assets are included, it would cease being one, unless you limit FileAccess from doing that but then it would disagree with the rest of the interface and would be incapable for clearly forbidding certain types of access behaviors, like FileAccess would have no idea if its trying to look at a URL with a VFS without checking the whole path. VFS is just honestly way more limiting then relying on protocols, I find it way more useful to actually see and indicate a clear accessing something without trying to traverse some path.

[Frontrider]

I think the question between "scheme" and "vfs" is what this is for.

• Scheme is the only option if this is meant to be the "everything client", with the resource path being a full http url, sql query (my impression from a few examples here that it is also effectively included in some form) etc.
• VFS is a good abstraction we can use if it was meant to be the thing it seems to be at the moment: file access from different locations. (currently local files and the pck's contents)

[nordyuki]

I think it's important for us now to clarify what a resource path is designed for, and more importantly, what it is not designed for, and then talk about the redesign accordingly.

As far as I'm concerned, it's currently used in two places:

1. To represent resources (images, models, scenes, scripts, etc.) throughout the game's asset folders, and act as a unique identifier when loading these resources. This usage is mainly present in .tscn and other Godot resource files (using the res:// paths), but also in scripts that loads resources from code (they may access the user:// paths).
2. To represent raw files or directories in the folders, mainly in user://. This is mainly used in scripts that dynamically loads packs, and in loading/saving game states.

If we want to redesign this system, whether by implementing new protocols, or by doing a complete redesign ground-up, we must support the two use cases above somehow, and try not to deviate too far from it. Resource paths are, after all, strings, and should not be abused to replace function calls.

Things that, in my opinion, could be supported after extending resource paths (regardless of how):

• Other directories not covered by current res:// and user:// paths, like the executable's base directory, or the Steam cloud save directory.
• Other packs that are dynamically loaded into the engine, or need their own namespace because they are unique enough. How to reference inter- and intra-package resources remains as a problem.
• C# script/node classes, or classes from other extension languages, that are referenced in Godot resource files, by their unique names (or fully-qualified names). These are non-file resources, and might require creating a clear distinction between the resources loading parts and the file accessing parts. This case is handled by the class registry.
• Remote resources, such as an image accessible using HTTP or other protocols. One must be careful about how these resources are loaded, since web resources should be untrusted. This usage should be replaced with a download into a temporary directory and using the resource from there.

Things that, in my opinion, should not be supported even after we extend resource paths:

• SQL queries (mentioned in this issue). You could and should do this using a strongly-typed utility function, instead of an untyped path string.
• Things that are more efficiently done using function calls compared to going through strings and paths, and are not usually referenced in resource files.

Finally, I would like to hear about other participants' ideas on what we should and shouldn't do in resource paths, especially from experienced Godot developers.

[Frontrider]

C# script/node classes, or classes from other extension languages, that are referenced in Godot resource files, by their unique names (or fully-qualified names). These are non-file resources, and might require creating a clear distinction between the resources loading parts and the file accessing parts.

This one did get me to think, but I do not think this is going to be a use case in the future. We know that c# will become GDExtension so you will not reference the c# class through the resource loader, but via the class registry. This invalidates itself over time, and that also applies for other GDExtension languages. For script languages (eg lua) a file path is more than fine as a reference.

Secondly, what you need for this is the ability to pass down an arbitrary string to a handler that returns the resource. Which in turn opens up the resource loader to handle anything people can plug into it. Could be useful, but I'd rather not go there.

Other packs that are dynamically loaded into the engine, or need their own namespace because they are unique enough. How to reference inter- and intra-package resources remains as a problem.

Remote resources, such as an image accessible using HTTP or other protocols. One must be careful about how these resources are loaded, since web resources should be untrusted. It's trivial to replace this usage with a download into a temporary directory and use the resource from there.

I only see the VFS as a solution for these two, because both are basically just file paths. They map seamlessly to that. But external files do have issues when you try to pull them in, and those may apply to remote files as well. May be more feasible to have this piggyback on the PCK loading.

[Spartan322]

Accessing a remote resource really does not work for a VFS, a VFS doesn't define any expectations for its interface (and specifically dealing with a remote resource absolutely needs user awareness) and they're not really ever used for that that (and I'm not just referring to local remotes like a local network) and I most certainly wouldn't want to use it for that purpose, a protocol gives clear expectations, or in the least can designate clearer expectations to the user, it distinguishes itself from a generic filesystem which it may never act like or may even outright violate.

[nordyuki]

We know that c# will become GDExtension so you will not reference the c# class through the resource loader, but via the class registry. This invalidates itself over time, and that also applies for other GDExtension languages.

If that's what's going to happen, I would then recommend restricting the use case for resource paths I mentioned above to just local path-like structures (directories and compressed directories).

[Frontrider]

(and specifically dealing with a remote resource absolutely needs user awareness)

Godot itself has a system/expectation for it, namely the load/preload methods. It is already expected that Godot has to properly load in larger resources when you call load on it, and preload is how you get it to be available before it is needed. (I already resigned from the "remote resource" stance, but that bit may still be relevant)

[TML233]

Maybe we could separate FileAccesses and path prefixes? Making FileAccess only access its own designated file source, like from the os, from pck or from the zip, or wrapping other file sources, like FileAccessEncrypted and FileAccessCompressed wrapping other FileAccess. This allows more flexibility when handling resource mounting. You can combine mulitple sources of file in a single path prefix.