Auto generate C# bindings for gdextensions

[tbillington]

Describe the project you are working on

3D multiplayer game targeting Steam using C#.

Mid-port from Unity.

Uses Steam Matchmaking/Networking to allow easy joining/inviting between friends, and Steam Datagram Relay (SDR) to achieve connections between players without port forwarding etc.

Describe the problem or limitation you are having in your project

Many important features and 3rd party integrations are implemented as GDExtensions in Godot. In our project: GodotSteam and SteamMultiplayerPeer.

However, with a csharp focused godot project these extensions are inaccessible.

We've written a GDScript "bridge" for for GodotSharp that exposes and proxies calls from CSharp, but it's time consuming and error prone.

Not having access to community extensions is a big limitation of Godot currently (for csharp developers/ex-unity).

Describe the feature / enhancement and how it helps to overcome the problem or limitation

With the continued focus on supporting csharp as a "first class" option in Godot, it follows that being able to access GDExtensions would be of increased importance.

Generating csharp bindings on extension "import" when csharp use is detected would bring csharp feature parity closer.

Ideally this would not be something extension developers have to individually and manually add to their extensions. It should be effortless/automatic, not requiring opt in. Otherwise I fear it just won't happen (the current situation).

Optional Extra

Support the work to allow generating bindings not just for csharp, but for other language extensions like rust and swift.

Achieving arbitrary language binding generation would greatly increase the capability of Godot as a multi-langauge extension and fully unlock it for developers from other languages.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

1. GDExtensions would include their source when being packaged. (I assume source is required to generate csharp bindings)

2. In the editor, upon download from the Godot Asset Library, Godot will generate csharp bindings and place them in the project, potentially alongside the extension in addons.

3. Any required changes to .csproj or .sln would be made after generating the bindings.

If this enhancement will not be used often, can it be worked around with a few lines of script?

If users are not using Godot_mono, or in future don't have the csharp GDExtension enabled, generating bindings can be skipped.

Is there a reason why this should be core and not an add-on in the asset library?

I make this proposal here because csharp code is in core/the godot main repository.

[LauraWebdev]

Working with GDExtensions and C# certainly has been a bit cumbersome! Great proposal!

[TokisanGames]

However, with a csharp focused godot project these extensions are inaccessible.

Some of our C# users have helped us document how to use Terrain3D in C# and are using it without issue. See the "Detecting" and "Calling" sections. So GDextensions in C# may be inconvenient, but are not inaccessible.

[Bromeon]

Is this information available in ClassDB? As such, access via reflection (as mentioned in the Terrain3D example) would always be a workaround:

     var terrain = ClassDB.Instantiate("Terrain3D");
     terrain.AsGodotObject().Set("storage", ClassDB.Instantiate("Terrain3DStorage"));
     terrain.AsGodotObject().Set("texture_list", ClassDB.Instantiate("Terrain3DTextureList"));
     terrain.AsGodotObject().Call("set_collision_enabled", true);

However, it would probably still be nice if Godot provided an option to include more classes in the C# code generation, as it already has the entire codegen infrastructure in place. This would allow type-safe and idiomatic APIs rather than dynamic ones.

Incidentally related: ongoing discussion of porting C# itself to GDExtension: #7895.

[raulsntos]

GDExtension classes are registered in ClassDB and we can just retrieve the information needed from there. No need for the GDExtension source code.

As mentioned by @Bromeon, any GDExtension class can be consumed using the reflection-like APIs available in GodotObject (same as interop with GDScript classes). This means consuming GDExtensions is currently possible, but not ideal (because we lose type safety).

We do want to improve support for GDExtensions in C#, which includes creating them as well as consuming them.

If we were to use the codegen that exists in the editor (the bindings generator that generates the glue for GodotSharp), then there are a few things to keep in mind:

• The bindings generator retrieves everything from ClassDB and generates the glue, we would probably need to refactor it so we can allow generating only certain classes (like the ones registered through GDExtensions).
• We may be using internal APIs in the generated bindings source code. This would mean code generated in an user project would not compile.
• The generated bindings source code use unsafe code blocks which would require user projects to also enable unsafe code blocks and, in my opinion, this is not acceptable.

Also, the proposed workflow assumes GDExtensions are added from the Asset Library using the editor to trigger the C# bindings generation. How would this work if an user places a GDExtension in their projects from outside the editor? For example, if I have a project in GitHub that I download which includes GDExtensions already.

I think the bindings generation should be part of the MSBuild compilation, either using Source Generators or MSBuild tasks. I mentioned this before in https://github.com/godot-rust/gdext/issues/166#issuecomment-1574199958.

[ChrisAbra]

Would it make sense to have GDExtensions produce their own JSON API document similar to godot --dump-extension-api extension_api.json?

That way extensions could reference each other, rather than relying on the too-late ClassDB registration? Then its something build time for source generators to hook into and would be for more than just dotnet usage: extensions could reference each other.

[dsnopek]

Would it make sense to have GDExtensions produce their own JSON API document similar to godot --dump-extension-api extension_api.json?

That way extensions could reference each other, rather than relying on the too-late ClassDB registration? Then its something build time for source generators to hook into and would be for more than just dotnet usage: extensions could reference each other.

This needs some testing, but this may already work to some degree.

If you run godot --dump-extension-api in a project that has some GDExtensions in it, it will actually put information for the classes from the GDExtensions into the extension_api.json. In theory, you should be able to then recompile your GDExtensions with this new JSON and it'll generate classes for them. I haven't actually ever tried this 2nd part, though :-)

But even if it does work, this isn't an ideal workflow, since it requires building all your GDExtensions, generating the extension_api.json and then re-building again (so, building twice).

[ChrisAbra]

But even if it does work, this isn't an ideal workflow, since it requires building all your GDExtensions, generating the extension_api.json and then re-building again (so, building twice).

But in this instance of the C# bindings, the build is MSBuild right so assuming the Extensions are already built and registered we could use that as the source instead of ClassDB directly?

Edit: that's work thats already happening somewhere right? I've never been able to find where

[GeorgeS2019]

@raulsntos

suggestion: distribute C# binding as nuget

The following step could only be done by the nuget provider using a GDExtension C# binding addon

Ideally

godot --dump-extension-api --gdextension will create each extension_api-gdextension-Name.json for each of the GDExtensions. Obviously, it is expected only one e.g. c++ GDExtension is present

The bindings generator retrieves everything from ClassDB and generates the glue, we would probably need to refactor it so we can allow generating only certain classes (like the ones registered through GDExtensions).

The user only consumes the nuget, no c# binding codes that require unsafe code compilation will be involved

We may be using internal APIs in the generated bindings source code. This would mean code generated in a user project would not compile.

The generated bindings source code uses unsafe code blocks which would require user projects to also enable unsafe code blocks and, in my opinion, this is not acceptable.

[Zetelias]

Maybe we could modify these steps to work and do a script to automate them ? They don't work with me at least

[GeorgeS2019]

Update to this topic? When will this feature planned?

@DmitriySalnikov

Do you have a recommendation?

[DmitriySalnikov]

Do you have a recommendation?

Don't use ClassDB.Call/Set/Get in the core of the engine for C# bindings as I have.

Anyway, the bindings generator is already in the engine. It requires modifications, but this is most likely not a high priority. I'm definitely not going to do that.

[GeorgeS2019]

@mihe => it seems ClassDB.Call/Set/Get is not recemmended

GodotObject.Get, GodotObject.Set and GodotObject.Call https://github.com/godot-jolt/godot-jolt/issues/632#issuecomment-1750546745

[mihe]

it seems ClassDB.Call/Set/Get is not recemmended

As mentioned in the comment you linked, they're meant to be a workaround. They're pretty much the only alternative available from GDExtension right now, and as pointed out earlier in this thread the situation is similar to interop with GDScript, where you would also rely on GodotObject (or ClassDB I guess), as seen in the documentation.

[GeorgeS2019]

@DmitriySalnikov => elaborate?

Anyway, the bindings generator is already in the engine. It requires modifications

@raulsntos => care to share your latest view?

This issue has an impact on Unity Users using c# moving to Godot.

[DmitriySalnikov]

=> elaborate?

Just a little experiment that doesn't work

C# project, which should be placed in the root of your Godot project: [base extensions project.zip](https://github.com/godotengine/godot-proposals/files/14046372/base.extensions.project.zip) Before building it, you need to build an editor with the patch, run `GodotSharp.generate_extensions_bindings()` in the editor (e.g. via `EditorScript`) and try to build godot project. And don't forget to add `.godot\mono\extension_bindings\bin\Debug\GodotSharpExtensions.dll` to the project dependencies. Patch for 4.2.1: ```diff diff --git a/modules/mono/editor/bindings_generator.cpp b/modules/mono/editor/bindings_generator.cpp index 36fdda46255d6dcb256619eab1b7d3619c0e3776..9afd060bbac98652f8fbcee95f458bfc42fe04b8 100644 --- a/modules/mono/editor/bindings_generator.cpp +++ b/modules/mono/editor/bindings_generator.cpp @@ -1228,6 +1228,113 @@ Error BindingsGenerator::generate_cs_core_project(const String &p_proj_dir) { return OK; } +Error BindingsGenerator::generate_cs_extensions_project(const String &p_proj_dir) { + ERR_FAIL_COND_V(!initialized, ERR_UNCONFIGURED); + + Ref da = DirAccess::create(DirAccess::ACCESS_FILESYSTEM); + ERR_FAIL_COND_V(da.is_null(), ERR_CANT_CREATE); + + if (!DirAccess::exists(p_proj_dir)) { + Error err = da->make_dir_recursive(p_proj_dir); + ERR_FAIL_COND_V_MSG(err != OK, ERR_CANT_CREATE, "Cannot create directory '" + p_proj_dir + "'."); + } + + da->change_dir(p_proj_dir); + da->make_dir("Generated"); + da->make_dir("Generated/GodotObjects"); + + String base_gen_dir = path::join(p_proj_dir, "Generated"); + String godot_objects_gen_dir = path::join(base_gen_dir, "GodotObjects"); + + Vector compile_items; + + for (const KeyValue &E : obj_types) { + const TypeInterface &itype = E.value; + + if (itype.api_type != ClassDB::API_EXTENSION) { + continue; + } + + String output_file = path::join(godot_objects_gen_dir, itype.proxy_name + ".cs"); + Error err = _generate_cs_type(itype, output_file); + + if (err == ERR_SKIP) { + continue; + } + + if (err != OK) { + return err; + } + + compile_items.push_back(output_file); + } + + // Generate native calls + + StringBuilder cs_icalls_content; + + cs_icalls_content.append("namespace " BINDINGS_NAMESPACE ";\n\n"); + cs_icalls_content.append("using System;\n" + "using System.Diagnostics.CodeAnalysis;\n" + "using System.Runtime.InteropServices;\n" + "using Godot.NativeInterop;\n" + "\n"); + cs_icalls_content.append("[SuppressMessage(\"ReSharper\", \"InconsistentNaming\")]\n"); + cs_icalls_content.append("[SuppressMessage(\"ReSharper\", \"RedundantUnsafeContext\")]\n"); + cs_icalls_content.append("[SuppressMessage(\"ReSharper\", \"RedundantNameQualifier\")]\n"); + cs_icalls_content.append("[System.Runtime.CompilerServices.SkipLocalsInit]\n"); + cs_icalls_content.append("internal static class " BINDINGS_CLASS_NATIVECALLS_EXTENSION "\n{"); + + cs_icalls_content.append(MEMBER_BEGIN "internal static ulong godot_api_hash = "); + cs_icalls_content.append(String::num_uint64(ClassDB::get_api_hash(ClassDB::API_CORE)) + ";\n"); + + cs_icalls_content.append(MEMBER_BEGIN "private const int VarArgsSpanThreshold = 10;\n"); + + for (const InternalCall &icall : method_icalls) { + if (icall.api_type != ClassDB::API_EXTENSION) { + continue; + } + Error err = _generate_cs_native_calls(icall, cs_icalls_content); + if (err != OK) { + return err; + } + } + + cs_icalls_content.append(CLOSE_BLOCK); + + String internal_methods_file = path::join(base_gen_dir, BINDINGS_CLASS_NATIVECALLS_EXTENSION ".cs"); + + Error err = _save_file(internal_methods_file, cs_icalls_content); + if (err != OK) { + return err; + } + + compile_items.push_back(internal_methods_file); + + // Generate GeneratedIncludes.props + + StringBuilder includes_props_content; + includes_props_content.append("\n" + " \n"); + + for (int i = 0; i < compile_items.size(); i++) { + String include = path::relative_to(compile_items[i], p_proj_dir).replace("/", "\\"); + includes_props_content.append(" \n"); + } + + includes_props_content.append(" \n" + "\n"); + + String includes_props_file = path::join(base_gen_dir, "GeneratedIncludes.props"); + + err = _save_file(includes_props_file, includes_props_content); + if (err != OK) { + return err; + } + + return OK; +} + Error BindingsGenerator::generate_cs_editor_project(const String &p_proj_dir) { ERR_FAIL_COND_V(!initialized, ERR_UNCONFIGURED); @@ -2166,7 +2273,11 @@ Error BindingsGenerator::_generate_cs_method(const BindingsGenerator::TypeInterf cs_type = cs_type.substr(0, cs_type.length() - 2); } - String def_arg = sformat(iarg.default_argument, cs_type); + String def_arg_str = iarg.default_argument; + if (iarg.def_param_value.get_type() >= Variant::VECTOR2 && iarg.def_param_value.get_type() < Variant::PROJECTION) + def_arg_str = def_arg_str.replace("-inf", "real_t.NegativeInfinity").replace("inf", "real_t.PositiveInfinity"); + + String def_arg = sformat(def_arg_str, cs_type); cs_in_statements << def_arg << ";\n"; @@ -2313,7 +2424,7 @@ Error BindingsGenerator::_generate_cs_method(const BindingsGenerator::TypeInterf const InternalCall *im_icall = match->value; - String im_call = im_icall->editor_only ? BINDINGS_CLASS_NATIVECALLS_EDITOR : BINDINGS_CLASS_NATIVECALLS; + String im_call = im_icall->api_type == ClassDB::API_EXTENSION ? BINDINGS_CLASS_NATIVECALLS_EXTENSION : (im_icall->editor_only ? BINDINGS_CLASS_NATIVECALLS_EDITOR : BINDINGS_CLASS_NATIVECALLS); im_call += "."; im_call += im_icall->name; diff --git a/modules/mono/editor/bindings_generator.h b/modules/mono/editor/bindings_generator.h index aa4e5ea093bc4f80ab7e8b8659807b8065d40deb..d0b0db0cb4f7d4fca2b5c9e3ea46a9d9d1c9ea15 100644 --- a/modules/mono/editor/bindings_generator.h +++ b/modules/mono/editor/bindings_generator.h @@ -599,6 +599,7 @@ class BindingsGenerator { struct InternalCall { String name; String unique_sig; // Unique signature to avoid duplicates in containers + ClassDB::APIType api_type = ClassDB::API_NONE; bool editor_only = false; bool is_vararg = false; @@ -610,10 +611,11 @@ class BindingsGenerator { InternalCall() {} - InternalCall(ClassDB::APIType api_type, const String &p_name, const String &p_unique_sig = String()) { + InternalCall(ClassDB::APIType p_api_type, const String &p_name, const String &p_unique_sig = String()) { name = p_name; unique_sig = p_unique_sig; - editor_only = api_type == ClassDB::API_EDITOR; + editor_only = p_api_type == ClassDB::API_EDITOR; + api_type = p_api_type; } inline bool operator==(const InternalCall &p_a) const { @@ -820,6 +822,7 @@ class BindingsGenerator { public: Error generate_cs_core_project(const String &p_proj_dir); + Error generate_cs_extensions_project(const String &p_proj_dir); Error generate_cs_editor_project(const String &p_proj_dir); Error generate_cs_api(const String &p_output_dir); diff --git a/modules/mono/glue/GodotSharp/GodotSharp/Properties/AssemblyInfo.cs b/modules/mono/glue/GodotSharp/GodotSharp/Properties/AssemblyInfo.cs index da6f293871089dbd16a5231d691dd124077da796..19beb385a3753f4648225197f99a8afc8942409e 100644 --- a/modules/mono/glue/GodotSharp/GodotSharp/Properties/AssemblyInfo.cs +++ b/modules/mono/glue/GodotSharp/GodotSharp/Properties/AssemblyInfo.cs @@ -1,3 +1,4 @@ using System.Runtime.CompilerServices; [assembly: InternalsVisibleTo("GodotSharpEditor")] +[assembly: InternalsVisibleTo("GodotSharpExtensions")] diff --git a/modules/mono/glue/GodotSharp/GodotSharpEditor/GodotSharpEditor.csproj b/modules/mono/glue/GodotSharp/GodotSharpEditor/GodotSharpEditor.csproj index 3ceb3b64dc7ece60a774ae03803a7789b21c0d6f..c810c0fb150766314de83dcd657092d556f218b6 100644 --- a/modules/mono/glue/GodotSharp/GodotSharpEditor/GodotSharpEditor.csproj +++ b/modules/mono/glue/GodotSharp/GodotSharpEditor/GodotSharpEditor.csproj @@ -37,6 +37,7 @@ +