Type-safe API for external classes (GDScript, GDExtension, C#)

[AribYadi]

How would I use a class that was defined in other GDExtensions? For example, I want to use this GDExtension, but I don't know how I would create the Wasm class. Can I even do this?

[Bromeon]

Hello! 🙂

We haven't really tested this scenario yet. At the moment, "external classes" are not accessible in a type-safe way, but you can use the reflection API Object::call() to interact with any class registered under Godot. There are a few more functions that might be interesting: get()/set() for property access, the signal methods, etc. From Rust side you would probably need to represent instances as Gd<T> where T is the base class.

In the future, what I could imagine would be a feature similar to the proposal in https://github.com/godot-rust/gdnative/issues/200: an explicit way to declare external APIs. This should probably work for GDScript and other GDExtensions likewise.

[PgBiel]

So, I was giving this issue some thought, and wanted to share some ideas.

For example, if you have a GDScript class named "Something" and you have methods something(arg1), somethingsomething(arg2) , then, instead of accepting it as a Variant or Object or something else everywhere, and manually calling methods through .call and stuff (as is today), we could use this hypothetical feature to create a type which "mirrors" the public interface of that class. The feature could look something like this:

#[derive(bikeshed::ExternClass)]
#[class(base = RefCounted)]  // or something
struct Something;  // macro would probs add some opaque fields

#[godot_api(bikeshed::extern)]  // defaults to using the struct name as class
// #[godot_api(bikeshed::extern(rename = OtherClass))] idk
impl Something {
  // perhaps some const or something too

  #[func]
  fn something(arg1: Variant /* or some more specific type */) -> Variant;

  #[func(rename = somethingsomething)]
  fn something_something(arg2: i32  /* only expect ints */) -> bool; /* you're sure only bool is returned */
}

(Usage of bikeshed:: indicates "this syntax is not final in any way and just illustrates the idea")

Alternatively, a trait could be used instead of impl Something, something like impl bikeshed::ExternalSomething for Something, but not sure. But I think the syntax above already covers most of the basic functionality at least.

Several points to consider

• What should the macros do?
  • Perhaps take the existing GodotClass macros but remove the auto-registration calls - just assume they're registered?
  • Should they implement some special version of ToGodot / FromGodot for these classes?
  • Are class names unique enough in Godot such that only specifying the class name (like above) would allow us to directly know which Godot class this Rust external class would represent at runtime?
    • What happens in case of conflict between e.g. different Extensions or plugins?
    • Does Godot expose any way to refer to an extension's or GDScript plugin's classes directly, or list them?
• How could we check for class declaration errors...
  • ...at compile-time? Impossible pretty much I think (other than checking for obvious errors in the declaration itself, unrelated to what actually exists in the game)
  • ...when starting the game? Could we perhaps add some "post-all-classes-registration" hook to check if the classes actually exist? Is it possible to dynamically create a class, thus turning such "eager" validation approach possibly moot? (Or at least forcing there to be an option to regulate "lazy" vs "eager" validity checking?)
  • ...while running the game? Say you call, in Rust, a function from the external type, but turns out you provided an invalid signature (which should mostly equate to missing required parameters). What should happen? Panic? (What if the function returns e.g. a bool but you predicted it to return an int in the Rust-side signature? Panic?)
    • Could we perhaps check a method's signature to be correct "eagerly" through some form of reflection? (This probably wouldn't give us the return type though, but amount of parameters would be helpful)
• What if the external class has another external class as a base? Or, even, what if we don't know its exact base?
  • One would have to recursively "shim" parent classes.
  • Alternatively, we could allow "imprecise bases" to make it easier to work with external classes; that is, the declared base has to be an ancestor, but not a direct parent of the external class.
    • This would also make "shims" more resilient to changes in external code. For instance, perhaps the parent class changes often between versions of that extension.
• Is there any way to possibly cause unsafety while using or creating such types?
  • If we just add type checks with panics everywhere, perhaps that can be avoided?
  • What if we add some form of fallibility while using external classes? (Perhaps have a way for "shim'd" functions to return some special Result if they return some unexpected type, such as ConversionError from #467 ?)
• How would we deal with multiple structs representing the same class in different ways?
  • Could we perhaps allow proper conversion between them?
  • Is there some way to check statically if they point to the same ClassName, such that you'd be able to easily convert Gd<A> to Gd<B> if both are external classes for the same Godot-side class?
• Related to the above, how to cast a Gd<Variant/Generic thing> to Gd<A> where A is an external class?
  • This is probably related to whatever the macros implement for the struct, such as To/FromGodot or whatever.
  • We could use a new trait like the proposed bikeshed::ExternalClass which contains some associated const with the ClassName of the Godot class implemented by the Rust-side external class.
• (For the far, distant future) Could there also be a way to have some form of "codegen" for plugins, extensions or whatnot, based on info obtained through Godot somehow, or through some common API, much like is done for native Godot types?
  • Certainly not needed for an MVP of the feature, but worth the thought

Feel free to add more questions here :slightly_smiling_face:

[Lamby777]

Feel free to add more questions here 🙂

I'm strongly in favor of painting the bikeshed purple. Let's discuss this further in the next 39 hourly stand-up meetings. :3

ok nah but seriously, I think that part about codegen for non-native stuff would be pretty interesting, but I have no idea how codegen works for this project already so kinda don't know how hard (or impossible?) that'd be to even implement (safe|ergonomical)ly.

[Bromeon]

Apparently, if we run Godot with other extensions enabled and then run --dump-extension-api, it should output those as well, and generate code for them. This is in principle similar to what https://github.com/godot-rust/gdext/pull/530 did with the GodotSteam module (although that one is a module, not an extension).

So to some extent, this may be possible to achieve today with the custom-godot feature, when running in a working dir that contains other extensions. I haven't tested this though. But it's definitely not the most ergonomic, and it requires strict unidirectional dependencies.

IIRC the godot or godot-cpp repos contains such an issue, but I can't find it (surprise)...

TLDR, I see a few ways forward:

1. We suggest the "load Godot with other extensions" approach.
2. Godot one days provides native support to combine multiple extensions.
3. We declare an extern API manually.

But at this point, I don't see much than gdext can do apart from implementing 3).

[TitanNano]

I originally posted this in #724.

An alternative idea to what has been posted so far could be to allow the user to define traits for the foreign interfaces they expect, and then provide a proc-macro that generates a duck-type implementation against Gd<T>. This would be entirely independent of whether the expected interface was defined via another GDExtension, a GDScript, C# or any other scripting language.

// ------------- types and traits provided by gdext -------------

/// Wrapper around Gd<T> on which traits will be implemented. This type cannot be constructed or 
/// obtained by the consumer.
pub struct ForeignClass(Gd<Object>);

impl ForeignClass {
    /// runtime validation of function signatures. could also take a full MethodInfo.
    pub fn has_method(&self, name: StringName, args: &[VariantType], ret: VariantType) -> bool {
        let method = self.0.get_method_list().iter_shared().find(|method| {
            method
                .get("name")
                .map(|method| method.to::<StringName>() == name)
                .unwrap_or(false)
        });

        let Some(method_args) = method.as_ref().and_then(|method| method.get("args")) else {
            return false;
        };

        let method_args = method_args.to::<Array<Dictionary>>();

        let matches = args
            .iter()
            .enumerate()
            .map(|(index, arg)| {
                method_args
                    .try_get(index)
                    .map(|m_arg| &m_arg.get_or_nil("type").to::<VariantType>() == arg)
                    .unwrap_or(false)
            })
            .all(|item| item);

        let return_matches = method
            .and_then(|method| method.get("return"))
            .map(|r| r.to::<VariantType>() == ret)
            .unwrap_or(false);

        matches && return_matches
    }

    /// pass through to Object::call
    pub fn call(&mut self, name: StringName, args: &[Variant]) -> Variant {
        self.0.call(name, args)
    }
}

/// helper trait for casting the foreign class into a trait object.
pub trait ForeignClassObject {
    type Base: GodotClass;

    fn from_foreign(fc: Box<ForeignClass>) -> Result<Box<Self>, ForeignClassError>;
}

/// extension trait for Gd, can be merged into the Gd impl.
pub trait GdExt<T: GodotClass> {
    fn try_to_foreign<O>(&self) -> Result<Box<O>, ForeignClassError>
    where
        O: ForeignClassObject + ?Sized,
        T: Inherits<O::Base> + Inherits<Object>;
}

impl<T: GodotClass> GdExt<T> for Gd<T> {

    /// cast into a duck-typed trait object. The compatibility is checked at runtime. 
    /// This is the only way to get an instance of ForeignClass.
    fn try_to_foreign<O>(&self) -> Result<Box<O>, ForeignClassError>
    where
        O: ForeignClassObject + ?Sized,
        T: Inherits<O::Base> + Inherits<Object>,
    {
        let obj = self.clone().upcast();
        let foreign = Box::new(ForeignClass(obj));

        /// compatebility is currently checked inside this function but could be moved into a separate call.
        O::from_foreign(foreign)
    }
}

#[derive(Debug)]
pub enum ForeignClassError {
    MissingMethod(StringName, Vec<VariantType>, VariantType),
}

// ------------- trait and impls inside the consumer gdextension -------------

/// user declared foreign interface
trait ITestScript {
    fn health(&mut self) -> u8;

    fn hit_enemy(&mut self, enemy: Gd<Node3D>);
}

/// proc-macro generates an implementation of the trait for ForeignClass by calling its methods via Object::class
impl ITestScript for ForeignClass {
    fn health(&mut self) -> u8 {
        self.call(StringName::from("health"), &[]).to()
    }

    fn hit_enemy(&mut self, enemy: Gd<Node3D>) {
        self.call(StringName::from("hit_enemy"), &[enemy.to_variant()])
            .to()
    }
}

/// implementation of the helper trait to cast a Box<ForeignClass> into the correct trait object.
impl ForeignClassObject for dyn ITestScript {
    type Base = Node3D;

    fn from_foreign(fc: Box<ForeignClass>) -> Result<Box<Self>, ForeignClassError> {

        // validate health method exists and is correct
        if !fc.has_method("health".into(), &[], VariantType::Int) {
            return Err(ForeignClassError::MissingMethod(
                "health".into(),
                vec![],
                VariantType::Int,
            ));
        }

        // validate hit_enemy method exists and is correct
        if !fc.has_method("hit_enemy".into(), &[VariantType::Object], VariantType::Nil) {
            return Err(ForeignClassError::MissingMethod(
                "hit_enemy".into(),
                vec![VariantType::Object],
                VariantType::Nil,
            ));
        }

        // cast once everything has been verified.
        Ok(fc as Box<Self>)
    }
}

[Bromeon]

An alternative idea to what has been posted so far could be to allow the user to define traits for the foreign interfaces they expect, and then provide a proc-macro that generates a duck-type implementation against Gd<T>. This would be entirely independent of whether the expected interface was defined via another GDExtension, a GDScript, C# or any other scripting language.

I also think this is the way to go, it's conceptually similar to https://github.com/godot-rust/gdnative/issues/200 (linked above) but with attribute instead of function-like proc-macros. The important point here is that the same mechanism can be used for GDScript, C# and other GDExtension bindings.

Ideally we can keep foreign types as close as possible to native ones, e.g. when it comes to using them within Gd<T>. It would feel natural if foreign classes behaved similarly to engine-provided ones like Node3D.

So, I'm not sure if the user-facing API should really be a trait -- dyn Trait objects are unwieldy to use in Rust, and similar arguments apply as in #278. I was first thinking that the user could declare a trait but then the macro would generate an actual struct -- which works, but may not be enough.

We also need to consider that there are different symbols that a class may expose:

• functions
• virtual functions (overridable by scripts)?
• constants
• properties
• signals
• possibly nested types/enums?

And in an ideal world, declaring a foreign/external class is similar to declaring an own type, so the constructs are not entirely different.

// Syntax subject to change, just illustrating idea
#[derive(GodotClass)]
#[class(foreign, base=Node3D)] // Base must be known. Also, what if it's another foreign class?
struct TestScript {
     // Properties:
     #[var]
     name: GString, // generates set_name, get_name

     // Signals:
     #[signal]
     on_attack: Signal<...>, // syntax TBD
}

#[godot_api(foreign)]
impl TestScript {
    // Functions:
    #[func]
    fn health(&mut self) -> u8;
    #[func]
    fn hit_enemy(&mut self, enemy: Gd<Node3D>);

    // Constants:
    #[constant]
    const DEFAULT_HP: i32;
}

Or, with trait:

#[godot_api(foreign)]
trait TestScript {
    // Some way to declare properties/signals, if there is no struct?

    // Functions:
    fn health(&mut self) -> u8;
    fn hit_enemy(&mut self, enemy: Gd<Node3D>);

    // Constants:
    const DEFAULT_HP: i32;
}

It might also be worthwhile to require unsafe in some way, as Rust cannot check memory safety in other languages. Syntactically, unsafe struct is not allowed, but unsafe mod, unsafe trait and unsafe impl {Trait} for {Type} are. Otherwise, it can also be a keyword inside attributes, like #[class(unsafe_foreign)].

[TitanNano]

Ideally we can keep foreign types as close as possible to native ones, e.g. when it comes to using them within Gd. It would feel natural if foreign classes behaved similarly to engine-provided ones like Node3D.

From my mediocre understanding, it looks like defining foreign types could be defined pretty much the same way as engine types, only using a slightly different way of dispatching the method calls. I just find it strange to declare a struct and impl block that does not have function bodies. It looks like a trait, but it's not, which makes it quite alien. I completely get the desire to make it fit in with the engine declared and user declared types, though.

#[class(foreign, base=Node3D)] // Base must be known. Also, what if it's another foreign class?

If base is another foreign type it could either be used, if it has been defined in rust, or any of the engine types in its inheritance chain could be used if the foreign base is not defined. That shouldn't really matter for interfacing with the class and only affect how it can be casted.

We also need to consider that there are different symbols that a class may expose:

functions
virtual functions (overridable by scripts)?
constants
properties
signals
possibly nested types/enums?

I think this should be kept in line with how the generated engine types work. Currently, that would be everything is a method and e.g. properties need a getter and setter method.

[fpdotmonkey]

Is there even a difference between foreign and engine types, so far as gdext is concerned? If they're both being generated from extension_api.json and Godot is able to provide all the pointers to useful functions, it doesn't seem they're much different beyond provenance. Maybe they could be generated into their own module, perhaps godot::foreign_extensions::{extension1, extension2, ...}.

[Bromeon]

Is there even a difference between foreign and engine types, so far as gdext is concerned?

Not that I'm aware of. I don't know the specifics of Godot's C# integration, but I'd expect function calls routed through the engine to just work, independently of the language they're defined in :slightly_smiling_face:

If we can generate code, a dedicated module sounds like a great idea, however I'm not sure if the JSON exposes which extension a certain class comes from :thinking:

[TitanNano]

Is there even a difference between foreign and engine types, so far as gdext is concerned?

Foreign types are not necessarily part of the extension_api.json if they are defined in a script (differentiating between foreign extension types and script types is an option, though). They can also change independent of the engine version (compared to core types), so the information in extension_api.json is not reliable.

[TitanNano]

however I'm not sure if the JSON exposes which extension a certain class comes from 🤔

It doesn't, example from a gdext class that shows up in the JSON:

{
  "name": "TerrainBuilderFactory",
  "is_refcounted": true,
  "is_instantiable": true,
  "inherits": "RefCounted",
  "api_type": "extension",
  "methods": [
    {
      "name": "create",
      "is_const": true,
      "is_vararg": false,
      "is_static": false,
      "is_virtual": false,
      "hash": 997836395,
      "return_value": {
        "type": "TerrainBuilder"
      },
      "arguments": [
        {
          "name": "tilelist",
          "type": "Dictionary"
        },
        {
          "name": "rotation",
          "type": "TerrainRotation"
        },
        {
          "name": "materials",
          "type": "Dictionary"
        }
      ]
    }
  ]
}

[fpdotmonkey]

They can also change independent of the engine version

I don't think this matters per se; if you're already using custom-godot, you're going to take whatever Godot version you get.

Re: extension_api.json issues, this seems a deficiency of upstream that would be relatively easy to solve. It already says that it's an extension (api_type: "extension"), it should probably also have a key extension_name. I would imagine they'd be amenable to adding that for 4.4.

[fpdotmonkey]

But now an additional idea: could this be done without custom-godot, perhaps by interfacing with the gdextension binary directly?

[Bromeon]

Re: extension_api.json issues, this seems a deficiency of upstream that would be relatively easy to solve. It already says that it's an extension (api_type: "extension"), it should probably also have a key extension_name. I would imagine they'd be amenable to adding that for 4.4.

I agree, such metadata would be very useful. Either a name that an extension itself declares, or the file name of the .gdextension file.

---

Foreign types are not necessarily part of the extension_api.json if they are defined in a script (differentiating between foreign extension types and script types is an option, though). They can also change independent of the engine version (compared to core types), so the information in extension_api.json is not reliable.

But now an additional idea: could this be done without custom-godot, perhaps by interfacing with the gdextension binary directly?

Very good point about the scripts. I don't think extension_api.json contains GDScript (or other script) definitions, but this is something we should test. Same with C#.

As I see it, there are two ways to query Godot for the available APIs:

1. CLI command --dump-extension-api which generates the JSON that we currently use.
2. Runtime reflection, such as ClassDB.get_class_list(). This should also include GDScript/C#/...

For both cases, the Godot binary is needed for codegen (like in api-custom, which is the new name of custom-godot). In the case of JSON, it is however possible to temporally separate the JSON extraction and the code generation, which can be interesting for CI scenarios. But it may be less useful if C#/GDScript aren't covered.

---

There is also the related issue:

• https://github.com/godot-rust/gdext/issues/615

[fpdotmonkey]

There's also an option for build-time reflection by creating a headless Godot app that that writes the API of an extension named from an arg or stdin to stdout.