Add support for defining GDExtension classes in GDScript

kisg commented 1 year ago

Describe the project you are working on

Godot Engine Development

As usual, in case the proposal is accepted by the Core Team, our team at Migeran will provide the implementation and support it through the review process.

This proposal is a refinement of our previous proposal: #3369

We acknowledge the fact that removing the scripting interface is not feasible and also not desired at this time, because it provides a different, more dynamic extension mechanism compared to the GDExtension class-based approach.

Describe the problem or limitation you are having in your project

With the improvements in 4.x, GDScript has become a very usable language that can be used for more generic development than just simpler game logic scripts. In particular, many Godot addons are developed in GDScript, but those classes are not easily accessible from other languages (e.g. C++ or C#). The limitation is that currently classes defined in GDScript are "script classes" and can only be added to nodes through the script interface.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

We propose to add a new annotation:

@gdextension

to the GDScript language. When specified in the beginning of the script (similar to the @tool annotation), the behavior of the script changes as follows:

Scripts marked with @gdextension cannot be added to an object as a script. The editor will not allow it, and at runtime, it will result in a runtime error and the script will not be attached to the object.
If an existing script is converted to a GDExtension, by specifying the @gdextension annotation, every object that still has the script attached will be marked with the error.
The scripts marked with the @gdextension annotation will define a new GDExtension class (e.g. it will be added to the ClassDB), and will behave the same as if it was defined in C++ or any other GDExtension-based language (e.g. soon C#)
Such classes will behave like built-in classes in the Editor as well, e.g. they will appear in the class lists.
The --dump-extension-api output will also include the specification for these classes, so bindings can be generated for other GDExtension languages (C++, Python, soon C# ... etc.)
@tool cannot be specified together with @gdextension. A GDExtension class's code will always run, just like a @tool script's code does.

The proposed feature is strictly "opt-in" based, unless the @gdextension annotation is specified, the script's behavior does not change in any way.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, this is a core GDScript language and implementation feature.

Is there a reason why this should be core and not an add-on in the asset library?

No, this is a core GDScript language and implementation feature.

kisg commented 1 year ago

@reduz @dsnopek Could you please take a look?

dsnopek commented 1 year ago

I think this is up to the GDScript team, if this is something they want to support. It's certainly technically possible, but would probably be quite a bit of work to support both the GDExtension API and the usual script language APIs.

The only thing I'd say from the GDExtension side, is that we still have a number of usability issues to fix for people who are writing gameplay code in GDExtension, and GDScript is primarily used for writing gameplay code. This is mitigated by this being opt-in, so folks could only write engine code with @gdextension and have the best of both worlds. However, I think this feature would certainly be more desirable a couple release cycles from now, when GDExtension will be more mature.

kisg commented 1 year ago

I think this is up to the GDScript team, if this is something they want to support.

Oh, I assumed you were part of that team. Is there a list somewhere of the Core Team members? Checking the blog posts maybe @vnen would have been more appropriate. :)

It's certainly technically possible, but would probably be quite a bit of work to support both the GDExtension API and the usual script language APIs.

This is the work that we offer to do.

The only thing I'd say from the GDExtension side, is that we still have a number of usability issues to fix for people who are writing gameplay code in GDExtension, and GDScript is primarily used for writing gameplay code.

Many add-ons are being written in GDScript - many times as tool scripts because there is no better way currently.

The only usability issue, that I know of is that GDExtension classes behave like engine classes in the editor, but the same technique can be used as with @tool scripts. Do you know of any other issues?

This is mitigated by this being opt-in, so folks could only write engine code with @gdextension and have the best of both worlds.

Yes, that is the intent.

However, I think this feature would certainly be more desirable a couple release cycles from now, when GDExtension will be more mature.

I think the best way to make sure that a technology becomes mature is to use it and push its boundaries further. This is what we did with the XR module when we made OpenXR extension implementation available from GDExtensions. This is also the plan for the C# language support.

kisg commented 1 year ago

@Calinou could you please add topic:gdscript as well?

bitbrain commented 1 year ago

This would allow addon developers to maintain one version of any class and make it available via gdextension mechanism to other languages like C# (given C# moves to GDExtension) or Rust.

Currently, people fork popular addons and add a Rust or C# code version to be able to access classes that are usually exposed via GDScript.

ryanabx commented 1 year ago

Copying over a few of the potential use cases from the contributors chat:

Writing addons in GDScript Currently, when addon classes are written in GDScript, other languages must use interop functions such as .call() to call a function and retrieve its result, and .set() and .get() to set and get properties, instead of just accessing the properties or functions with a . . Additionally, because GDScript uses the scripting layer, other languages are not able to create a class that extends the functionality of a GDScript class. All these limitations tend to confuse end users of GDScript addons, and discourages using GDScript as an addon language. Allowing addon makers to use a @gdextension annotation or an equivalent measure (open to discussion) would let the class be registered in ClassDB, allowing it to be referenced the same way as normal Godot classes in other languages (C#, Rust, etc.)

True Object-Oriented patterns For users of the Object-Oriented paradigm, there is usually confusion when coming to GDScript and finding out that creating a script that extends a core class does not make a real class. Having an opt-in way to define explicitly that a GDScript class should be an extension class sounds like a fine idea to appease this crowd.

@kisg

As usual, in case the proposal is accepted by the Core Team, our team at Migeran will provide the implementation and support it through the review process.

Just to clarify, your team would be interested in implementing this feature if accepted? If so, that eases any potential worries from our GDScript side about workload when many teams are currently stretched thin.

@dsnopek @kisg

However, I think this feature would certainly be more desirable a couple release cycles from now, when GDExtension will be more mature.

I think the best way to make sure that a technology becomes mature is to use it and push its boundaries further. This is what we did with the XR module when we made OpenXR extension implementation available from GDExtensions. This is also the plan for the C# language support.

I can see this issue both ways. On the one hand, pushing the feature as early as 4.3 could give users ample opportunity to test and provide feedback on the change as soon as possible, potentially easing the hiccups of moving C# to GDExtension. RE: #7895

On the other hand, we don't want to push a feature too early and risk a bad reputation for GDExtension. If the GDExtension team is not ready for something commonly used like GDScript to have an easy way of using GDExtension before things are ironed out, then we should respect that. I think it's important that GDExtension be presented in its best form from the get go OR we make it very very clear that the annotation provides experimental access to GDExtension and that it's not representative of GDExtension when it's more mature.

Frontrider commented 1 year ago

Currently, when addon classes are written in GDScript, other languages must use interop functions such as .call() to call a function and retrieve its result, and .set() and .get() to set and get properties, instead of just accessing the properties or functions with a . . Additionally, because GDScript uses the scripting layer, other languages are not able to create a class that extends the functionality of a GDScript class. All these limitations tend to confuse end users of GDScript addons, and discourages using GDScript as an addon language. Allowing addon makers to use a @gdextension annotation or an equivalent measure (open to discussion) would let the class be registered in ClassDB, allowing it to be referenced the same way as normal Godot classes in other languages (C#, Rust, etc.)

Technically, we could say that it would cause situation where gdscript is called by c#/rust which is called by gdscript then Rust again. (constant context switching, then the dev can't see where the frames went) Breeding ground for a lot of small hacks.

But. This would still be a very good power to have. Make the Godot ecosystem a little bit more coherent through the common API. The gain outweighs the possible problems from misuse.

BenMcLean commented 1 year ago

We acknowledge the fact that removing the scripting interface is not feasible and also not desired at this time, because it provides a different, more dynamic extension mechanism compared to the GDExtension class-based approach.

Why is this? Seems like GDscript should be treated the same as every other scripting language and if GDExtension isn't dynamic enough then it should be expanded until it is.

Frontrider commented 1 year ago

We acknowledge the fact that removing the scripting interface is not feasible and also not desired at this time, because it provides a different, more dynamic extension mechanism compared to the GDExtension class-based approach.

Why is this? Seems like GDscript should be treated the same as every other scripting language and if GDExtension isn't dynamic enough then it should be expanded until it is.

GDScript is dynamically typed, but not as dynamic as something like Javascript (can't just append methods and fields to existing objects). IF it can do a check and restrict it to fully typed gdscript then I think there are no problems. (or if gdextension has/gets an "any" type)

bitbrain commented 1 year ago

What would happen in these cases?

Example 1: extending non-extension script

class_name SomeClass extends Node2D

@gdextension
class_name SharedClass extends SomeClass

Example 2: extending extension script

@gdextension
class_name SomeClass extends Node2D

class_name SharedClass extends SomeClass

also to confirm: with https://github.com/godotengine/godot-proposals/issues/6416 potentially introducing traits, would you be able to annotate these trait files as well?

Frontrider commented 1 year ago

Traits are not real types, so I'd say no. Unless GDExtension gets support for the idea of a "template/interface", which then needs to be understood by the target language somehow.

Exporting those sounds like a typechecking nightmare. Also, the trait's features are already present on the exported class, at worst it can just export that type and include the properties from all traits. (pretty much erase them)

AnidemDex commented 1 year ago

kisg commented 1 year ago

We acknowledge the fact that removing the scripting interface is not feasible and also not desired at this time, because it provides a different, more dynamic extension mechanism compared to the GDExtension class-based approach.

Why is this? Seems like GDscript should be treated the same as every other scripting language and if GDExtension isn't dynamic enough then it should be expanded until it is.

@BenMcLean Right now Godot has 2 different method for extending types:

the traditional class inheritance-based (also called the GDExtension classes, but that is slightly a misnomer in my opinion)
the attachment of "scripts" to an existing object instance. Note, that you can implement a "script" in any language, since the Scripting API was also exported over GDExtensions API, but using GDScript is the most common.

The 2 methods are not equivalent, and while I usually favor the first method, there are valid use cases for the second method, and there are community members, who like to structure their projects like that. That is why I wrote that the removal of this "scripting interface" is not feasible at this time.

kisg commented 1 year ago

Isn't this related to #3369 ?

It is, as it is stated in the beginning of the proposal. That proposal is more drastic, and not feasible (or desired) at this time.

kisg commented 1 year ago

What would happen in these cases?

Example 1: extending non-extension script
class_name SomeClass extends Node2D
@gdextension
class_name SharedClass extends SomeClass

This is not allowed, because regular scripts have no "real" type.

Example 2: extending extension script
@gdextension
class_name SomeClass extends Node2D
class_name SharedClass extends SomeClass
This will work the same way, as it works today with a GDExtension class implemented in C++.

also to confirm: with #6416 potentially introducing traits, would you be able to annotate these trait files as well?

I will have to look into that in more detail, but from a quick look, it would be possible to support traits in GDExtension scripts, hopefully without specially annotating those trait files. It is also a question of whether traits (or "mixins") would be added to the GDExtension class system or if this would remain a GDScript-only feature.

Shadowblitz16 commented 1 week ago

I don't get it whats the point in keeping scripts anyways? They are basically a one component version of unity's mono behaviors, except for the indirection part.

With the current way things work currently you get two cons..

one being you don't get a static interface
two being you can only have one per node

Code quality will be way worse if it's dynamically changeable because you can't ensure that the interface of the script and scene will be there.

You could export the dependencies and set them in the inspector, but if you change the script on the node at runtime you have to set those up everytime you change the script.

It would be a much better work flow if when you extend a type you actually make a new engine type. (aka gdextension) There could be a class called Behavior that we can extend for users that like the current way things are except then they could have more then 1 per node.

So you then get...

one way of providing a static interface
one way of attaching multiple behaviors per node

Here would be the difference...

# Not a script extends Node2D as a engine class
class_name MyNode2D extends Node2D

and

# A behavior for a node2D, multiple can be added per node.
class_name MovementBehavior extends Behavior[Node2D]

vs

# Not a script extends Node2D as a engine class
@gdextension
class_name MyNode2D extends Node2D

and

# A behavior for a node2D, multiple can be added per node.
class_name MovementBehavior2D extends Node2D

godotengine / godot-proposals