Script language support GDExtension

touilleMan commented 2 years ago

Describe the project you are working on

Describe the problem or limitation you are having in your project

Godot-Python is currently based on GDNative (and it PluginScript part). However the GDNative system is deprecated in favor of GDExtension (which seems very slick by the way 😄) which doesn't support declaring a 3rd party scripting language.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Exposing 3rd party scripting language seems somewhat similar to what is done with the XR interface: 3rd party developer should expose there XR extension as a class inheriting XRInterfaceExtension, then register it against the XRserver singleton exposed by Godot.

There is already a ScriptServer in Godot, however it is not register in ClassDB and hence cannot be access from GDExtension.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

There is two solutions I see:

1 - Modify `ScriptServer` to be exposed by ClassDB

This is the straightforward approach. However ScriptServer might be complicated to modify (for instance it has to deal with Script that is already exposed by ClassDB and ScriptInstance that is not, given script can also access ClassDB it may lead to complex situations such as a script instance manipulating it own object...)

2 - Create a proxy class to expose scripting language to ClassDB

This is in case modifying ScriptServer directly is not an option.

This approach is similar to GDNative's PluginScript :

a proxy class is exposed to ClassDB, it is available as a script_server singleton
a ScriptLanguageInterfaceExtension class is also exposed in ClassDB
3rd party addon implement a MyLanguage class inheriting ScriptLanguageInterfaceExtension and use the script_server singleton to register an instance of MyLanguage
script_server itself instantiate an internal proxy class (similar to PluginScriptLanguage) that store the MyLanguage instance and redirect it methods calls

Last but not least, #3369 totally change the way scripts works, so I guess this proposal should be accepted/refused before working on the current one ;-)

If this enhancement will not be used often, can it be worked around with a few lines of script?

This is a core change

Is there a reason why this should be core and not an add-on in the asset library?

This is a modification to have 3rd party scripting language as add-on

Zireael07 commented 2 years ago

Apart from Python, there are at least two other scripting languages supported this way - Rust and Lua

touilleMan commented 2 years ago

@Zireael07 are you sure Rust currently uses Pluginscript ? I have the feeling Rust binding works a lot like C++ binding (and hence GDExtension has already this usecase covered)

touilleMan commented 2 years ago

@gilzoide @Geequlim as developers of Godot-Lua and Godot-Javascript, do you have any opinion on this ? 😃

kisg commented 2 years ago

I would suggest a different solution, based on my proposal for GDScript: https://github.com/godotengine/godot-proposals/issues/3369

Based on this solution Python and other dynamic languages could simply expose the defined classes as GDExtension classes. The GDExtension type system is very dynamic (essentially very similar to e.g. Objective-C), and allows you to dynamically register any class that is defined in Python (or any other dynamic language).

To clarify: what I am proposing here does not require any change in the Godot core engine itself. My linked proposal goes a step further and removes the notion of "script" from the engine altogether by converting the GDScript runtime to also expose full-blown GDExtension classes.

This method would naturally work for any other scripting language as well:

For Lua: I would probably create a Sol3 based binding for the GDExtension API, and then create some helpers to make it easy to define GDExtension classes / objects at runtime.

For JavaScript: Facebook created a very nice, JS Engine independent API called JSI. They originally created it for React Native (I am using it in a commercial project), but it is a clean C++ library with implementations for JSC, Hermes and V8. Based on my experience with JSI so far, it should be straightforward to create a dynamic, bi-directional GDExtension binding, that is: consume any GDExtension class in JS / TS, and also define GDExtension classes in JS / TS.

If you are interested, I would be happy to discuss the technical details / contribute code to the project as my time permits.

sHaggYcaT commented 2 years ago

@kisg just FYI: Python is strongly typed - it's not like JavaScript. Once object is created, Python can't change its type in the runtime. Also, following pep483/pep484 Python can also define vars with specific type, using typing, like you, guys, do it in the GDscript, c#, c++, etc

Does it make sense somehow for design of this proposal?

Another question is: can we be sure that #3369 proposal is not affects a performance? For example(but maybe not limited), for Cpython the most expansive thing is usually launching a new runtime(so, it's good to have script launched, but waiting). Also, another expansive thing is sync requests, and it's reason why in python's devs has been done so many work around asyncio, and why in the modern high-compute applications people, if use python, use "multiprocessing" together with asyncIO.

touilleMan commented 2 years ago

@kisg

I would suggest a different solution, based on my proposal for GDScript: https://github.com/godotengine/godot-proposals/issues/3369

Your solution seems appealing given it brings closer first party scripts like GDScript (that are implemented within Godot and have access to the full C++ codebase) and 3rd party like Python/Js/Lua (that can only access the Godot classes that have been exposed by ClassDB).

To clarify: what I am proposing here does not require any change in the Godot core engine itself.

Once #3369 is implemented (which is a highly complicated task I think 😄 ), there still language integration within the IDE that is needed. Unlike compiled language like Rust where a .so extension contains the actual class, in a dynamic language the .so extension contains the support for the language, and Godot has then to use this to load resources (e.g. .py files for the Python extension).

Currently there is the LanguageServer singleton in Godot that is used to register new script languages (howere it is not available from GDExtension like I said). So no matter what we need to modify LanguageServer so that it is accessible from GDExtension, and introduce a ScriptLanguage class that should be inherited to implement the differents needed methods (so load a script file, but also format code, support debugging/breakpoints/profiling, code highlighting etc.)

Once this is done, GDScript would really be just like Python/Lua/Js from a interface point of view.

@sHaggYcaT

Once object is created, Python can't change its type in the runtime.

you underestimated Python power 😄

>>> class Dog:
...   def bark(self):
...     return "whooof !"
...
>>> class Duck:
...   def bark(self):
...     return "Oink !"
...
>>> scruffy = Dog()
>>> scruffy.bark()
'whooof !'
>>> scruffy.__class__ = Duck  # wtf !
>>> scruffy.bark()
'Oink !'
>>> type(scruffy)
<class '__main__.Duck'>

Another question is: can we be sure that https://github.com/godotengine/godot-proposals/issues/3369 proposal is not affects a performance?

Performance is mostly about method call dispatch here. If anything, I guess 3369 would increase performances a bit given it would simplify the dispatch by removing having to check if a script attached to the node has the method we are looking for.

kisg commented 2 years ago

Hi @touilleMan,

You are of course right, I completely ignored the fact that "some people" might want to use the Godot Editor to edit scripts other than GDScript :)

That said, I think that to support different languages (not just scripting, but also compiled languages like C++ or C#), the Godot Editor could use the Language Server Protocol from VS Code. This would allow us to reuse e.g. the Python language server that was created for VS Code, or the C++ / Rust language server, without reinventing the wheel. Debugging these languages could then work through the Debug Adapter Protocol, which is already implemented in Godot on some level, although probably only as a server to support GDScript debugging from VSCode.

About the runtime side: Actually my original idea was a bit different. With the new GDExtension based scripting language support, it would no longer be necessary to "attach" a script to a node. My idea was that when a project is loaded, it would load the script runtime like a standard GDExtension library, and then (using some logic) this script runtime would look up all the scripts inside the project and load them. This could be done using e.g. a main Python module, which then imports and registers all the other classes with the GDExtension runtime. At that point all the classes (nodes, resources, anything that can be defined in GDExtensions) would be available to the rest of the project, as if it was written in C++.

Of course, this could add significant delay to the startup times of an application. One idea to solve this, which might need to be added to GDExtensions, is support for lazy registration of classes (I have not checked yet if this is already supported): If a class is not in ClassDB, GDExtensions could query libraries that have registered a "lookup handler" class, to see if they can provide the requested class. If yes, then the script runtime (or this could even be useful for C++ in bigger projects) would register the requested class (and any dependant class to reduce roundtrips) in the ClassDB and return requested class.

My idea is that this would even work inside the editor, so as you work on your project in Python, when you save your code, it would be automatically reloaded, and the GDExtension classes re-registered with the editor, so you can directly use them e.g. in your scenes as you edit them.

What do you think?

touilleMan commented 2 years ago

the Godot Editor could use the Language Server Protocol from VS Code. [...]

Not reinventing the wheel is a good thing, so LSP support is appealing. However it is yet another big change for Godot that currently has it own API for language support in the editor.

So I would say this is not something to focus on for the moment (it can be added later on anyway), having a crude implementation of #3369 in Godot4.0 is much more interesting ;-)

One idea to solve this, which might need to be added to GDExtensions, is support for lazy registration of classes

What you are describing is actually quite similar to the ResourcesLoader node currently used in Godot (a ResouceLoader instance is registered against a resource manager singleton with the information about the file extension that can be loaded and a callback for the actual load). So I guess this approach will be kept (given it is required for GDScript no matter what.

touilleMan commented 2 years ago

as @aaronfranke noticed, https://github.com/godotengine/godot/pull/59553 totally implement solution 1) of this proposal 🥳 🍾 🎉

Calinou commented 2 years ago

Closing, as this is now implemented in master and will be in 4.0.

raniejade commented 2 years ago

What are the plans about adding documentation about this? TBH, this shouldn't be marked as done without proper documentation. How can developers use this without knowing how it works/where to start. With GDNative, only handful of people know how it works and the only way to figure out how something works is asking in discord or dig into github issues.

Calinou commented 2 years ago

What are the plans about adding documentation about this? TBH, this shouldn't be marked as done without proper documentation. How can developers use this without knowing how it works/where to start. With GDNative, only handful of people know how it works and the only way to figure out how something works is asking in discord or dig into github issues.

There's an issue tracking documentation progress: https://github.com/godotengine/godot-docs/issues/5618

We close proposals soon as they're implemented, as we prefer tracking documentation elsewhere. The Godot proposals repository is only about tracking feature implementation, not their documentation.

godotengine / godot-proposals