GDExtension: Add binding hints for low-level API functions (non-scripting)

[Ivorforce]

Describe the project you are working on

Godot internals.

Describe the problem or limitation you are having in your project

We have two systems for binding functions:

• gdextension_interface.h, where the 'low level GDExtension magic' happens, e.g. registering the extension or querying bound methods.
• ClassDB / extension_api.json / _bind_methods, where 'high level' core functions are bound.

The gdextension_interface.h is the more 'manual' of the two, because all functions have to be defined and adopted 'manually' by GDExtension maintainers.

Unfortunately, some functions are bound through gdextension_interface.h, although they could also be bound through the simpler function binding interface. For example:

• string.resize(5) (bound through gdextension_string_resize)
• image.ptrw() (bound through gdextension_image_ptrw)

There's a ton more examples; in fact a good portion of the gdextension_interface.h is binding low level methods. The quoted reason (as far as I know) for these functions not being bound through the ClassDB interfaces is that they should not be exposed to scripting. For example, image.ptrw() returns a uint8_t *. GDScript for example would not be capable of dealing with the pointer, and anyway, uint8_t * cannot be represented safely by Variant. As for string.resize(5), this is technically callable from GDScript, but it would be meaningless because individual characters couldn't be assigned, and until the new characters are fully set (including the terminating \0), the resized String is prone to crashing.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

I'm proposing to bind these functions through the ClassDB system anyway. These functions would be marked as "low level", such that they're callable only from compatible low level programming languages. For example, they would be bound to C++ and Rust GDExtensions, but not to GDScript and Python.

Binding through ClassDB would also tie these functions in better with ClassDB based systems, such as documentation, or defining functions from one GDExtension to be callable from other GDExtensions.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I am not sure how much work this would be to implement. I've looked into the code enough to believe it's possible, but not enough to make a concrete change list that would be required to do this. As a start, the functions would be available through ptrcall only (no call or validated_call), so it would not be required to wrap types with Variant.

The existing gdextension_interface.h functions would need to persist until Godot 5.0, for backwards compatibility.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No.

Is there a reason why this should be core and not an add-on in the asset library?

It's core.

[dsnopek]

Is the general idea that we can mark methods as being GDExtension-only, and not exposed to scripting?

That's something that I think would be useful to have.

For example, image.ptrw() returns a uint8_t *. GDScript for example would not be capable of dealing with the pointer, and anyway, uint8_t * cannot be represented safely by Variant.

We have support for dealing with pointers already, but it's not fully working. It's using GDExtensionPtr<uint8_t> (or GDExtensionConstPtr<uint8_t>), but there is a situation it's currently not working for, which I think is return values? Anyway, that would just be a matter of fixing that bug :-)

As a start, the functions would be available through ptrcall only (no call or validated_call), so it would not be required to wrap types with Variant.

I don't know that we need to restrict it to just ptrcalls?

[Ivorforce]

Is the general idea that we can mark methods as being GDExtension-only, and not exposed to scripting?

Yes, that's the general idea.

We have support for dealing with pointers already, but it's not fully working. It's using GDExtensionPtr (or GDExtensionConstPtr), but there is a situation it's currently not working for, which I think is return values? Anyway, that would just be a matter of fixing that bug :-) I don't know that we need to restrict it to just ptrcalls?

I suppose that's true; pointers probably account for most low level interactions. I can imagine there being other types that cannot be safely packed in a Variant, but I don't have a concrete example. So maybe this won't be an actual problem.

[dsnopek]

I don't know that we need to restrict it to just ptrcalls?

I suppose that's true; pointers probably account for most low level interactions. I can imagine there being other types that cannot be safely packed in a Variant, but I don't have a concrete example. So maybe this won't be an actual problem.

Even if we did restrict it to ptrcalls, they only ever pass pointers. So, even if there was some other data type, the ptrcall calling convention would only give a pointer to it anyway

[Naros]

Is the general idea that we can mark methods as being GDExtension-only, and not exposed to scripting?

@dsnopek @Ivorforce I also think this can serve a myriad of other benefits, too.

If I examine some GUI controls, such as GraphEdit, I notice a significant amount of protected class state that lacks getter and setter methods. This makes extending controls like this extremely painful with GDExtension, but would be a breeze with C++ modules. If we could provide getters and setters for all this state, but restrict them to GDExtension or low-level language access only, we could begin to extend core classes in GDExtension, just as you would in C++ modules.

For folks looking to make editor tools where we expect that there is often a 1:1 relationship between Editor and Plugins, having this be available is another way we could expose editor classes in a safe way for tooling but without compromising the goals of keeping that behavior restricted from scripting.

I know for me, being able to reuse or customize a Godot editor class rather than rewriting it from scratch would have saved me weeks worth of development on my plugin alone.