Implement a cross-language async/coroutine mechanism

Describe the project you are working on

Using GDExtension c++ binding in my own project.

I have implemented a class named "Future" to realize, this proposal just to describe my solution and try to make it become a core feature for crossing languages async mechanism.

Because if we use C++ with GDScript to implement game logic, we are likely to have a need for mutual waiting, unless we avoid this situation deliberately.

I'm not sure if it's just my personal need, I want some feedback to decide whether to add this feature into the godot source code or not.

Describe the problem or limitation you are having in your project

I need to binding c++ function which calls a async method from GDScript and wait for its completion, then return a result to GDScript. Obviously, I need to wait this bound method in GDScript, too.

In c++, I can get a Object * which class name is "GDScriptFunctionState" from an async method in GDScript, and wait its signal "completed" ( but "GDScriptFunctionState" is not a valid class in c++ binding of GDExtension).

In GDScript, there have not a elegent way to await a c++ bound methods.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Like GDScriptionFunctionState, we can implements a class "Future( may be we have a more appropriate name to name it)" which represents an async task, and hold a "completed" signal to return a result when its task completed.

So we can pass this object cross language and connect its "completed" signal to await an async task.

How to do some thing async this depend on specific language self, it is not a question here. In other words, "Future" is an interface/promise of an async task.

Now we can do something like this:

// In c++
// SomeClass
Ref<Future> echo_async(Object* p_obj, const StringName &p_async_method){
    Ref<Future> gds_task = p_obj->call(p_async_method);
    ERR_FAIL_NULL_V(gds_task, {});
    Ref<Future> cpp_task;
    cpp_task.instantiate();
    gds_task->connect("completed", callable_mp(this, &SomeClass::echo_gds_task).bind(cpp_task));
}

void echo_gds_task(const Variant &p_result, const Ref<Future>& p_cpp_task){
    UtilityFunctions::print("GDS task result: ", p_result);
    p_coo_task->emit("comopleted", p_result);
}

# in GDScript

func do_some_aync(future: Future) -> String:
    await get_tree().create_timer(1).timeout
    var result = "completed in gdscript"
    Future.completed.emit(result)    
    return result 

func get_future_in_gds() -> Future:
    var ret= Future.new()
    do_some_async(ret)
    return ret

func _ready() -> void:
    var some_obj = SomeClass.new()
    var echo_result = await obj.echo_async(self, "get_future_in_gds").completed
    print("C++ taks result: ",  echo_result)

Or use "GDScriptFunctionState" to simplify GDScript code:

// In c++
// SomeClass
Ref<Future> echo_async(Object* p_obj, const StringName &p_async_method){
    Variant gds_result = p_obj->call(p_async_method);
    if (Object* obj = Object::cast_to<Object>()) {
        if (obj->is_class("GDScriptFunctionState")){
            Ref<Future> cpp_task;
            cpp_task.instantiate();
            obj->connect("completed", callable_mp(this, &SomeClass::echo_gds_task).bind(cpp_task)));
        }
    }
    ERR_FAIL_V({});
}

void echo_gds_task(const Variant &p_result, const Ref<Future>& p_cpp_task){
    print_line("GDS task result: ", p_result);
    p_coo_task->emit("comopleted", p_result);
}

# in GDScript

func do_some_aync() -> String:
    await get_tree().create_timer(1).timeout
    var result = "completed in gdscript"
    return result 

func _ready() -> void:
    var some_obj = SomeClass.new()
    var echo_result = await obj.echo_async(self, "do_some_aync").completed
    print("C++ taks result: ",  echo_result)

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

This feature can be realized by just implementing "Future":

class Future: public RefCounted{
protected:
    static void _bind_methods(){
        ADD_SIGNAL(MethodInfo("completed". PropertyInfo(Variant::NIL, "result")));
        // Bind methods ...    
    }
public:
    bool is_completed() const { return completed;}
    Variant get_result() const {return result;}
    void make_complete(const Variant& p_result) {
        completed = true;
        result = p_result;
        emit_signal("completed", result);
    }

    // We can add some uility funcions here....
private:
      bool completed = false;
      Variant result;
};

Optionally, we can do some special operation to make GDScriptFunctionState to be contained in "extension_apis.json" when dumping extension apis. So we can realize awaitng an async GDScript method with typed safe in other languages. ( We still not need to expose "GDScriptFunctionState" neither editor or GDScript.)

Here have some addtional features:

C++20 coroutine:

I know godot is using c++17, so this is mainly used for c++ binding of GDExtension.

If we allow c++20, we can implement a specialized template of Ref and SignalAwaiter to make all thing elegent

#if __cplusepluse > 202002L
tempalte<>
class Ref<Future>{
      /* Declear all thing which same with Ref<T> ...*/
public:
    struct promise_type{
        // ....
    }

    struct awaiter{
        // ....
        Variant await_resume(const Variant &p_result) {
            future->make_completed(p_result);
            return p_result;
        }

        Ref<Future> future;
    }

    awaiter operator co_await() { return awaiter{*this}; }
};

struct SignalAwaiter {
    SignalAwaiter(Signal p_signal){
         Signal->connect(callable_mp(&SignalAwaiter::_on_signal_emitted));
    }

    bool await_ready() { return false; }
    void await_suspand(std::coroutine_handle<> h) { handle = h;}
    Variant await_result() { return result;}
private:
    void _on_signal_emitted(const Variant& p_result = {} ){
        result = p_result;
        handle.resume();
    }

    Variant result;
    std::corutine_handle<> handle;
};
#endif // __cplusepluse > 202002L

Now we can do this:

// C++
// SomeClass

Ref<Future> echo_async(Object * p_obj, const StringName& p_async_method){
        Ref<Future> gds_task = p_obj.call(p_async_method);
        //  "ERR_FAIL_NULL_COV" like "ERR_FAIL_NULL_V". but use "co_return" instead of "return".
        ERR_FAIL_NULL_COV(gds_task, Variant());
        auto res = co_await SignalAwaiter(Signal(gd_task, "completed"));
        UtilityFunctions::print("gds result: ", res);
        co_return res;
}

# GDScript

func do_some_aync() -> Future:
    # .....

func _ready() -> void:
    var some_obj = SomeClass.new()
    var echo_result = await some_obj.echo_async(self, "do_some_aync").completed
    print("C++ taks result: ",  echo_result)

We can use "GDScriptFunctionState" to get rid of Future in GDScript, too ( refer to previos section).

Typed Future

Like typed Array, we can implemente a "TypedFuture< ResultType >" inheirt from Future or just make "Future" as a tempalte( Future< ResultType >). Then make GDScript can await a "Future/TypedFuture" wihout ".completed", and make its returen value typed can be detected automatically like typed Array ( I don't know the detail of this mechanism).

# GDScript

func some_task() -> Future[Vector3]:
    # ....

func _ready() -> void:
    var result = await some_task() # Strip ".completed"
    print(result.length()) # "length()" can be prompted automatically .

If this enhancement will not be used often, can it be worked around with a few lines of script?

The core concept of this proposal is very simple, users can define a simple "Future" class like my solution by themselves.

Is there a reason why this should be core and not an add-on in the asset library?

This feature is about GDScript and any binding of GDExtension , it is best to be supported by core.

godotengine / godot-proposals