Callbacks please! Usable for async?

[minesworld]

pythonnet enables C# functions being called from Python. Very usefull, like redirecting stdout and stderr to whatever the developer likes...

Here a modified example from https://github.com/pythonnet/pythonnet/discussions/1794

using var scope = Py.CreateScope();

scope.Exec(
"""
import sys

class NetConsole(object):
    def __init__(self, writeCallback):
        self.writeCallback = writeCallback

    def write(self, message):
        self.writeCallback(message)

    def flush(self):
        pass

def setConsoleOut(writeCallback):
    sys.stdout = NetConsole(writeCallback)
"""
);

dynamic setConsoleOutFn = scope.Get("setConsoleOut");

var writeCallbackFn = (string message) => {
    if (message != "\n")
        Console.Write("[Py]: ");

    Console.Write(message);
};

setConsoleOutFn(writeCallbackFn);

scope.Eval("print('hello world')");

Callbacks

Is something like that possible in CSnakes? Too bad I have no clue how to call into dotnet from Python and havn't analyzed yet how pythonnet does it. But as pythonnet does "crazy" stuff, like accessing CPython object internals it might not be the preferred way anyway.

With CSnakes magic something like this should be possible:

import sys

class NetConsole(object):
    @CSnakes.Replaceble
    def write(self, message: str) -> None:
        raise Exception("not replaced with dotnet yet")

    @CSnakes.Replaceble
    def flush(self) -> None:
        raise Exception("not replaced with dotnet yet")

sys.stdout = NetConsole()

CSnakes SourceGenerator could generate the sources for a class like:

class PythonEngineer : IDisposable 
{
   // generic implementation, maybe also generic static functions usable as the developer likes
}

class NetConsoleEngineer : PythonEngineer 
{
   NetConsoleEngineer(PyObject c,           // python object which defines the class to replace 
                      string attribute);    // name of the class

  // does this initializer work for functions defined outside classes too, like c is a PyDict etc. ?

   NetConsoleEngineer InjectWrite(Action<PyObject, string> f) 
   { 
      /* get and store the original python function
         internal calls like 
           PythonEngineerCallable<PyObject, string>.CreateFor("def write(self, message: str) -> None:", f) 
         to construct the PyFunction as the callback
         replace the original function with the callback */ 

      return this; // enable call chains
   }
   NetConsoleEngineer EjectWrite() 
   { 
     /* restore the original python function and release and null its reference */
    return this;  
   }

  NetConsoleEngineer InjectFlush(Action<PyObject> f) { /* ... */ return this; }
  NetConsoleEngineer EjectFlush(PyObject c) { /* ... */ return this; }
}

which could be use like:

using var engineer = new NetConsoleEngineer(pyCode, "NetConsole");

engineer
  .InjectWrite( (self, message) => {
    Console.Write(message); 
  })
  .InjectFlush( (self) => {} );

// now the engineer instance holds references to the original python functions

// run the python code ... when finished:

engineer
  .EjectWrite() // restores and PyDecRef on python function
  .EjectFlush(); 

// dispose of engineer does PyDecRef on pyCode (after all not yet callbacks are ejected)

// this might look like more code than pythonnet, but it also sets the flush callback too...
// and as the pythonnet code does the cleanup automically (does it??), 
// so the Eject calls could be left away here  too

This might be not thread safe, having "side effects" (here: changing a class definition while other python code might use it at the same time). But this should be no argument against it as developer how will use it that way should know what they are doing (documentation...) .

Those developers who want to have no thread safety issues etc. could use PythonEngineerCallable<T>.CreateFor() to create a callback which is then used as shown in the pythonnet code...

As it is possible to give Python decorators parameters, the generated C# source could also:

• be able to call the original Python implementation, as InjectWrite(Action<PyObject, string, PyFunc> f);where the last PyFunc argument is the original func . Would enable to create "hybrid" Python/C# classes where C# can inherit the method and call the "super" function... for those who want that. Maybe automatically generated this way if the decorator is given an "override" (or something else C# like) argument?

To be able to prevent the dotnet runtime from catching outside variables etc. :

• to pass the "Engineer" instance usable as

  
  class NetConsoleEngineer 
  {
  void InjectWrite(Action<NetConsoleEngineer, PyObject, string> f); 
  }

class MyNetConsoleEngineer : NetConsoleEngineer { string SomeValue { get; set; } }

using var engineer = new MyNetConsoleEngineer(pyCode, "NetConsole") { SomeValue = "?" };

engineer.InjectWrite( (e, self, message) => { var myNetConsoleEngineer = e as MyNetConsoleEngineer; Console.Write(message + myNetConsoleEngineer.SomeValue); });

- pass any object given on intialization of the "Engineer" as

class NetConsoleEngineer { NetConsoleEngineer(PyObject c, string attribute, object anything); void InjectWrite(Action<object, PyObject, string> f); }

using var engineer = new NetConsoleEngineer (pyCode, "NetConsole", "?");

engineer.InjectWrite( (value, self, message) => { Console.Write(message + value); });

- bypass Mashalling and instead call and return nint values. That will be usefull if performance is critical and such CAPI calls are preferred, like

class NetConsoleEngineer { NetConsoleEngineer(PyObject c, string attribute, object anything); void InjectWrite(Action<object, nint, nint>); }

List messages = new();

using var engineer = new NetConsoleEngineer (pyCode, "NetConsole", messages);

engineer.InjectWrite( (messages, self, message) => { CAPI.PyIncRef(message); ((List)messages).Append(message); });

or any possible combination as requested by the decorators arguments. The provided CSnakes python decorators are dummys, just return the decorated function "as is"... 

Of course it should be possible to replace every python function defined anywhere. And static function calls like `PythonEngineerCallable<T>.CreateFor()` should enable created a Python function object from C# usable as a callback. When used in this way its up to the developer how to pass additional needed parameters on callback.

As I'm not a C# language expert I guess that the shown API could be better. **But the most important problem to solve is how CPython can call back into dotnet.** 

I guess it might be possible using CPython cffi or ctypes . **Best would be a "native" implementation provided as a module by the one who compiled the used CPython DLL . Maybe its possible to come up with something the maintainer of the python NuGet package will include or provide as an additional NuGet package...**

This way CSnakes would be better then pythonnet regarding compability. Looks like pythonnet does some "interersting" stuff, detecting the layout of the Python object struct. Fact is: pythonnet isn't compatible with 3.14 yet (which might be of other reasons).

## Generic async wrapper possible?

Having a callback from CPython to C# it should be possible to generate a async C# function as

class PythonEngineer { static PyFunc GeneratePyFuncWrappingAsyncPyFunc(PyFunc F, PyFunc pyResultCallback , PyFunc pyIsCancelledCallback) { // return a generated python function which: // "takes" the C# callbacks to an awaitable queue and cancellation token, either by arguments // or being called using a given locals dict etc. ...

// the generated python function does something like:

//    try: 
//       result = await F() # F can abort/return on if pyIsCancelledCallback() is True
//                          # will work as result = F() too... 
//                          # but await shown as a way to combine C# async with CPython async
//    except Exception:
//       pass # should be given back too?? maybe a Tuple of result,ex ??
//    finally:
//      pyResultCallback(result) # enqueue into awaitable C# queue

} }

class PythonExecutor { static async Task CallPythonAsync(PyFunc f, CancellationToken token=CancellationToken.None) { // should be a predefined as much as possible, like do not parse the function definition each time... var pyIsCancelledCallback = PythonEngineerCallable<Func>.CreateFor( "def f() -> bool:",
() => { return token.IsCancellationRequested(); // that would use automatic marshalling for the result } );

 var  resultQueue = new AsyncQueue<T>();

var pythonType = PythonEngineer.PythonTypeString(T); // if something like that is possible somehow..., 
                                                     // should give back "int" if the generics T is int

var pyResultCallback = PythonEngineerCallable<Action<T>>.CreateFor(
 $"def f(x: {pythonType}) -> None:",
 (x) => {
  resultQueue.Enqueue(x);
 }
);

var syncPyFunc = PythonEngineer.GeneratePyFuncWrappingAsyncPyFunc(f, pyResultCallback, pyIsCancelledCallback);

// // give the python interpreter the syncPyFunc to execute in a "fire and forget" manner //

// at this point we don;t need to hold any references to the python callbacks anymore // as they will be kept alive by python itself // // is that ensured by catching all exceptions? or will this leak if the python interpreter is brought down "somehow"? // in such case - are non leaking async calls possible anyway?

 return await resultQueue.DequeueAsync(token); // throw exception: await tuple and check for it...

} }

that used from C# like 

import asyncio import random

def async asyncAiFunc() -> int: await asyncio.sleep(20)

how to access and call pyIsCancelledCallback() for code running in a loop etc. ??

for a "hard" termination of a python thread (if executed in one, see later) there might

be a CPython API call which injects signals or so

but something like that would be implemented on the C# side on a different level...

return random.choice([ 42, 23 ])

var asyncPythonFunc = pythonObject.GetAttr("asyncAiFunc"); var result = await PythonExecutor.CallPythonAsync(asyncPythonFunc);

// would be "usefull" if arguments to the Python func could be passed too...

or with syntax sugar if possible: 

var result = await pythonObject.GetAttr("asyncAiFunc").CalledAsync(); // where does the etc. go ?

or internally by the SourceGenerator etc. ...

Some work to finish the design the API and implement it, but as if done as shown its only the "easy" part. Just the C# conventions for async are fullfilled. 

( CSnakes could restrict the usage in any way suitable for the implementation. Like telling the developer not to recurse async calls etc. or anything which will fail. )

More of a problem is that how Python "async" calls from C# into Python should work depends on the Python code: the `// give the python interpreter the syncPyFunc to execute in a "fire and forget" manner` part will be different and should be customizable somehow.

It could be as "simple" as generating a wrapper to be used with

`int Py_AddPendingCall(int (*func)(void*), void *arg)`

which is part of the Stable API and looks promising. OK - https://docs.python.org/3/c-api/init.html#sub-interpreter-support says that there is no guarantee that func is being called, but maybe it is called with a usable success rate? If it is not - the developer might provide a CancellationToken to set a timeout ...

Other might want to run the python function in a new python thread. Or queue it to a worker ... 

How that C# API could be made compatible with possible sub-interpreters is beyond my knowledge like how to call/address specific subinterpreter from the CAPI ?

**Much work to do, as a start anything how to call C# from Python is helpfull for me... C Code is welcome too, just have to get CPython compiled by myself to write a module.**

PS: Tony - if such C# DLL forth and back calls are in your Python Internals book - let me know. will buy it... otherwise money is an "issue" for me at the moment.

[minesworld]

Would be nicer if the generated class would be named NetConsoleClassEngineer...

There could/should be an option to the decorator how the generated class is named.

At least that might be usefull for those who do not generate the C# code automatically but call a tool which does so.

[minesworld]

Besides the missing GILusage it could be debated which parts might call the CAPI as "raw" (nint), IntPtror PyObject.

Maybe a layered approach gives the most options for all...

[minesworld]

var asyncPythonFunc = pythonObject.GetAttr("asyncAiFunc");
var result = await PythonExecutor<int>.CallPythonAsync(asyncPythonFunc);

is wrong as PythonExecutoruses staticfunctions that way.

To be able to pass the final wrapped python func to the interpreter in a way its compatible with the developers model, an instance of PythonExecutorshould be used.

That way PythonExecutorcould be subclassed so the CallPythonAsyncnot only does the "right thing" but even different usage models are possible by using the "correct" instance.

Beste design case would be the need to override just the " FireAndForget " method

[tonybaloney]

Generators are the best option for calbacks, async functions (coroutines) are fancy wrappers around the generator system in Python.

Utilizing C#.NET's Task system you would execute a callback from a generator whenever it yielded a value in a thread:

from typing import Generator

def example_generator(length: int) -> Generator[str, int, bool]:
    for i in range(length):
        x = yield f"Item {i}"
        if x:
            yield f"Received {x}"

    return True

var mod = Env.TestGenerators();
var generator = mod.ExampleGenerator(3);

var callback = new Action<string>(
    // This is the callback that will be called from the generator
    s => Assert.Equal(generator.Current, s)
);

// Wait for the generator to finish
var task = Task.Run(() =>
{
    while (generator.MoveNext())
    {
        // Simulate a callback
        callback(generator.Current);
        // Optionally send a value back to the generator
        generator.Send(10);
    }
    // Optionally return a value from the generator
    Assert.True(generator.Return);
});

I'll submit this test to main to demonstrate it and make sure it's tracked for regressions.

The Pending calls API is really unreliable and only gets called in the interpreter loop (so not when embedded like CSnakes) and in certain scenarios.