Publishing `PyAwaitable`

[ZeroIntensity]

Have you searched for this discussion already?

• [X] I have already searched for this topic!

Content

After discussing this topic here and here, the reference implementation of PyAwaitable (included in _view) should be published for use as an external library. I'm not sure whether this should go under view.py as a special release, or as a separate repository. However, I think view.py should document it on it's own docs, and not under a different domain.

The following needs to be done, according to discussion:

• Document the full API of PyAwaitable
• Rename the API to use a different prefix, as requested by one of the core developers. I'm thinking awaitable_ will work.
• Write a specification for PyAwaitable, in order to create a PEP later on.
• Explain how it works, as that will be much needed if a PEP is to be created.

Some explanation as to how PyAwaitable works was already written for #19, so that can likely be used for the future.

Additional Info

No response

[AraHaan]

Instead of changing the prefix outright I suggest having a define named USE_PY_PREFIX to force Py prefix as used currently for those of us who prefer to use that prefix in their code so we can force it (and also perhaps for it to be header only).

[ZeroIntensity]

That's already been done. Most of this issue has been completed already in a private repository, I just need to get around to publishing it. In terms of design, the API is pretty much set in stone at this point, as the documentation is already written.

[AraHaan]

Nice, I would love if that private repository could be made public soon so people can submodule it in their codebases and start using it in their code.

Also I am interested how one can implement something like this using that api as well:

async def __get_result(query: str) -> sqlite3.Row | None:
    """Internal API. DO NOT USE."""
    async with asqlite.connect('Bot.db') as connection:
        async with connection.cursor() as cursor:
            await cursor.execute(query)
            result: sqlite3.Row | None = await cursor.fetchone()

    return result

async def __get_results(query: str) -> list[sqlite3.Row] | None:
    """Internal API. DO NOT USE."""
    async with asqlite.connect('Bot.db') as connection:
        async with connection.cursor() as cursor:
            await cursor.execute(query)
            results: list[sqlite3.Row] | None = await cursor.fetchall()

    return results

async def __add_or_delete_item(query: str):
    """Internal API. DO NOT USE."""
    async with asqlite.connect('Bot.db') as connection:
        async with connection.cursor() as cursor:
            await cursor.execute(query)
            await connection.commit()

async def __add_or_delete_items(query: str, values: list):
    """Internal API. DO NOT USE."""
    async with asqlite.connect('Bot.db') as connection:
        async with connection.cursor() as cursor:
            await cursor.executemany(query, values)
            await connection.commit()

Where asqlite is a version of sqlite3 that works well in async code as the sync version is not well suited for it and for which asqlite is written in pure python itself. Also if that code can be migrated to C I could also migrate most of my async code in my discord.py bot on the core parts of it (except __init__.py and __main__.py as I like having it as a local editable "install" that is in package form) and have it's command parts remain as pure python code.

[ZeroIntensity]

PyAwaitable does not have any extra support for async with or async for yet, but you can still call the actual protocol methods. In your case, it would look something like this (pseudo-code, I'm too lazy to do all the shenanigans with attribute lookup and vectorcall right now):

static int body(PyObject* awaitable, PyObject* cursor) {
    PyObject* query;
    PyObject* values;
    awaitable_unpack(awaitable, &query, &values);
    awaitable_await(awaitable, cursor.executemany(), NULL, NULL);
    awaitable_await(awaitable, cursor.commit(), NULL, NULL);

    return 0;
}

static int callback(PyObject* awaitable, PyObject* connection) {
    PyObject* ctx = connection.cursor()
    PyObject* aenter = connection.__aenter__();
    awaitable_await(awaitable, aenter, body, NULL);
    return 0;
}

static PyObject* add_or_delete_items(PyObject* self, PyObject* query, PyObject* values) {
    PyObject* awaitable = awaitable_new();
    PyObject* connection = asqlite.connect("Bot.db")
    PyObject* aenter = connection.__aenter__() // this is a coroutine!
    awaitable_await(awaitable, aenter, callback, NULL);
    awaitable_save(awaitable, 2, query, values, NULL);

    return awaitable;
}

I suppose that I could add something like a awaitable_with that stores the __aexit__ for later. That will be after an initial release, though.

[AraHaan]

Also I noticed that your module init function in the _view extension could make use of the things I have in my own project's defines.h file:

/*
 * defines.h
 */
#pragma once
/*
 * for Visual Studio when the python include folder is not in
 * the include path by default for VS2022 when opening without
 * a project file. Uncomment these when doing such.
 */
// #include <C:/Program Files/Python311/include/Python.h>
// #include <C:/Program Files/Python311/include/structmember.h>
// #include <C:/Program Files/Python311/include/datetime.h>
#include <Python.h>
#include <structmember.h>
#include <datetime.h>
#include <stdbool.h>

/* inline helpers. */
static inline void _SetAllListItems(PyObject *all_list, int count, ...) {
  va_list valist;
  va_start(valist, count);
  for (int i = 0; i < count; i++) {
    PyList_SET_ITEM(all_list, i, PyUnicode_FromString(va_arg(valist, const char *)));
  }
  va_end(valist);
}

static inline PyObject *_DecrefModuleAndReturnNULL(PyObject *m) {
  Py_XDECREF(m);
  return NULL;
}

/* Allows to create instances of other C extension module types within another type in the very same C extension code. */
static inline PyObject *_PyType_CreateInstance(PyTypeObject *type, PyObject *args, PyObject *kwargs)
{
  PyObject *instance = type->tp_new(type, Py_None, Py_None);
  if (type->tp_init(instance, args, kwargs) != 0)
  {
    PyErr_Print();
    Py_XDECREF(instance);
    return NULL;
  }

  return instance;
}

static inline PyObject *_PyObject_GetCallableMethod(PyObject *obj, const char *name)
{
  PyObject *method = PyObject_GetAttrString(obj, name);
  if (!PyCallable_Check(method))
  {
    Py_XDECREF(method);
    return NULL;
  }

  return method;
}

/* defines needed in the module init function. If only they were in the Python.h include file. */
#define PY_CREATE_MODULE(moduledef) \
PyObject *m; \
m = PyModule_Create(&moduledef); \
if (m == NULL) \
  return NULL
#define PY_CREATE_MODULE_AND_DECREF_ON_FAILURE(moduledef, decref) \
PyObject *m; \
m = PyModule_Create(&moduledef); \
if (m == NULL) { \
  Py_XDECREF(decref); \
  return NULL; \
}
#define PY_TYPE_IS_READY_OR_RETURN_NULL(type) \
if (PyType_Ready(&type) < 0) \
  return NULL
#define PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(name, type) \
if (PyModule_AddObject(m, Py_STRINGIFY(name), Py_NewRef(&type)) < 0) \
  return _DecrefModuleAndReturnNULL(m)
#define PY_ADD_ALL_ATTRIBUTE(count, ...) \
/* Create and set __all__ list. */ \
PyObject* all_list = PyList_New(count); \
if (all_list == NULL) { \
  Py_XDECREF(m); \
  return NULL; \
} \
_SetAllListItems(all_list, count, __VA_ARGS__); \
/* Set __all__ in the module. */ \
if (PyModule_AddObject(m, "__all__", all_list) < 0) { \
  Py_XDECREF(all_list); \
  return _DecrefModuleAndReturnNULL(m); \
} \
return m

/* Allows to create instances of other C extension module types within another type in the very same C extension code. */
#define PyType_CreateInstance(type, typeobj, args, kwargs) ((type *)_PyType_CreateInstance(typeobj, args, kwargs))
#define PyObject_GetCallableMethod(type, name) _PyObject_GetCallableMethod((PyObject*)type, name)

Also in my C extension I create the m object (using PY_CREATE_MODULE) after checking if the types are ready so then there would be no need to recref at all if the types are not actually ready which allowed m to plac thm inside of a macro that can be usable for all defined extension type objects.

[ZeroIntensity]

Feel free to make a PR with these changes.

[AraHaan]

Feel free to make a PR with these changes.

Alright I will, also in x64 compilation mode might want to look in changing int on offset and the loop on aw_values_size to Py_ssize_t because when compiling for 64 bit mode it is defined as long long and not int so it results in a "possible loss of data." warning. As such I will PR a fix for that first.

[ZeroIntensity]

@AraHaan are you familiar with C++'s std::future? Some bindings for PyAwaitable could be nice to C++ developers, but unfortunately I don't know anything about C++ asynchronous code. This goes for Rust as well.

Ideally, PyAwaitable could be used to power some async support in PyO3 and PyBind11 (IIRC PyBind has an open issue for adding async somewhere).

[AraHaan]

@AraHaan are you familiar with C++'s std::future? Some bindings for PyAwaitable could be nice to C++ developers, but unfortunately I don't know anything about C++ asynchronous code. This goes for Rust as well.

Ideally, PyAwaitable could be used to power some async support in PyO3 and PyBind11 (IIRC PyBind has an open issue for adding async somewhere).

I have not tried std::future much at all yet. But I can agree with that. But yes I agree on using this to power async support in python extension modules.

I even just now implemented this one as a test on my C extension to my discord bot with success:

static PyObject *_capi_foo(PyObject *mod, PyObject *args) {
  PyObject *awaitable = PyAwaitable_New();
  printf("bar\n");
  return awaitable;
}

And the test from the console:

Python 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import _capi
>>> await _capi.foo()
  File "<stdin>", line 1
SyntaxError: 'await' outside function
>>> import asyncio
>>> asyncio.new_event_loop().run_until_complete(_capi.foo())
bar
>>> async def run_foo():
...     await _capi.foo()
...
>>> asyncio.new_event_loop().run_until_complete(run_foo())
bar
>>> exit()

[ZeroIntensity]

I have a few people I could ask then. The main issue that I'm having trouble fixing before I can release it is that I can't figure out how to link against the PyAwaitable extension module. If you could help me figure that out, I can release it later today. See the problem on Stack Overflow (in my case, _module refers to the built _pyawaitable extension).

[AraHaan]

I have a few people I could ask then. The main issue that I'm having trouble fixing before I can release it is that I can't figure out how to link against the PyAwaitable extension module. If you could help me figure that out, I can release it later today. See the problem on Stack Overflow (in my case, _module refers to the built _pyawaitable extension).

Sadly the only way I can think of is look at how python3XX.dll is linked on the built in extensions. Only way I can think of is to static link in the awaitable.obj from compiling the c file and have the extension author who is linking it in register the types into their module using the header file.

Example this is my _capi.c file:

/* defines for module init. */
#include "../include/defines.h"

/* for defining awaitable functions in C extension code. */
#include "../include/awaitable.h"

/* Entity include files. */
#include "../include/Activity.h"
#include "../include/DiscordToken.h"
#include "../include/Emoji.h"
#include "../include/ExpiringVerification.h"
#include "../include/Guild.h"
#include "../include/Role.h"
#include "../include/Tier.h"
#include "../include/TierDescription.h"

/* Task include files. */
#include "../include/AsyncExecutorTask.h"
#include "../include/AsyncBackgroundTask.h"

static PyObject *_capi_foo(PyObject *mod, PyObject *args) {
  PyObject *awaitable = PyAwaitable_New();
  printf("bar\n");
  return awaitable;
}

static PyMethodDef _capi_members[] = {
  { "create_activity", (PyCFunction)_capi_create_activity, METH_O, NULL },
  { "create_discordtoken", (PyCFunction)_capi_create_discordtoken, METH_O, NULL },
  { "create_emoji", (PyCFunction)_capi_create_emoji, METH_O, NULL },
  { "create_expiringverification", (PyCFunction)_capi_create_expiringverification, METH_O, NULL },
  { "create_guild", (PyCFunction)_capi_create_guild, METH_O, NULL },
  { "create_role", (PyCFunction)_capi_create_role, METH_O, NULL },
  { "create_tier", (PyCFunction)_capi_create_tier, METH_O, NULL },
  { "create_tierdescription", (PyCFunction)_capi_create_tierdescription, METH_O, NULL },
  { "foo", (PyCFunction)_capi_foo, METH_NOARGS, "Test function to test out defining awaitable functions in C extension code." },
  { NULL } /* Sentinel */
};

static struct PyModuleDef _capi = {
  .m_base = PyModuleDef_HEAD_INIT,
  .m_name = Py_STRINGIFY(_capi),
  .m_doc = "Discord Bot C API that includes async background tasks and database entity types.",
  .m_size = -1,
  .m_methods = _capi_members,
  .m_slots = NULL,
  .m_traverse = NULL,
  .m_clear = NULL,
  .m_free = NULL,
};

PyMODINIT_FUNC PyInit__capi(void) {
  /* Entity Type Objects. */
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyActivityObject);
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyDiscordTokenObject);
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyEmojiObject);
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyExpiringVerificationObject);
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyGuildObject);
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyRoleObject);
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyTierObject);
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyTierDescriptionObject);
  /* Async Background Tasks Type Objects. */
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyAsyncExecutorTaskObject);
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyAsyncBackgroundTaskObject);
  /* Types for C defined awaitable python functions. */
  PY_TYPE_IS_READY_OR_RETURN_NULL(PyAwaitable_Type);
  PY_TYPE_IS_READY_OR_RETURN_NULL(_PyAwaitable_GenWrapper_Type);
  PY_CREATE_MODULE(_capi);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(Activity, PyActivityObject);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(DiscordToken, PyDiscordTokenObject);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(Emoji, PyEmojiObject);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(ExpiringVerification, PyExpiringVerificationObject);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(Guild, PyGuildObject);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(Role, PyRoleObject);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(Tier, PyTierObject);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(TierDescription, PyTierDescriptionObject);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(AsyncExecutorTask, PyAsyncExecutorTaskObject);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(AsyncBackgroundTask, PyAsyncBackgroundTaskObject);
  /* Types for C defined awaitable python functions. */
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(Awaitable, PyAwaitable_Type);
  PY_TYPE_ADD_TO_MODULE_OR_RETURN_NULL(_GenWrapper, _PyAwaitable_GenWrapper_Type);
  PY_ADD_ALL_ATTRIBUTE(19, Py_STRINGIFY(Activity), Py_STRINGIFY(DiscordToken), Py_STRINGIFY(Emoji), Py_STRINGIFY(ExpiringVerification), Py_STRINGIFY(Guild), Py_STRINGIFY(Role), Py_STRINGIFY(Tier), Py_STRINGIFY(TierDescription), "create_activity", "create_discordtoken", "create_emoji", "create_expiringverification", "create_guild", "create_role", "create_tier", "create_tierdescription", "foo", Py_STRINGIFY(AsyncBackgroundTask), Py_STRINGIFY(AsyncExecutorTask));
}

[ZeroIntensity]

How do libraries like NumPy and PyTorch handle this? AFAIK they have a way to link their C API via setuptools. Though, perhaps another thread could be made on discourse about adding a specification for packaging C libraries, or at least adding some documentation for it.

[AraHaan]

Also I feel like the api to PyAwaitable should include these helper return defines:

#define Py_RETURN_AWAITABLE return awaitable
#define Py_RETURN_NEW_AWAITABLE PyObject *awaitable = PyAwaitable_New(); \
return awaitable

And it would make it much simpler for developers to remember what they return even if they don't touch their code in many months or even in some cases years to quickly remember what the return value should be.

[ZeroIntensity]

That's fair, but again I think would be something for after an initial release. I can give you access to the private repository to make issues, if you would like. I'm not so sure about Py_RETURN_NEW_AWAITABLE, since that wouldn't do anything useful. Py_RETURN_NEW_AWAITABLE would make the function usable via await, but it wouldn't have any callbacks or loaded coroutines, so it wouldn't do anything and just immediately raise a StopIteration.

[AraHaan]

I can give you access to the private repository to make issues, if you would like.

That would be great to have.

[ZeroIntensity]

I just made the repository public. Ignore what the docs say about installation, I haven't set it all up yet.

[ZeroIntensity]

This issue can be marked as close from view.py's side of it.