search

python / cpython

The Python programming language
https://www.python.org/
Other
61.16k stars 29.52k forks source link

[C API] Removed private _PyArg_Parser API has no replacement in Python 3.13 #112136

Open vstinner opened 8 months ago

vstinner commented 8 months ago

Copy of @tacaswell's message:

aio-libs/multidict is also using _PyArg_Parser and friends (e.g. https://github.com/aio-libs/multidict/blob/18d981284b9e97b11a4c0cc1e2ad57a21c82f323/multidict/_multidict.c#L452-L456 but there are many)

The _PyArg_Parser API is used by Argument Clinic to generate efficient code parsing function arguments.

Since Python 3.12, when targeting Python internals, Argument Clinic initializes _PyArg_Parser.kwtuple with singletons objects using the _Py_SINGLETON(name) API. This API cannot be used outside Python.

Private functions with a _PyArg_Parser argument:

cc @serhiy-storchaka

Linked PRs

vstinner commented 8 months ago

A code search on _PyArg_Parser in PyPI top 8,000 projects (2023-11-15) found 5 projects:

nickelpro commented 8 months ago

These APIs are effectively required for using METH_FASTCALL | METH_KEYWORDS which is a documented part of the API.

Notably METH_FASTCALL is part of the Stable ABI and requires the not-mentioned-here-but-related _PyArg_ParseStack().

"Private, but documented, and partially considered part of the stable ABI" is not the same thing as "Private". Removing these functions leaves no API to interact with these documented call mechanisms. Unless you intend to remove METH_FASTCALL and its derivatives from the documentation and the stable ABI, something must replace these private functions.

Anecdotally, we make extensive use of _PyArg_ParseStack() and _PyArg_ParseStackAndKeywords() internally at NYU in high-traffic CPython methods. I don't think a code search of PyPI is a responsible way to gauge the existence of C API consumers.

vstinner commented 8 months ago

What I'm trying to do here is to list projects using these projects, see how they use the API, and which public API is needed.

Anecdotally, we make extensive use of _PyArg_ParseStack() and _PyArg_ParseStackAndKeywords() internally at NYU in high-traffic CPython methods.

Do you have some examples? Which calling convention do you use? METH_FASTCALL, METH_FASTCALL | METH_KEYWORDS, and/or something else?

"Private, but documented, and partially considered part of the stable ABI" is not the same thing as "Private". Removing these functions leaves no API to interact with these documented call mechanisms.

This situation is unfortunate. METH_FASTCALL was created as a private API but apparently, it was adopted outside Python before it was standardized by PEP 590 – Vectorcall: a fast calling protocol for CPython.

As the author of METH_FASTCALL, I would like to add a clean public, documented and tested API to make these calling convention fully usable :-)

In terms of functions, so far the following functions were mentioned:

Is the number of arguments always a signed Py_ssize_t type? Or do we have to care about PyVectorcall_NARGS() which uses an unsigned size_t type? If I understood correctly, unsigned size_t is only used to call a function. But inside a function implementation, the calling convention is to pass a number of arguments as a signed Py_ssize_t.

nickelpro commented 8 months ago

Do you have some examples? Which calling convention do you use? METH_FASTCALL, METH_FASTCALL | METH_KEYWORDS, and/or something else?

I don't think we do anything fancy, basically the same straightforward code you would see pop out of argument clinic:

// Tiny METH_FASTCALL example function
static PyObject *print_func(PyObject *self,
    PyObject *const *args, Py_ssize_t nargs) {
  const char *str;
  if(!_PyArg_ParseStack(args, nargs, "s", &str))
    return NULL;
  puts(str);
  Py_RETURN_NONE;
}

or

// Tiny METH_FASTCALL | METH_KEYWORDS example function
static PyObject* prefix_print(PyObject* self, PyObject* const* args,
    Py_ssize_t nargs, PyObject* kwnames) {
  static const char* _keywords[] = {"", "prefix", NULL};
  static _PyArg_Parser _parser = {.keywords = _keywords,
      .format = "s|s:prefix_print"};

  const char* str;
  const char* prefix = NULL;
  if(!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, &str,
         &prefix))
    return NULL;

  if(prefix)
    printf("%s", prefix);
  puts(str);

  Py_RETURN_NONE;
}

for METH_FASTCALL and METH_FASTCALL | METH_KEYWORDS respectively. I imagine if we converted the latter usage to vectorcall it would be identical with the exception of replacing nargs with PyVectorcall_NARGS(nargsf), so vectorcall vs METH_FASTCALL isn't really the problem here unless I'm completely missing some public API for parsing vectorcall arguments.

A quick Ctrl-F shows no one using kwtuple yet, but I think kwtuple is a basically good feature (and we use pre-initialized global string objects in some other places), so I would be sad to see that capacity go away as well.

I would like to add a clean public, documented and tested API to make these calling convention fully usable

Ya 100%. I understand I'm just some random and you're the core dev here, just wanted to raise the objection that breaking this functionality without having that replacement ready doesn't seem like a win to me. The APIs are already internal, there's no stability contract, but if they exist anyway in an unbroken state, what's the harm in leaving them accessible until a replacement is available?

As a practical matter if these disappeared we would be replicating the declarations in our own headers for the Python versions where no solutions existed, and that seems like a pointless maintenance burden to push out into the world.

Is the number of arguments always a signed Py_ssize_t type? Or do we have to care about PyVectorcall_NARGS() which uses an unsigned size_t type?

PyVectorcall_NARGS() accepts a size_t but returns an ssize_t, from the point of view of the parser nargs is always signed.

webknjaz commented 6 months ago

@vstinner in case of multidict, here's the PR that started using this private API https://github.com/aio-libs/multidict/pull/681. For now, I reverted it under Python 3.13: https://github.com/aio-libs/multidict/pull/909. The original PR implies there's a 2.2x difference in performance, so it'd be useful to get an equivalent replacement.

AdamWill commented 2 weeks ago

blender also uses _PyArg_Parser extensively: https://projects.blender.org/blender/blender/search?q=_PyArg_Parser . I am not enough of a C coder to know how easy it would be to port all those uses to the public argument parsing API.

vstinner commented 2 weeks ago

@AdamWill:

blender also uses _PyArg_Parser extensively: https://projects.blender.org/blender/blender/search?q=_PyArg_Parser . I am not enough of a C coder to know how easy it would be to port all those uses to the public argument parsing API.

I wrote PR gh-121262 to restore the private _PyArg_Parser structure and the private _PyArg_ParseTupleAndKeywordsFast() function. Is it enough for Blender?

AdamWill commented 1 week ago

@vstinner on a naive grep it looks like it should be enough, but blender at least also uses _PySet_NextEntry, so I can't get a complete compile done to verify unless that's put back in or it's ported over to the generic iterator stuff (which I'm not a good enough C coder to do myself).

vstinner commented 1 week ago

@vstinner on a naive grep it looks like it should be enough

Good. I was only asking for "PyArg" APIs.

vstinner commented 1 week ago

I restored the removed private _PyArg_Parser structure and the private _PyArg_ParseTupleAndKeywordsFast() function in the change https://github.com/python/cpython/commit/f8373db153920b890c2e2dd8def249e8df63bcc6. I close the issue.

nickelpro commented 1 week ago

I think this has been prematurely closed as no solution has been provided for the other _PyArg_Parser API functions, which are not 1-to-1 clones of _PyArg_ParseTupleAndKeywordsFast()

vstinner commented 1 week ago

Ok, I reopen the issue.