search

python / cpython

The Python programming language
https://www.python.org
Other
63.36k stars 30.34k forks source link

problem calling variadic functions imported from dynamic libraries under macOS on Apple Silicon #114810

Open applebud opened 9 months ago

applebud commented 9 months ago

Bug report

Bug description:

I see a problem calling variadic functions imported from dynamic libraries under macOS on Apple Silicon. I’m using a bit of code sample from https://docs.python.org/3/library/ctypes.html#specifying-the-required-argument-types-function-prototypes First with Intel macOS Python (running via Rosetta):

% python3
Python 3.11.6 (main, Nov  3 2023, 03:33:27) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.version()
'Darwin Kernel Version 23.3.0: Wed Dec 20 21:31:00 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6020'
>>> platform.processor()
'i386'
>>> from ctypes import *
>>> libc = cdll.LoadLibrary("libc.dylib")
>>> printf = libc.printf
>>> printf.argtypes = [c_char_p, c_char_p, c_int, c_double]
>>> printf(b"String '%s', Int %d, Double %f\n", b"Hi", 10, 2.2)
String 'Hi', Int 10, Double 2.200000
37
>>>

And now with native Apple Silicon Python:

% python3
Python 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.version()
'Darwin Kernel Version 23.3.0: Wed Dec 20 21:31:00 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6020'
>>> platform.processor()
'arm'
>>> from ctypes import *
>>> libc = cdll.LoadLibrary("libc.dylib")
>>> printf = libc.printf
>>> printf.argtypes = [c_char_p, c_char_p, c_int, c_double]
>>> printf(b"String '%s', Int %d, Double %f\n", b"Hi", 10, 2.2)
String '(null)', Int 0, Double 0.000000
40
>>>

As you can see the results are incorrect.

CPython versions tested on:

3.11

Operating systems tested on:

macOS

ronaldoussoren commented 9 months ago

See also: https://github.com/python/cpython/issues/107844, and this part of the documentation: https://docs.python.org/3/library/ctypes.html#calling-variadic-functions

In this particular case you shouldn't have to use argtypes for printf, or rather only have to specify the first argument. That doesn't necessarily work in general though.

The issue here is that on macOS/arm64 the calling conventions for regular and "variadic" arguments are different. During the initial port of Python to macOS/arm64 we choose to deduce which arguments are regular and which are variadic by checking the length of argtypes. That works most of the time, but not always as you have noticed.

We should add some (optional) API that explicitly tells ctypes which arguments are regular and which are variadic (for example by adding an attribute to function pointer object that denotes the index of the first variadic argument).

That is an API change and can be included in 3.13 at the earliest.

applebud commented 9 months ago

Yes, thanks, that workaround does work, with the catch that I have to effectively cast the double by hand:

>>> printf.argtypes=[c_char_p]
>>> printf(b"String '%s', Int %d, Double %f\n", b"Hi", 10, 2.2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ctypes.ArgumentError: argument 4: TypeError: Don't know how to convert parameter 4
>>> printf(b"String '%s', Int %d, Double %f\n", b"Hi", 10, c_double(2.2))
String 'Hi', Int 10, Double 2.200000
37