python / cpython

The Python programming language
https://www.python.org
Other
62.59k stars 30.04k forks source link

ctypes should return composite types from callbacks #49960

Open c88ee54d-2edd-4b67-ba1b-3bb18bc351ff opened 15 years ago

c88ee54d-2edd-4b67-ba1b-3bb18bc351ff commented 15 years ago
BPO 5710
Nosy @theller, @amauryfa, @freakboy3742, @albertz

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['ctypes', 'type-feature'] title = 'ctypes should return composite types from callbacks' updated_at = user = 'https://bugs.python.org/gholling' ``` bugs.python.org fields: ```python activity = actor = 'freakboy3742' assignee = 'none' closed = False closed_date = None closer = None components = ['ctypes'] creation = creator = 'gholling' dependencies = [] files = [] hgrepos = [] issue_num = 5710 keywords = [] message_count = 8.0 messages = ['85654', '85731', '85749', '198501', '255582', '306730', '306731', '306778'] nosy_count = 7.0 nosy_names = ['theller', 'amaury.forgeotdarc', 'gholling', 'freakboy3742', 'Albert.Zeyer', 'Pam.McANulty', 'Mason.Bially'] pr_nums = [] priority = 'normal' resolution = None stage = 'test needed' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue5710' versions = ['Python 2.7', 'Python 3.4'] ```

c88ee54d-2edd-4b67-ba1b-3bb18bc351ff commented 15 years ago

We have an application that calls a 3rd party library that returns a structure (by value) from a callback. I'm including some sample code that duplicates the behavior. The problem is that return types from callbacks cannot be anything other than simple datatypes (c_int, c_float, ..., c_void_p). For other datatypes (STRUCTURE, POINTER, ...), ctypes returns the following error:

"invalid result type for callback function"

The error message comes from callback.c, in function AllocFunctionCallback.

I think this may be a duplicate of issue bpo-1574584.

I've tested this on Windows and linux; I'm including a Makefile and source code that works on Windows/cygwin.

------ file: Makefile ----

all: hello.dll hello.exe hello_dll.exe clean: rm *.exe *.dll

hello.exe: hello.h hello.c hello_main.c gcc hello.c hello_main.c -o hello.exe --save-temps

hello.dll: hello.h hello.c gcc -mno-cygwin -shared hello.c -o hello.dll --save-temps

hello_dll.exe: hello.h hello.c gcc hello_main.c -L. -lhello -o hello_main.exe

------ file: hello.h ----

struct helloStruct {
  int i;
  float f;
  int i2;
  int i3;
};

float fxn (struct  helloStruct callback());

------ file: hello.c ----

#include <stdio.h> 
#include "hello.h"

float fxn (struct helloStruct callback()) {
  struct  helloStruct result = callback();

printf ("i: %d\n", result.i); printf ("f: %f\n", result.f); return result.f * result.i; }

------ file: hello_main.c ----

#include <stdio.h> 
#include "hello.h"

struct helloStruct callback();

int main (int argc, char **argv) {
  float f;
  struct helloStruct result;
  printf ("Hello world\n");
  f = fxn (callback);
  printf ("Callback result: %f\n", f);
}

struct helloStruct callback () {
  struct helloStruct result;
  result.i = 10;
  result.f = 3.14159;
  return result;
}

int int_callback () {
  return 42;
}

------ file: hello.py ----

from ctypes import cdll, c_char, c_int, c_float, Structure, CFUNCTYPE,
POINTER, c_char_p

class helloStruct (Structure):
    pass
helloStruct._fields_ = [
    ('i', c_int),
    ('f', c_float)    ]

def callback():
    print ("callback()")
    hs = helloStruct()
    hs.i = 10
    hs.f = 25.5
    return hs

libc = cdll.msvcrt
#helloLib = libc.load_library("hello")
#helloLib = libc.hello
helloLib = cdll.hello

helloLib.fxn.restype = helloStruct
# It looks like only simple return types are supported for
# callback functions.  simple = c_int, c_float, ...
# Python bug # 1574584 - status: closed.
#    Suggests posting to ctypes-users, but I don't see any recent activity.
#
TMP_FCN = CFUNCTYPE (POINTER(c_char))           # Error message
#TMP_FCN = CFUNCTYPE (c_char_p)                  # Runs, but invalid result
#TMP_FCN = CFUNCTYPE (c_void_p)                  # Runs, but invalid result
#TMP_FCN = CFUNCTYPE (c_int)                  # Runs, but invalid result
#TMP_FCN = CFUNCTYPE (POINTER(c_int))         # Error message
#TMP_FCN = CFUNCTYPE (POINTER(helloStruct))   # Error message
#TMP_FCN = CFUNCTYPE (helloStruct)            # Error message
callback_fcn = TMP_FCN (callback)
result = helloLib.fxn (callback_fcn)

# 2.5

print "result: ", result

# 3.0 print ("result: ", result)

theller commented 15 years ago

There is a problem returning arbitrary complicated ctypes types from callbacks. Consider that a callback function returns a structure; the structure itself may contain 'char *' field for example. The callback function creates the structure and fills in the fields. The 'char *' field contains a pointer to a nul-terminated string; the ctypes structure object must keep the Python string alive as long as the structure itself. It does this via the private '_objects' instance variable. If the callback function now returns the structure, the Python ctypes object usually will go out of scope and will be garbage collected, together with its '_objects'. So, the pointer(s) contained in the structure will become invalid now, and even simple structure fields will become invalid.

However, the callback function result will be used by some other code and will surely crash.

In principle it should be possible to make code like this work; one would have to maintain the returned objects elsewhere and clean them up when they are no longer used anywhere; but I think this gets too complicated.

amauryfa commented 15 years ago

But isn't this purely a user-side concern?

For example, if I want to use a function such as QBuffer::setBuffer in the Qt library: http://doc.trolltech.com/4.4/qbuffer.html#setBuffer I must keep a reference to the buffer as long as the QBuffer is alive, or expect a crash.

Returning a pointer from a function is always tough, even in C: everybody has already tried to return the address of a local variable... the pointer must belong to some container that outlives the function call. ctypes is not different in this aspect. The same precautions as in C apply. And with a warning note in the documentation, there seems to be no reason to limit the return type of a callback.

1cbd51ed-0c85-4ad8-9beb-1b25f971f6ac commented 11 years ago

I agree with Amaury that this is purely a user side concern. While I think it's important to note the behaviour of ctypes in the case that Thomas describes, I believe it's more important to fully support the range of behaviours allowed by C function callbacks.

I see the use cases for complex return types that don't fall under the concerns raised by Thomas as the following:

Because I need this for my current project I will work on writing a patch.

bf28a09e-6d72-4c82-a4b1-8cc669e32358 commented 8 years ago

Any update here?

bebc4138-1558-4d27-8ecf-b8b86bdcdb67 commented 6 years ago

In the example code, the 'result' variable is declared on the stack in the callback() function. So it will be effectively when callback() returns.

Also the "result" variable in main() is never referenced. A pointer main()'s "result" variable should be passed to callback() as an argument ("&result") and callback(struct helloStruct *result) should populate it (via result->i = 10; etc)


  struct helloStruct result;
  result.i = 10;
  result.f = 3.14159;
  return result;
}```
bebc4138-1558-4d27-8ecf-b8b86bdcdb67 commented 6 years ago

oops - "In the example code, the 'result' variable is declared on the stack in the callback() function. So it will be effectively when callback() returns." should be "In the example code, the 'result' variable is declared on the stack in the callback() function. So it will be effectively DESTROYED when callback() returns."

I hate waking up at 4:15a.

12cd5ddb-1b49-4d91-ac6c-8fce79b8b268 commented 6 years ago

For those interested, we developed a workaround for this in Rubicon:

https://github.com/pybee/rubicon-objc/pull/85/files

The fix involves using ctypes to access ctypes own internals, and build a modified version of the Structure data type that is able to perform a copy when used as returned value.

Hopefully we'll be able to get this into the form of a patch for ctypes that is acceptable to Python core.

freakboy3742 commented 3 days ago

A follow up - the Rubicon patch has been updated to account for the changes to ctypes introduced in Python 3.13.0a6, and to account for the resolution of #81061.

https://github.com/beeware/rubicon-objc/blob/main/src/rubicon/objc/ctypes_patch.py