Open orent opened 1 year ago
I would welcome this very much. It would also be very nice if we had annotations for incomplete types if possible. A natural next step would be using generics for pointers. I don't know though whether generics have to stabilize regarding how generic parameters are accessed during runtime (see https://github.com/python/typing/issues/629). If ctypes wouldn't insist on capitalized types, we would get code like this
class Incomplete(Struct):
some_data : c_int
children : Pointer['Incomplete']
which looks very clean and natural imho.
You are welcome to open a separate issue. Your proposal is orthogonal to this one.
I would like to point out the problems with this approach.
The problem is that type checkers interpret the return type of fields solely as c_foo
.
As shown below, even if c_int
is specified for a field, the type returned at runtime is int
.
>>> import ctypes
>>>
>>> class Foo(ctypes.Structure):
... pass
...
>>> Foo._fields_ = [('x', ctypes.c_int)]
>>> Foo.x
<Field type=c_long, ofs=0, size=4>
>>> type(Foo.x)
<class '_ctypes.CField'>
>>> foo = Foo()
>>> foo
<__main__.Foo object at 0x0000023CDB80B8C0>
>>> foo.x
0
>>> foo.x = 3
>>> foo.x
3
>>> foo.x = ctypes.c_int(2)
>>> foo.x
2
In order to convey appropriate type information by annotating for fields like below, a special treatment needs to be introduced to type checkers.
class Foo(Structure):
x: c_int
This effort requires reaching out not only to the cpython
community but also to the broader community of type checker developers.
Currently, in typeshed
, fields are annotated with subclasses of _SimpleCData
, like ctypes.wintypes.POINT
. However, since this approach leads to type information mismatches between runtime and static, I am considering submitting an issue in typeshed
to address this matter.
If implementing such functionality, it might be beneficial to make CField
public and allow its definition similar to fields within Django models.
However, same as in the discussion, I also think "there is a need to maintain both syntaxes and make them mutually exclusive".
class Foo(Structure):
x = CField(c_int)
I made a dumb wrapper for ctypes to support this sort of thing at DrInfiniteExplorer/dtypes a few years back after I got tired of writing hundreds of definitions with _fields_
for a few days straight.
I didn't work extensively with proper typecheckers at the time as my main project was kind of rushed, but I did hack in a simple way to forward-declare structs to make pointers, as well as simple this-type-pointers.
I've recently gotten a bit more active with my projects and might spend more time on this.One thing that has been nagging me is that (as I've learned) tuples aren't valid types for annotations, so I'm thinking that bitfields could be declared with something like bitfield : Annotated[ctypes.c_uint8, Bitfield(2)]
and have structify
turn that into a proper _fields_
tuple.
What are your thoughts and plans for this kind of thing? I think I'll continue making small improvements to dtypes
, but if the official ctypes
gets cooler then I'm down with that, and either way we might all benefit from discussing and sharing ideas.
I had something similar because I'm too lazy to declare structs and unions using the regular syntax. Instead, I have something like that:
@cschema
class MyStruct(cStruct):
i: ctypes.c_int
class a(cStruct):
x: ctypes.c_longlong
y: ctypes.c_longlong
which makes it equivalent to
class MyInnerStruct(ctypes.Structure):
x: ctypes.c_longlong
y: ctypes.c_longlong
class MyStruct(ctypes.Structure):
_fields_ = [('i', ctypes.c_int), ('a', MyInnerStruct)]
With this approach, I can have nested classes and unions. also, I used cStruct
and cUnion
as new classes in order to add metaclass keyword arguments support (which I cannot do on th native ctypes.Structure
and ctypes.Union
). For instance:
@cschema
class MyStruct(cStruct):
class _(cStruct, anon=True):
x: ctypes.c_longlong
y: ctypes.c_longlong
becomes equivalent to
class MyInnerStruct(ctypes.Structure):
x: ctypes.c_longlong
y: ctypes.c_longlong
class MyStruct(ctypes.Structure):
_anonymous_ = ['_']
_fields_ = [('_', MyInnerStruct)]
Similarly, I can have something like:
class MyStruct(cStruct, pack=32):
pass
class MyUnion(cUnion, pack=32):
pass
instead of
class MyStruct(ctypes.Structure):
_pack_ = 32
class MyUnion(ctypes.Union):
_pack_ = 32
Note that this last construction is only useful when using nested unions. I've added other features such as:
@cschema
class MyClass(cStruct):
field: cArray[ctypes.c_int, 32]
to be equivalent to
class MyClass(ctypes.Structure):
_fields_ = [('field', ctypes.c_int * 32)]
Again, cArray
is a custom class with a special __class_getitem__
implementation. Similary, I have something like:
@cschema
class MyClass(cStruct):
field: cPointer[ctypes.c_int]
instead of
class MyClass(ctypes.Structure):
_fields_ = [('field', ctypes.POINTER(ctypes.c_int))]
For type-checkers, I implemented my own mypy plugin since this construction is quite hacky. Note that I needed this @cschema
decorator to be able to process the class body like for dataclasses and the special cStruct
& co classes to support metaclass keyword arguments.
If someone is interested, I could try to make this implementation more elegant and perhaps close to how dataclasses
are used.
Turn this:
Into this:
See discussion on https://discuss.python.org/t/annotation-based-sugar-for-ctypes/26579
Working on implementation