Generate type stubs at the same time as generating modules

junkmd commented 2 years ago

Abstract of proposal

I wish client.GetModule generates Friendly.pyi type stub at same time as generating _xxxxxxxxxxxx_xxxxxxxx_xxxxxxxx_xxxxxxxxxxxx_x_x_x.py and Friendly.py runtime modules.

Rationale

comtypes dynamically creates type-defined modules. This is a feature that other COM manipulation libraries don't have.

However, the type definitions are imported from the wrapper module _xxxxxxxxxxxx_xxxxxxxx_xxxxxxxx_xxxxxxxxxxxx_x_x_x.py to the user-friendly module Friendly.py as shown below, so all type information is hidden from the type checker.

from comtypes.gen import _xxxxxxxxxxxx_xxxxxxxx_xxxxxxxx_xxxxxxxxxxxx_x_x_x
globals().update(_xxxxxxxxxxxx_xxxxxxxx_xxxxxxxx_xxxxxxxxxxxx_x_x_x.__dict__)
__name__ = 'comtypes.gen.Friendly'

In terms of coding usability, it is no different than dynamically calling methods and attributes from a Dispatch object, and it is difficult to figure out what API the module has.

Also, the methods and properties of the derived classes of IUnknown and CoClass in the generated module are defined by the metaclass, so the type checker cannot obtain information on arguments and return values.

I would like to add the feature to generate type stubs in a format according to PEP561 at the same time as the runtime module to client.GetModule to figure out easily what API the module has.

When

After the next version of 1.1.12 is released.

and

After the refactoring of modules that defines the base class that is imported into the generated module is finished.

and

After the providing type stubs or adding type hint comments in a format according to PEP484 into some static modules below.
- comtypes/__init__.py
- comtypes/automation.py
- comtypes/client/__init__.py
- comtypes/client/_genetate.py

junkmd commented 2 years ago

Memo for type hint symbols

Question: Does it require backward compatibility with generic-alias for builtin classes, related to PEP585?

Answer: No, "There are currently no plans to remove the aliases from `typing`."

see below; https://github.com/python/typing/issues/1230 https://discuss.python.org/t/concern-about-pep-585-removals/15901 https://github.com/microsoft/pylance-release/issues/3066

junkmd commented 2 years ago

Memo for type variable

`IUnknown.QueryInterface` is useful for casting

from typing import Optional
from typing import Type  # depreciated in >= Py3.9, see PEP585.
from typing import TypeVar

_T_IUnknown = TypeVar("_T_IUnknown", bound=IUnknown)

class IUnknown(c_void_p):
    def AddRef(self) -> int: ...
    def QueryInterface(self, interface: Type[_T_IUnknown], iid: Optional[GUID] = ...) -> _T_IUnknown: ...
    def Release(self) -> int: ...

capture_of_vs_code

junkmd commented 2 years ago

Memo for `Pointer` policies

If `def somefunc()` returns instance that type is `POINTER(IUnknown)` in runtime, it will be `def somefunc() -> IUnknown`

Because it is not a lie. And good way to know what APIs the class has.

isinstance(CreateObject("Scripting.Dictionary"), POINTER(Scripting.IDictionary)) returns True,
isinstance(CreateObject("Scripting.Dictionary"), Scripting.IDictionary) returns True.
The type checker does not know what methods and attributes it has from POINTER(IUnknown). Most developers will want the IUnknown interface.
- ctypes.pointer behaves like generics with type stubs, even though it is not generics at runtime.
  - I think this is probabaly because it can be easily expressed "the value of a pointer is c_int" as pointer[c_int].
  - However, this is possible because ctypes definitions only have like c_int those have interfaces with no different from _SimpleCData.
  - It is not assumed that the object such as comtypes.IDispatch can have more than the methods and attributes that _SimpleCData has.

junkmd commented 2 years ago

ctypes.pointer behaves like generics with type stubs, even though it is not generics at runtime.

Currently, ctypes.pointer is defined as function in the type stub as well as in the implementation. ctypes.pointer(_CT) returns ctypes._Pointer[_CT]. see https://github.com/python/typeshed/pull/8446 and code snippet.

junkmd commented 2 years ago

I thought what to do about the problem of COMError and CopyComPointer being Unknown because there is no type stub for _ctypes.

I posted an issue to typeshed for adding new stub file(python/typeshed#8571).

I got agreement to add stdlib/_ctypes.pyi, so I submitted a PR(python/typeshed#8582) and it was merged.

junkmd commented 2 years ago

The Array was also Unknown because it was imported from _ctypes.

I will submit a PR to change the Array import from _ctypes to import from ctypes since both are the same stuff.

https://github.com/enthought/comtypes/blob/cc9a0131edc76bd92073f75e9737aad40cd10c58/comtypes/automation.py#L556

junkmd commented 2 years ago

PEP585 will be updated.

see python/peps#2778

junkmd commented 1 year ago

Type hinting for statically defined module, like `comtypes/init.py`.

I realized that it would be hard to update the .pyi files statically defined to match the updating .py files statically defined in same time as well.

So I considered writing type annotations in the .py files so that they would not affect them at runtime so that they would be compatible with the older versions.

It is able to write type annotations in according to PEP484's "placed in a # type: comment".

I tried static type checking by mypy and pyright(pylance) in VSCode, Py38 env.

The result is below.

a = ... # type: A | B(runtime available in Py3.10) works for both, but list[Any] and (A | B) -> None raise error by mypy.

Therefore, we must use generics like typing.List instead of builitins.list, even if it is annotations in comments. Adding only-type-checking symbols are easily by tying.TYPE_CHECKING. So we don't need afraid that unexpected symbols are added in runtime.

if sys.version_info >= (3, 5):
    from typing import TYPE_CHECKING
else:
    TYPE_CHECKING = False

if TYPE_CHECKING:
    from typing import List, Tuple

a = []  # type: List[Tuple[str, int]]

(see https://peps.python.org/pep-0484/#runtime-or-type-checking)

junkmd commented 1 year ago

I would like to use a similar process to generate runtime code and type stub code.

I will create a base class that abstracts the tools.codegenerator.CodeGenerator class. I would like to implement a concrete class for runtime code generation and a concrete class for type stub code generation inherited from that base class.

However, defining these abstract and concrete classes in the same tools.codegenerator will result in complicated class names and bloated code in a single module.

Therefore, tools/codegenerator.py should be separated into some module files.

kdschlosser commented 1 year ago

it is actually super easy to do this. There is already pieces in place that 1/2 way do the job. Since the generated files are python source files and not compiled C extensions there is no need to generate pyi files. What needs to be done is a cleaning up of the output code from the generator and actual methods/properties made in the interfaces classes for the com functions. This shouldn't take but a few hours to get done.

The other thing that is going to be needed is the complete array of various windows data types. I have a really large assortment of them already coded up here #256 I have since expanded that to include data types for HSTRING (which actually works properly) and a couple of others as well.

The importing of a generated file and copying that modules dictionary really had got to go. That kills an IDE's intellisense and code completion tools. I am willing to help out on this if it is wanted. Just point me where to go.

OH Just thought it worth mentioning that I have made and almost 100% complete PROPVARIANT structure with almost all of the data types supported.

junkmd commented 1 year ago

@kdschlosser

Thank you for your interest in this enhancement.

Thank you for coming back with your knowledge of COM and ctypes.

As a matter of fact, I don't care how it provides type hints as long as the type information is recognized by the IDE or type checker.

The plan to generate pyi files was for compatibility with Python2.

junkmd commented 1 year ago

@kdschlosser

This package still supports Python 2.7. However, a milestone has now been set for the removal of support for Python2. It will be about one year from the time discussed in #216 on https://github.com/enthought/comtypes/issues/216#issuecomment-1207158728. I also plan to do some refactoring now so that there will be less dead code when the package comes to support only Python 3.

You pointed out in #364, it is also for backward compatibility. In Python2, inline annotations would result in a SyntaxError, so I was using annotation comments as advocated in PEP484. For example, by pylance, annotation comments can still be used to display type information hover and work static type checking.

junkmd commented 1 year ago

@kdschlosser

I agree that there should be type hints in the runtime code generated by the codegenerator. However, since there is also the matter of #216, I have come up with a compromise.

Set the conditional branch on whether Python3.6 or older (sys.version_info >= (3, 6): ...)
- With Python3.5, where typing is starting to be supported by default, but there are some problems at runtime. For example, ClassVar is not supported before 3.5.2. We also believe that this condition is reasonable since we plan to support only Python3.6+ once we drop Python2.
The older version of Python will call the current codegenerator process.
The newer version of Python will call your proposed process.
For other statically defined modules, leave them as comment annotations; when we drop Python2, it may not be difficult to change them to inline annotations, as there is a tool like https://github.com/ilevkivskyi/com2ann.

junkmd commented 1 year ago

@kdschlosser

modules dictionary

The Friendly.py issue is also covered in #328. It would be great if it were easy to stop manipulating the module dictionary.

a really large assortment of them already coded up here #256

This is a very nice suggestion.

PROPVARIANT

I'm pretty sure you mean #263.

My concern is that if all of these are included at once in one PR, it may be difficult to review. It would be helpful if you could make the PR as small as possible, as close to a commit unit as possible, or in the way recommended by google's Small CLs. And need to be aware of testing.

Please share in this issue about the overall changes.

Thank you for taking the time to read this long article.

kdschlosser commented 1 year ago

No I don't mean #263.

I did a redo of the code and made it easy to access the data.

I did want to point out then when doing either inline type hinting or the commented way this is how you should go about doing it.

This is from comtypes.typeinfo


class ITypeLib(IUnknown):
    _iid_: GUID = GUID("{00020402-0000-0000-C000-000000000046}")

    # type-checking only methods use the default implementation that comtypes
    # automatically creates for COM methods.

    def GetTypeInfoCount(self) -> int:
        """Return the number of type informations"""
        return self._GetTypeInfoCount()

    def GetTypeInfo(self, index: int) -> "ITypeInfo":
        """Load type info by index"""
        return self._GetTypeInfo(index)

    def GetTypeInfoType(self, index: int) -> TYPEKIND:
        """Return the TYPEKIND of type information"""
        return self._GetTypeInfoType(index)

    def GetTypeInfoOfGuid(self, guid: GUID) -> "ITypeInfo":
        """Return type information for a guid"""
        return self._GetTypeInfoOfGuid(guid)

    def GetTypeComp(self) -> "ITypeComp":
        """Return an ITypeComp pointer."""
        return self._GetTypeComp()

    def GetDocumentation(
        self,
        index: int
    ) -> Tuple[str, str, int, Optional[str]]:
        """Return documentation for a type description."""
        return self._GetDocumentation(index)

    def ReleaseTLibAttr(self, ptla: "TLIBATTR") -> None:
        """Release TLIBATTR"""
        self._ReleaseTLibAttr(byref(ptla))

    def GetLibAttr(self) -> "TLIBATTR":
        """Return type library attributes"""
        return _deref_with_release(self._GetLibAttr(), self.ReleaseTLibAttr)

    def IsName(self, name: str, lHashVal: Optional[int] = 0) -> Optional[str]:
        """Check if there is type information for this name.

        Returns the name with capitalization found in the type
        library, or None.
        """
        from ctypes import create_unicode_buffer
        namebuf = create_unicode_buffer(name)
        found = BOOL()
        self.__com_IsName(namebuf, lHashVal, byref(found))
        if found.value:
            return namebuf[:].split("\0", 1)[0]
        return None

    def FindName(
        self,
        name: str,
        lHashVal: Optional[int] = 0
    ) -> Optional[Tuple[int, "ITypeInfo"]]:
        # Hm...
        # Could search for more than one name - should we support this?
        found = c_ushort(1)
        tinfo = POINTER(ITypeInfo)()
        memid = MEMBERID()
        self.__com_FindName(
            name,
            lHashVal,
            byref(tinfo),
            byref(memid),
            byref(found)
        )

        if found.value:
            return memid.value, tinfo  # type: ignore
        return None

If memory serves I believe this is how the COM interfaces are set up.

you have the ability to set methods using the same name as the underlying COM method. You can access the method so that the "in" and "out" flags work properly by calling the method with a preceding underscore. you can also access the COM method by preceding the method name with "com", when going this route the "in" and "out" flags are ignored and you have to pass the data containers as is needed.

Structures and Unions from ctypes are set up a bit differently

class N11tagTYPEDESC5DOLLAR_203E(Union):
    # C:/Programme/gccxml/bin/Vc71/PlatformSDK/oaidl.h 584

    @property
    def lptdesc(self) -> TYPEDESC:
        return TYPEDESC()

    @lptdesc.setter
    def lptdesc(self, value: TYPEDESC):
        pass

    @property
    def lpadesc(self) -> tagARRAYDESC:
        return tagARRAYDESC()

    @lpadesc.setter
    def lpadesc(self, value: tagARRAYDESC):
        pass

    @property
    def hreftype(self) -> int:
        return int()

    @hreftype.setter
    def hreftype(self, value: int):
        pass

    _fields_ = [
        # C:/Programme/gccxml/bin/Vc71/PlatformSDK/oaidl.h 584
        ('lptdesc', POINTER(tagTYPEDESC)),
        ('lpadesc', POINTER(tagARRAYDESC)),
        ('hreftype', HREFTYPE),
    ]

I know that looks a little goofy but what happens on the back end is when the class gets built the properties get overwritten by what is in fields

An IDE does not see the c code that works that magic so it is a seamless transition.

Using the mechanism you are using makes the code more difficult to read. A comment can be added to the methods that get overridden saying that they get overridden by what is in fields

I would really consider using the data_types module I have written. It will make the generated typelibs match what is actually in the typelib. I know that C code really has no difference between things like a LONG and an INT or a HANDLE and a HWND, it would still be nice to have the proper data type being seen in the generated code.

I also want to change up the data_types file so that instead of a variable being created for a specific data type a subclass of the data type is used instead.

I have done this for enumerations and I have cleverly come up with a way to wrap an enumeration item in a manner that allows it to be identified by its name or by its value.

kdschlosser commented 1 year ago

You also have some odd type hinting. Here is the type hinting you have set up for a pointer to IUnknown

Type[_Pointer[IUnknown]]]

This would be easier

    from ctypes import POINTER
    from typing import TypeVar, Type

    _TV_POINTER = TypeVar("_TV_POINTER", bound=POINTER)
    _T_POINTER = Type[_TV_POINTER]

and that allows your type hint to be _T_POINTER[IUnknown] without Python pitching a fit when the code is run

The use of TYPE_CHECKING really should only be reserved for the importation of modules that would cause a circular import. You would need to wrap whatever the type hint is that is in the module with double quotes in order for it to work properly.

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from some_module import SomeClass

def do() -> "SomeClass":
    ...

kdschlosser commented 1 year ago

Those suggestions make the code a lot easier to read. They are only suggestions and you can do whatever it is that you like.

You should have the maintainer of comtypes create a new branch and label that branch so it aligns with a milestone/project that has been made for dropping support for Python 3. This allows modifications to be made now without effecting the master branch and it also allows the code to get used and tested.

junkmd commented 1 year ago

@kdschlosser

No I don't mean https://github.com/enthought/comtypes/pull/263.

I did a redo of the code and made it easy to access the data.

If so, please PR that revised version. Regardless of type hinting, it should be a useful addition to comtypes.

This is from comtypes.typeinfo

This is certainly a good way to go.

The method defined by the metaclass is annotated with Callable, the argument names that are available in runtime are lost from type information.

If memory serves I believe this is how the COM interfaces are set up.

you have the ability to set methods using the same name as the underlying COM method. You can access the method so that the "in" and "out" flags work properly by calling the method with a preceding underscore. you can also access the COM method by preceding the method name with "com", when going this route the "in" and "out" flags are ignored and you have to pass the data containers as is needed.

I understood this behavior when I refactored the metaclass that generates methods(#367, #368 and #373).

Structures and Unions from ctypes are set up a bit differently

I have some opinions on this. I think would be the way to work for Python3.

class N11tagTYPEDESC5DOLLAR_203E(Union):
    lptdesc: TYPEDESC
    ...
    _fields_ = [
        ('lptdesc', POINTER(tagTYPEDESC)),
        ...
    ]

I don't want to have something recognized as a property that is not a property in runtime. I think this way would be a good fit since it is truly an annotation to a "instance variable".

I would really consider using the data_types module I have written. It will make the generated typelibs match what is actually in the typelib. I know that C code really has no difference between things like a LONG and an INT or a HANDLE and a HWND, it would still be nice to have the proper data type being seen in the generated code.

I also want to change up the data_types file so that instead of a variable being created for a specific data type a subclass of the data type is used instead.

I have done this for enumerations and I have cleverly come up with a way to wrap an enumeration item in a manner that allows it to be identified by its name or by its value.

This should be really great.

I would like you to implement this feature in a different issue scope than this, regardless of type hinting.

junkmd commented 1 year ago

@kdschlosser

    from ctypes import POINTER
    from typing import TypeVar, Type

    _TV_POINTER = TypeVar("_TV_POINTER", bound=POINTER)
    _T_POINTER = Type[_TV_POINTER]

This should be a useful generics. Thanks for the advice.

junkmd commented 1 year ago

@kdschlosser

The use of TYPE_CHECKING really should only be reserved for the importation of modules that would cause a circular import. You would need to wrap whatever the type hint is that is in the module with double quotes in order for it to work properly.

Previously, inadvertently adding/existing symbols in a module would cause a bug when a COM object with the same name existed on the type library(This caused #330).

Currently, I have set __known_symbols__ for some modules, so adding symbols for type hints is not likely to cause any problems(see #360).

So when we drop the Python 2 system and no longer need the workaround, we should use the way you suggested.

kdschlosser commented 1 year ago

I feel that making a branch to start working on dropping python2 is the way to go. There is really no sense in doing all the type hints for python 2 and 3 just to have to change them all in a year from now.

junkmd commented 1 year ago

@kdschlosser

I feel that making a branch to start working on dropping python2 is the way to go. There is really no sense in doing all the type hints for python 2 and 3 just to have to change them all in a year from now.

I agree.

So I will create a new branch once I get agreement from the other maintainers.

If we can do that, then let's proceed with the type hinting in there.

junkmd commented 1 year ago

@vasily-v-ryabov @cfarrow @jaraco

@kdschlosser knows how to do better with inline annotations for this type hinting enhancement without generating pyi files.

This is a feature that is not available with the current Python2 support.

So I would like to create a separate drop_py2 branch and plan to merge that into master when Python2 is removed from support in the near future.

If there are no objections in the next week, or if you agree, we will proceed in that direction.

Please consider this.

junkmd commented 1 year ago

@kdschlosser

I sent mention to the other maintainers.

While waiting for replies, please PR any features you would like to merge into the current master.

I will review them.

junkmd commented 1 year ago

I created #392 to discuss regarding Python2 drops.

kdschlosser commented 1 year ago

I have some opinions on this. I think would be the way to work for Python3.
class N11tagTYPEDESC5DOLLAR_203E(Union):
    lptdesc: TYPEDESC
    ...
    _fields_ = [
        ('lptdesc', POINTER(tagTYPEDESC)),
        ...
    ]
I don't want to have something recognized as a property that is not a property in runtime. I think this way would be a good fit since it is truly an annotation to a "instance variable".

And my rebuttal to this is 2 fold. First is it is a property that gets created (see below) and second is docstrings!! While I know that docstrings can be added to attributes they are only for sphinx and not a built in python feature. They might show up in an IDE if the IDE supports sphinx style attribute docstrings. Doing it the way I have suggested is 100% working with all IDEs that support pyhton because properties are a built in feature of python and not sphinx.

https://github.com/python/cpython/blob/main/Modules/_ctypes/stgdict.c

line 585.

 if (isStruct) {
            prop = PyCField_FromDesc(desc, i,
                                   &field_size, bitsize, &bitofs,
                                   &size, &offset, &align,
                                   pack, big_endian);

and then in https://github.com/python/cpython/blob/main/Modules/_ctypes/cfield.c

line 208

static int
PyCField_set(CFieldObject *self, PyObject *inst, PyObject *value)
{
    CDataObject *dst;
    char *ptr;
    if (!CDataObject_Check(inst)) {
        PyErr_SetString(PyExc_TypeError,
                        "not a ctype instance");
        return -1;
    }
    dst = (CDataObject *)inst;
    ptr = dst->b_ptr + self->offset;
    if (value == NULL) {
        PyErr_SetString(PyExc_TypeError,
                        "can't delete attribute");
        return -1;
    }
    return PyCData_set(inst, self->proto, self->setfunc, value,
                     self->index, self->size, ptr);
}

static PyObject *
PyCField_get(CFieldObject *self, PyObject *inst, PyTypeObject *type)
{
    CDataObject *src;
    if (inst == NULL) {
        return Py_NewRef(self);
    }
    if (!CDataObject_Check(inst)) {
        PyErr_SetString(PyExc_TypeError,
                        "not a ctype instance");
        return NULL;
    }
    src = (CDataObject *)inst;
    return PyCData_get(self->proto, self->getfunc, inst,
                     self->index, self->size, src->b_ptr + self->offset);
}

Look at the function names PyCField_set and PyCField_get

It is a property that is created.

Now unfortunately there is no mechanics in place for docstrings and the properties set in place that get overridden by the fields happens when the class is built so for purposes of sphinx the docstrings would be useless and at runtime help could not be used but.. for the purposes of an IDE those docstrings do get displayed.

kdschlosser commented 1 year ago

gotta go to the backend mechanics to see what is actually happening.

Personally I like how comtypes works and how a method doesn't get overridden by what is in _methods_ and instead an alternative attribute gets created. This allows for special handling of that method to be performed. With ctypes Unions and Structures this is not the case and to be honest it frankly sucks because whatever data gets passed to a specific field you might want to alter/change. An example of this would be VARIANTBOOL which uses -1 for True and 0 for False so if a field using that data type you have to know to pass a -1 to it instead of passing True. This could be handled properly if a property could be created without it being overridden during the creation of the class. Not being able to pass True kind of removes the simplicity of use aspect. With respect to VARIANTBOOL I did find a way around this on the "get" side of things by overriding the value get/set descriptor with my own versions. I have no way to handle the set aspect because the value get/set descriptor isn't used in the backend to set the field. I have not dug into how it is done to see if it is even possible to override it. I am sure that I could override the fields that are created for a structure or union but only after the structure/union class has been created and that makes for some pretty ugly code.

junkmd commented 1 year ago

@kdschlosser

First is it is a property that gets created (see below) and second is docstrings!!

Thank you for the great commentary! I respect your knowledge of cpython implementation.

>>> import ctypes
>>> class Foo(ctypes.Structure):
...     pass
... 
>>> Foo._fields_ = [("ham", ctypes.c_int)] 
>>> Foo.ham
<Field type=c_long, ofs=0, size=4>
>>> isinstance(Foo.ham, property) 
False

From the above, I was concerned that the Structure field was not a builtins.property, but I did not check until the implementation in cpython. Thanks for pointing this out.

>>> Foo.ham.__get__ 
<method-wrapper '__get__' of _ctypes.CField object at 0x0000017E53D8D3C0>
>>> Foo.ham.__set__ 
<method-wrapper '__set__' of _ctypes.CField object at 0x0000017E53D8D3C0>

Since both Field and builtins.property are data-descriptors and property is a "special-cases" unlike other custom descriptors, decorating with property seems to be in order.

As for docstring, I think it is inevitable that it will be lost in runtime. But

the purposes of an IDE those docstrings do get displayed

is helpful for developers.

kdschlosser commented 1 year ago

yessir. That is why these kinds of brain storming sessions are needed. we want to make sure the code is going to be right and that it is going to work properly.

The back end c code for ctypes is hard to follow. It takes a while to track down what exactly is going on. and if you call type on the class method that is set for the field the type of CField gets returned.

There technically speaking is a difference between a property and a get/set descriptor tho they do function almost identical. A property is just a convenience class for a get/set descriptor and it implements mechanisms to be used as decorators. It also provides some additional functions instead of calling __get__ and __set__ and __delete__. Those methods are fget, fset and fdel.

I am willing to bet if you did Foo.ham.fget you would probably end up with an attribute error. That means that the CField class is not a subclass of Property and is it's own class that implements __get__ and __set__. for the purposes of type hinting and docstrings they will serve the purpose.

junkmd commented 1 year ago

@kdschlosser

As property discussion reminds me, comtypes has the custom descriptors, named_property and bound_named_property.

I had almost written type hints for these, but deliberately avoided starting on them because they would be too complicated.

I may be wrong in some areas, but this is what I imagined.

from typing import (
    Any, Callable, Generic, NoReturn, Optional, overload, SupportsIndex,
    TypeVar
)

_R_Fget = TypeVar("_R_Fget")
_T_Instance = TypeVar("_T_Instance")

class bound_named_property(Generic[_R_Fget, _T_Instance]):
    def __init__(
        self,
        name: str,
        fget: Optional[Callable[..., _R_Fget]],
        fset: Optional[Callable[..., Any]],
        instance: _T_Instance
    ) -> None: ...
    def __getitem__(self, index: SupportsIndex) -> _R_Fget: ...
    def __call__(self, *args: Any) -> _R_Fget: ...
    def __setitem__(self, index: SupportsIndex, value: Any) -> None: ...
    def __iter__(self) -> NoReturn: ...

class named_property(Generic[_R_Fget, _T_Instance]):
    def __init__(
        self,
        name: str
        fget: Optional[Callable[..., _R_Fget]] = ...,
        fset: Optional[Callable[..., Any]] = ...,
        doc: Optional[text_type] = ...
    ) -> None: ...

    @overload
    def __get__(self, instance: _T_Instance, owner: Optional[_T_Instance] = ...) -> bound_named_property[_R_Fget, _T_Instance]: ...
    @overload
    def __get__(self, instance: None, owner: Optional[_T_Instance] = ...) -> named_property[_R_Fget, None]: ...

However, this would lose the type information of the arguments, so I considered a pattern using ParamSpec, which was introduced in Python 3.10.

# It's PEP585 and PEP604 style. Please think them will be changed to `typing` symbols.
_T = TypeVar("_T")
_R = TypeVar("_R")
_T_Inst = TypeVar("_T_Inst")
_GetP = ParamSpec("_GetP")
_SetP = ParamSpec("_SetP")

class bound_named_property(Generic[_T_Inst, _GetP, _R, _SetP]):
    name: str
    instance: _T_Inst
    fget: Callable[Concatenate[_T_Inst, _GetP], _R]
    fset: Callable[Concatenate[_T_Inst, _SetP], None]
    def __init__(self, name: str, fget: Callable[Concatenate[_T_Inst, _GetP], _R], fset: Callable[Concatenate[_T_Inst, _SetP], None], instance: _T_Inst) -> None: ...
    def __getitem__(self, index: Any) -> _R: ...
    def __call__(self, *args: _GetP.args, **kwargs: _GetP.kwargs) -> _R: ...
    def __setitem__(self, index: Any, value: Any) -> None: ...
    def __iter__(self) -> NoReturn: ...

class named_property(Generic[_T_Inst, _GetP, _R, _SetP]):
    name: str
    fget: None | Callable[Concatenate[_T_Inst, _GetP], _R]
    fset: None | Callable[Concatenate[_T_Inst, _SetP], None]
    __doc__: None | str
    def __init__(self, name: str, fget: Callable[Concatenate[_T_Inst, _GetP], _R] | None = ..., fset: Callable[Concatenate[_T_Inst, _SetP], None] | None = ..., doc: str | None = ...) -> None: ...
    @overload
    def __get__(self, instance: None, owner: type[_T_Inst]) -> named_property[None, _GetP, _R, _SetP]: ...
    @overload
    def __get__(self, instance: _T_Inst, owner: type[_T_Inst] | None) -> bound_named_property[_T_Inst, _GetP, _R, _SetP]: ...
    def __set__(self, instance: _T_Inst, value: Any) -> NoReturn: ...

Please let me know what you think about named_property and bound_named_property.

kdschlosser commented 1 year ago

type hinting those classes are pointless. actually type hinting most of what is in that file is pointless. This is because it is never really going to end up getting used at all. All of the stuff in the _memberspec file is for the dynamic creation of the methods and properties that are used when a class is dynamically built. Those type hints will never be realized when the user creates a COM interface. There is nothing that can be done for the users code and they would be responsible for the type hinting of their code.

It's because of the dynamic nature of how it works.

Technically speaking those classes should not be public classes either and they should be prefixed with an "_"..

junkmd commented 1 year ago

@kdschlosser

I see, so there is a better way to express the runtime behavior of named_property as type hinting.

And I also think _memberspec module implementation needs some refactoring futhermore.

kdschlosser commented 1 year ago

The back end mechanics of comtypes was designed for early Python 2. A lot has changed since then and I am sure that there is a lot of refactoring of the code that can be done. I don't fully grasp why there is like a line of classes used to set a simple property... But I am sure it probably had something to do with getting comtypes to work with early versions of Python 2.

kdschlosser commented 1 year ago

At some point I will dive into it in depth and see what is actually happening with the back end.

junkmd commented 1 year ago

@kdschlosser

I am curious about the methods of each class patched by _cominterface_meta._make_specials, too.

https://github.com/enthought/comtypes/blob/7e90e54d2131b35edef3e06f98f26a255f3dfa42/comtypes/__init__.py#L407-L503

These correspond to one class per attribute name that is _NewEnum, Count or Item. After all, I think it is unlikely that any one of these will be patched, and all three will usually be patched in order to implement COM collection class behavior.

In any case, this implementation is a bad fit with static type analysis.

I'm sure you have some great ideas for this too.

junkmd commented 1 year ago

I will close this issue since it has been shifted to #400 and #401, related to inline type annotations.

The scope of this issue resulted in only adding PEP484-compliant type annotations for statically defined modules and refactoring of the module and code generation process.

@kdschlosser Please help us with #400 and #401 as well.

enthought / comtypes