Fix `TypeError` when defining enumeration types

junkmd commented 3 months ago

This is a patch to prevent the TypeError reported in https://github.com/enthought/comtypes/issues/524#issuecomment-2045252482.

In comtypes, names of enumeration members are also used as names for constants at the wrapper module level. Leveraging this feature, the functionality to define enumeration types within friendly modules was implemented in #475.

However, some COM libraries have duplicated names between enumeration members and CoClass or Interface names. In other words, integers may not be assigned to these names, they may actually be of a different type. When such a situation occurs and comtypes attempt to assign a non-integer object to the value of an enumeration member, a TypeError is raised.

In this PR, for resolving the aforementioned issue, comtypes assigns the integer literal values to the members of the enumeration types instead of referencing the namespace defined within the wrapper module.

coderPE commented 2 months ago

Still an issue with comtypes=1.4.2 installed on python 3.11.09 while trying to access CSI api. comtypes=1.3.1 on python 3.11.09 works fine

Error shown: TypeError('Attempted to reuse key: %r' % key .........

junkmd commented 2 months ago

@coderPE

Thank you for the bug report.

I have a few questions for you. Please answer below as long as there are no issues with NDAs or licenses.

Please provide a detailed traceback. An example is https://github.com/enthought/comtypes/issues/524#issuecomment-2045252482, which provides us with detailed information about which module and which enumeration type the error occurred in.
I assume that an error occurred when calling GetModule or CreateObject, but please tell us the arguments you passed to those functions.
To understand the situation in more detail, please upload the files that were generated in …/path/to/your/env/pkgs/comtypes/gen/… when you encountered the error to your public repository.
If you have any API specification documentation for the "CSI api”, please share it with us.

Since there are theoretically an infinite number of COM type libraries, some may have type information configurations that we have not anticipated before.

I do not know the specifications of all COM type libraries, so I would like you to share the information and knowledge you have with the community to help us solve the problem.

Best regards

coderPE commented 2 months ago

Arguments passed: sap_object = comtypes.client.GetActiveObject("CSI.SAP2000.API.SapObject") This is the line that I'm executing, I don't think this has any arguments in it. gen.zip

The gen folder is attached.

API link: https://wiki.csiamerica.com/display/kb/OAPI The API access is bundled with the software license so I don't believe I can share it.

Detailed traceback is below: `--------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[2], line 8 6 # Check if a running instance of SAP2000 is already open 7 try: ----> 8 sap_object = comtypes.client.GetActiveObject("CSI.SAP2000.API.SapObject") 9 sap_model = sap_object.SapModel 10 print("Attached to the running instance of SAP2000.")

File c:\Users\.conda\envs\py030900\lib\site-packages\comtypes\client__init__.py:188, in GetActiveObject(progid, interface, dynamic) 186 if dynamic: 187 return comtypes.client.dynamic.Dispatch(obj) --> 188 return _manage(obj, clsid, interface=interface)

File c:\Users\.conda\envs\py030900\lib\site-packages\comtypes\client__init__.py:196, in _manage(obj, clsid, interface) 194 obj.dict["__clsid"] = str(clsid) 195 if interface is None: --> 196 obj = GetBestInterface(obj) 197 return obj

File c:\Users\.conda\envs\py030900\lib\site-packages\comtypes\client__init__.py:121, in GetBestInterface(punk) 118 tlib = tinfo.GetContainingTypeLib()[0] # typelib 120 # import the wrapper, generating it on demand --> 121 mod = GetModule(tlib) 122 # Python interface class 123 interface = getattr(mod, itf_name)

File c:\Users\.conda\envs\py030900\lib\site-packages\comtypes\client_generate.py:128, in GetModule(tlib) 126 if mod is not None: 127 return mod --> 128 return ModuleGenerator(tlib, pathname).generate()

File c:\Users\.conda\envs\py030900\lib\site-packages\comtypes\client_generate.py:245, in ModuleGenerator.generate(self) 243 for ext_tlib in codegen.externals: # generates dependency COM-lib modules 244 GetModule(ext_tlib) --> 245 return [_create_module(name, code) for (name, code) in codebases][-1]

File c:\Users\.conda\envs\py030900\lib\site-packages\comtypes\client_generate.py:245, in (.0) 243 for ext_tlib in codegen.externals: # generates dependency COM-lib modules 244 GetModule(ext_tlib) --> 245 return [_create_module(name, code) for (name, code) in codebases][-1]

File c:\Users\.conda\envs\py030900\lib\site-packages\comtypes\client_generate.py:217, in _create_module(modulename, code) 215 # clear the import cache to make sure Python sees newly created modules 216 importlib.invalidate_caches() --> 217 return _my_import(modulename)

File c:\Users\.conda\envs\py030900\lib\site-packages\comtypes\client_generate.py:28, in _my_import(fullname) 26 if comtypes.client.gen_dir and comtypes.client.gen_dir not in g.path: 27 g.path.append(comtypes.client.gen_dir) # type: ignore ---> 28 return importlib.import_module(fullname)

File c:\Users\.conda\envs\py030900\lib\importlib__init__.py:127, in import_module(name, package) 125 break 126 level += 1 --> 127 return _bootstrap._gcd_import(name[level:], package, level)

File :1030, in _gcd_import(name, package, level)

File :1007, in _find_andload(name, import)

File :986, in _find_and_loadunlocked(name, import)

File :680, in _load_unlocked(spec)

File :850, in exec_module(self, module)

File :228, in _call_with_frames_removed(f, *args, **kwds)

File c:\Users\.conda\envs\py030900\lib\site-packages\comtypes\gen\CSiAPIv1.py:708 704 eHingeLocationType_OffsetFromIEnd = 2 705 eHingeLocationType_OffsetFromJEnd = 3 --> 708 class eHingeDistributionType(IntFlag): 709 eHingeDistributionType_NonlinearBeamColumn = 1 710 eHingeDistributionType_DistributedPlasticity = 2

File c:\Users\.conda\envs\py030900\lib\site-packages\comtypes\gen\CSiAPIv1.py:712, in eHingeDistributionType() 710 eHingeDistributionType_DistributedPlasticity = 2 711 eHingeDistributionType_EqualSpacing = 3 --> 712 eHingeDistributionType_EqualSpacing = 4 713 eHingeDistributionType_EqualSpacing = 5

File c:\Users\.conda\envs\py030900\lib\enum.py:133, in _EnumDict.setitem(self, key, value) 130 key = 'order' 131 elif key in self._member_names: 132 # descriptor overwriting an enum? --> 133 raise TypeError('Attempted to reuse key: %r' % key) 134 elif key in self._ignore: 135 pass

TypeError: Attempted to reuse key: 'eHingeDistributionType_EqualSpacing'`

junkmd commented 2 months ago

Thank you for your information.

The following is my speculation.

Error investigation

TypeError('Attempted to reuse key: ... is an error that occurs when we define a duplicate member within enum.Enum (or its subclass).

>>> from enum import IntFlag
>>> class Foo(IntFlag):
...     SPAM = 1
...     HAM = 2
...     SPAM = 1
...     BACON = 3
... 
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in Foo
File "...\Python310\lib\enum.py", line 134, in __setitem__
    raise TypeError('Attempted to reuse key: %r' % key)
TypeError: Attempted to reuse key: 'SPAM'

Until Python 3.10, cases where this error occurred were exemplified in the documentation, but from 3.11 onwards, it is no longer mentioned.
- https://docs.python.org/3.10/library/enum.html#duplicating-enum-members-and-values

I have checked the CSiAPIv1.py (hereinafter referred to as the friendly module) and _F896D55D_8BDF_4232_B9AB_4B210897A81D_0_1_0.py (hereinafter referred to as the wrapper module) files you provided.

Surprisingly, it seems that the type information for eHingeDistributionType in the CSiAPIv1 type library file defines a member with a different value using the same name eHingeDistributionType_EqualSpacing.

# In `gen/_F896D55D_8BDF_4232_B9AB_4B210897A81D_0_1_0.py`, L102-L108;
# values for enumeration 'eHingeDistributionType'
eHingeDistributionType_NonlinearBeamColumn = 1
eHingeDistributionType_DistributedPlasticity = 2
eHingeDistributionType_EqualSpacing = 3
eHingeDistributionType_EqualSpacing = 4
eHingeDistributionType_EqualSpacing = 5
eHingeDistributionType = c_int  # enum

# In `gen/CSiAPIv1.py`, L708-L713;
class eHingeDistributionType(IntFlag):
    eHingeDistributionType_NonlinearBeamColumn = 1
    eHingeDistributionType_DistributedPlasticity = 2
    eHingeDistributionType_EqualSpacing = 3
    eHingeDistributionType_EqualSpacing = 4
    eHingeDistributionType_EqualSpacing = 5

`comtypes` specifications

codegenerator generates code based on the information of the COM type library. No error avoidance measures have been taken so far when processing enumeration information with defined duplicate members. This is because, from the basic principles of COM and the specifications of enumerations in most languages, it is hard to imagine that the same name member is defined in an enumeration. However, since it is actually happening in CSiAPIv1, it may be possible to define such an enumeration depending on the language that implemented the behavior of the API, and it may not cause an error when building.

Related information

I found a project on GitHub that is "Implementing the CSI api in Rust and using Tauri for the GUI". https://github.com/PaoloLupo/albars The enumeration eHingeDistributionType was defined in the codebase of that project. https://github.com/PaoloLupo/albars/blob/3b5cbfbe773318f250160a1ff8332966b436e7f4/src-tauri/crates/alba-api/src/bindings.rs#L11488-L11494 According to this, the members with the value 4 and 5 of eHingeDistributionType are assigned the names eHingeDistributionType_ContinuousSupport and eHingeDistributionType_UserDefined. The name eHingeDistributionType_EqualSpacing is used only for the member with the value 3. Information about eHingeDistributionType could not be found on GitHub other than this project and this thread, and it was not found even when googled.

I don't know whether the cause of this name duplication is a discrepancy in the definition and specification of CSiAPIv1.tlb, or a bug due to the implementation of the software.

I am considering the solution that comtypes should take, so please wait for a while.

junkmd commented 2 months ago

@coderPE

In the following branch, to prevent the TypeError('Attempted to reuse key: ...') error, I have added workarounds to codegenerator to comment out members of the enumeration type that have duplicate names. https://github.com/junkmd/comtypes/tree/fix_525_attempted_to_reuse_key

Please execute the following command to install it and try it in your environment. pip install https://github.com/junkmd/comtypes/archive/refs/heads/fix_525_attempted_to_reuse_key.zip

I have confirmed that this change has no impact on the COM type libraries used in CI and the environments I can verify. And I have checked how the enumeration type is defined in doctest.

On the other hand, I do not have the CSI-API. I also do not know of any COM type libraries that have type information for an enumeration type that defines members with the same name. Therefore, I can only rely on you to try this in your environment whether this change will work properly.

Whether this change works or not, please share the contents of …/path/to/your/env/pkgs/comtypes/gen/… after running your script.

Best regards

junkmd commented 2 months ago

@coderPE

Is there an update on this?

If we can confirm that this patch does not cause any errors in your script, I am considering including it in the next release scheduled for early June.

enthought / comtypes