Open bluenote10 opened 7 months ago
The error message Can't find/import 'my_native_module.module_a.KeysView'
is factually correct.
The my_native_module.module_a
does not have KeyView
, but has KeysView[my_native_module.module_a.MyKeyA]
(sic!).
The root problem is that pybind11 generates classes with brackets in the names, for example:
ItemsView[my_native_module.module_a.MyKeyA, bool]
I choose to skip such classes during discovery to avoid generating stubs with invalid syntax, e.g.:
class ItemsView[my_native_module.module_a.MyKeyA, bool]:
...
You don't usually see the problem because:
pybind11-stubgen
strips the current module path from the annotations
my_native_module.module_a.KeysView[my_native_module.module_a.MyKeyA]
becomes
KeysView[my_native_module.module_a.MyKeyA]
KeysViews
is replaced with typing.KeyViews
If you use the type from another module first step fails and you see a strange error message.
What can be done about them? I think the specific module-level annotations like my_native_module.module_a.KeysView
do not bring any value compared to typing.KeyView
, so they could be universally replaced.
The only problem is robustly detecting such classes in references and avoiding disturbing error messages.
What can be done about them? I think the specific module-level annotations like my_native_module.module_a.KeysView do not bring any value compared to typing.KeyView, so they could be universally replaced.
Yes, this is in fact what we have been doing so far in a manual post-processing when using the mypy stub generator, and this heuristic works reasonably well so far.
Of course it means that the names KeysView
, ValuesView
, and ItemsView
are more or less reserved names that get special treatment. If a binding wants to expose a user-defined class that shares that name, things could fall apart. However, this could be easily worked around by simply avoided that particular name, i.e., even if a C++ type happens to be called KeysView
it could be exported under a different name like "KeysViewCpp"
or so. Considering that these three types already have a meaning in the Python type system that might be a sensible thing to do anyway to avoid confusion.
Consider the following example:
The pattern here is:
KeyA
andKeyB
, one defined in sub-modulemodule_a
the other inmodule_b
.module_a
defines a mapdict[KeyA, int]
in line (1)module_b
defines two maps,dict[KeyA, bool]
anddict[KeyB, bool]
.Trying to generate stubs for the module fails with:
The issue seems to be triggered by the co-existence of lines (1) and (2), i.e., if
KeysA
gets used inmodule_a
in a map, it somehow "moves theKeysView[...]
type over" into that module. By commenting out line (1), theKeysView
ends up in the other sub-module which works. Conversely, commenting out line (2) also works, because thenmodule_b
is not trying to reference the invalidKeysView
path.For comparison, when commenting out line (1) the symbols in submodule a/b are divided as (trimmed outputs of
dir(my_native_module.module_X)
):Note that all the
KeysView
are local to b. Commenting in line (1) "moves" theKeysView
intomodule_a
leading to the problematic module path: