Collections are not usable

[felixSchl]

I have reported the same problem on the nutti/fake-bpy-module repository, and the fact that the behavior is the same across both projects does make me doubt that I am somehow using it wrong, however, here is what's happening:

To me the problem appears to be that we are using typing.Union instead of some form of intersection type as it's trying to prove that all types in the union support iteration.

There appears to be no equivalent typing.Intersection type in mypy, so here's the best i could come up with:

class ObjectCollection(
    typing.Sequence[bpy.types.Object],
    typing.Mapping[str, bpy.types.Object],
    bpy.types.SceneObjects,
    # HOWEVER, cannot enable this due to incompatible .items() declarations
    # bpy.types.bpy_prop_collection
):
    pass

objects: ObjectCollection = ...
o = objects.get("foo")

EDIT: I forgot to mention that I have had a look around the codebase and reason for opening ticket on this project in addition to fake-bpy-modules is that this module seems to have a more accessible implementation. I was able to find the code that generates the union here: https://github.com/mysticfall/bpystubgen/blob/1c0762bfefcac322619d5e7d29b04afcb5250eea/bpystubgen/parser.py#L188-L193 However, I am not sure how I would emit these fake classes from there.

[felixSchl]

This could work:

T = typing.TypeVar("T")
S = typing.TypeVar("S")

class FakeCollection(
    typing.Sequence[T],
    typing.Mapping[str, T],
    typing.Generic[S, T],
):
    pass

os: FakeCollection[bpy.types.SceneObjects, bpy.types.Object] = ...
for o in os:
    print(o.name)

For collections without a mixed in super class it could be:

os: FakeCollection[None, bpy.types.Object] = ...
for o in os:
    print(o.name)

That way we don't have to emit anything as we can just emit the FakeCollection once on top of the file and the use it.

[alinsavix]

[disclaimer: I'm not related to the bpystubgen project, I've just done a good bit of work making solid type stubs for blender for my own use, so have run into a lot of this kind of thing]

Yeah, this (in my experience) seems close to the right way to fix it. In my own (horribly incomplete) type stubs, I actually ended up doing something like this (which is largely equivalent):

class bpy_prop_collection(typing.Generic[K, T]):
    def __len__(self) -> int:
        ...

    def __iter__(self) -> typing.Iterator[T]:
        ...

    def __next__(self) -> T:
        ...

    def __getitem__(self, key: K) -> T:
        ...

    def __contains__(self, key: K) -> bool:
        ...

...and then making sure each actual collection type correctly inherited bpy_prop_collection (with appropriate typing). I didn't even use the union in things that referenced it after that (because e.g. bpy.types.Scene.objects can in reality only ever have a type of SceneObjects, so the union type ends up not really being needed (that I've found thusfar, anyhow).

I'd actually been intending to open an issue here basically going "y'all have any interest in trying to improve some of the rough edges, and can I help?" and have just been too busy, but maybe this comment will suffice for asking that question.

(I think one of the problems we'd be likely to run into in the grander scheme of things is that the blender api docs simply don't have all of the information required to generate a complete set of accurate type stubs (because there are things that are effectively subclasses/superclasses in the C code, but at the python level don't show an actual relationship, when in reality there is one). Might be possible to manage for collections, but the plot goes thicker than just that. Still, it's something I'd love to dig into deeper.)

[johhnry]

@alinsavix After investigating through this, your answer seems to be the most robust one ^^

Thought I'll add some modifications:

• bpy_prop_collection is not an iterator in itself so it doesn't need to implement __next__ (the iterator is rather bpy_prop_collection_iter, see type(iter(bpy.data.objects)) for example)
• Since we never call __contains__ directly and having the __iter__ method is sufficient for typing we can remove the later one

from typing import Generic, Iterator, TypeVar

K = TypeVar("K")
V = TypeVar("V")

class bpy_prop_collection_iter(Iterator[V]):
    def __next__(self) -> V:
        ...

class bpy_prop_collection(Generic[K, V]):
    def __len__(self) -> int:
        ...

    def __iter__(self) -> bpy_prop_collection_iter[V]:
        ...

    def __getitem__(self, k: K) -> V:
        ...

The weird thing is that collections have methods from a class that is not in the inheritance hierarchy:

>>> type(bpy.data.objects).__mro__
(<class 'bpy_prop_collection'>, <class 'bpy_prop'>, <class 'object'>)

>>> bpy.data.objects.tag
<bpy_func BlendDataObjects.tag()>

How is this possible?

[alinsavix]

How is this possible?

This is one of those things that happens when things are set up in the C code, rather than at the python level -- this is what I was referencing above when I said "there are things that are effectively subclasses/superclasses in the C code, but at the python level don't show an actual relationship, when in reality there is one". There's a lot that happens behind the scenes that isn't visible at the python level, nor visible in the generated documentation.

In this specific case, I believe the magic line is here: https://github.com/blender/blender/blob/a62f6c82909590a730cdb89b190370d0853ad641/source/blender/makesrna/intern/rna_main_api.c#L994

[johhnry]

Thanks for the follow up, that's what I was wondering. So it's dynamically constructed in the C code, interesting!

Have you succeeded in making BlendDataObjects iterable too and have an Union of both? Maybe the inheritance solution provided by @felixSchl is the only way to "merge" two classes with their methods.