False positive list comprehension for union type source iterable

couling commented 1 week ago

Bug Report

MyPy incorrectly asses the type of list comprehensions built from union types. I can't see a way to make this pass typing checks. This and it even affects TypeVar. My only option is a # type: ignore.

The problem arises where a variable foo has a type of Iterable[Bar] | Iterable[Baz] and this is then used as part of a list comprehension [v for v in foo]. MyPy incorrectly evaluates the list comprehension as list[Bar | Baz] not list[Bar] | list[Baz].

I had thought passing a TypeVar might be a simple work around, but it appears the TypeVar is stripped off and only it's binding is maintained.

To Reproduce

I ran into this with code like this:

from typing import TypeVar

class Foo:
    item_id: str

class Bar:
    item_id: str

class FooWrapper:
    items: list[Foo]

class BarWrapper:
    items: list[Bar]

_T = TypeVar('_T', bound=FooWrapper | BarWrapper)

def filter_this(value: _T, filter_val: str) -> None:
    # Mypy Error on this line 👇
    value.items = [item for item in value.items if item.item_id == filter_val]

With the MyPy error:

error: Incompatible types in assignment (expression has type "list[Foo | Bar]", variable has type "list[Foo] | list[Bar]")  [assignment]
Found 1 error in 1 file (checked 1 source file)

Expected Behavior

A list comprehension formed from iterable Iterable[Foo] | Iterable[Bar] should have type list[Foo] | list[Bar]

Actual Behavior

A list comprehension formed from iterable Iterable[Foo] | Iterable[Bar] is assessed as having type list[Foo | Bar]

Your Environment

Mypy version used: mypy 1.11.2 (compiled: yes)
Mypy command-line flags:
Mypy configuration options from mypy.ini (and other config files):
Python version used: python 3.11

ChengPuPu commented 1 week ago

I am very interested in this task, Can you assign this task to me?

couling commented 1 week ago

Looking at this again, the problem I've faced is perhapse a little deeper than I'd first thought. What I've written above is certainly true, but I've missed one point which may go beyond MyPy's capability.

To properly evaluate the above code example (which is safe, and theoretically type safe), there needs to be some level of conditional evaluation.

Specifically for TypeVar("_T", bound=A | B), the only way to correctly determine the type safety is to evaluate the function under the condition that _T is A and seperately under the condition that _T is B. It could do this in two passes, one for _T==A and one for T==B, or it could do this by maintaining some notion of conditional typing: IE foo is type A if _T is A else foo is type B if _T is B ...

Since both of these may be a segnificant extension to MyPy's logic, I'm now a little less certain whether what I need is a feature request. As it stands, I think this bug report is still self contained, even if I'm unsure whether it will fix my problem.

python / mypy