Introduce an Intersection

[ilevkivskyi]

This question has already been discussed in #18 long time ago, but now I stumble on this in a practical question: How to annotate something that subclasses two ABC's. Currently, a workaround is to introduce a "mix" class:

from typing import Iterable, Container

class IterableContainer(Iterable[int], Container[int]):
    ...

def f(x: IterableContainer) -> None: ...

class Test(IterableContainer):
    def __iter__(self): ...
    def __contains__(self, item: int) -> bool: ...

f(Test())

but mypy complains about this

error: Argument 1 of "__contains__" incompatible with supertype "Container"

But then I have found this code snippet in #18

def assertIn(item: T, thing: Intersection[Iterable[T], Container[T]]) -> None:
    if item not in thing:
        # Debug output
        for it in thing:
            print(it)

Which is exactly what I want, and it is also much cleaner than introducing an auxiliary "mix" class. Maybe then introducing Intersection is a good idea, @JukkaL is it easy to implement it in mypy?

[JukkaL]

Mypy complains about your code because __contains__ should accept an argument of type object. It's debatable whether this is the right thing to do, but that's how it's specified in typeshed, and it allows Container to be covariant.

I'm worried that intersection types would be tricky to implement in mypy, though conceptually it should be feasible. I'd prefer supporting structural subtyping / protocols -- they would support your use case, as IterableContainer could be defined as a protocol (the final syntax might be different):

from typing import Iterable, Container

class IterableContainer(Iterable[int], Container[int], Protocol):
    ...

def f(x: IterableContainer) -> None: ...

class Test:
    def __iter__(self): ...
    def __contains__(self, item: int) -> bool: ...

f(Test())  # should be fine (except for the __contains__ argument type bit)

[ilevkivskyi]

It would be really cool to implement protocols. Still, in this case intersection could be added as a "syntactic sugar", since there would be a certain asymmetry: Assume you want a type alias for something that implements either protocol, then you write: IterableOrContainer = Union[Iterable[int], Container[int]] But if you want a type alias for something that implements both, you would write: class IterableContainer(Iterable[int], Container[int], Protocol): ... I imagine such asymmetry could confuse a novice. Intersection could then be added (very roughly) as:

class _Intersection:
    def __getitem__(self, bases):
        full_bases = bases+(Protocol,)
        class Inter(*full_bases): ...
        return Inter

Intersection = _Intersection()

then one could write: IterableContainer = Intersection[Iterable[int], Container[int]]

[JukkaL]

Intersection[...] gets tricky once you consider type variables, callable types and all the other more special types as items. An intersection type that only supports protocols would be too special purpose to include, as it's not even clear how useful protocols would be.

[ilevkivskyi]

I understand what you mean. That could be indeed tricky in general case.

Concerning protocols, I think structural subtyping would be quite natural for Python users, but only practice could show whether it will be useful. I think it will be useful.

[gvanrossum]

This keeps coming up, in particular when people have code that they want to support both sets and sequences -- there is no good common type, and many people believe Iterable is the solution, but it isn't (it doesn't support __len__).

[ilevkivskyi]

I think Intersection is a very natural thing (at least if one thinks about types as sets, as I usually do). Also, it naturally appears when one wants to support several ABCs/interfaces/protocols.

I don't think that one needs to choose between protocols and Intersection, on the contrary they will work very well in combination. For example, if one wants to have something that supports either "old-style" reversible protocol (i.e. has __len__ and __iter__ methods) or "new-style" (3.6+) reversible protocol (i.e. has __reversed__ method), then the corresponding type is Union[Reversible, Intersection[Sized, Iterable]].

It is easy to add Intersection to PEP 484 (it is already mentioned in PEP 483) and to typing.py, the more difficult part is to implement it in mypy (although @JukkaL mentioned this is feasible).

[gvanrossum]

For cross-reference from #2702, this would be useful for type variables, e.g. T = TypeVar('T', bound=Intersection[t1, t2]).

[jeffkaufman]

Intersection[FooClass, BarMixin] is something I found myself missing today

[matthiaskramm]

If we had an intersection class in typing.py, what would we call it?

Intersection is linguistically symmetric with Union, but it's also rather long. Intersect is shorter, but it's a verb. Meet is the type-theoretic version and also nice and short, but, again, you'd expect Union to be called Join if you call Intersection Meet.

[jeffkaufman]

As a data point, I first looked for Intersection in the docs.

[ilevkivskyi]

Just as a random idea I was thinking about All (it would be more clear if Union would be called Any, but that name is already taken). In general, I don't think long name is a big issue, I have seen people writing from typing import Optional as Opt or even Optional as O depending on their taste. Also generic aliases help in such cases:

T = TypeVar('T')
CBack = Optional[Callable[[T], None]]

def process(data: bytes, on_error: CBack[bytes]) -> None:
    ...

[mitar]

I just opened #483 hoping for exactly the same thing. I literally named it the same. I would be all for Intersection or All to allow to require a list of base classes.

[ilevkivskyi]

Requests for Intersection appear here and there, maybe we should go ahead and support it in mypy? It can be first put in mypy_extensions or typing_extensions. It is a large piece of work, but should not be too hard. @JukkaL @gvanrossum what do you think?

[gvanrossum]

I think we should note the use cases but not act on it immediately -- there are other tasks that IMO are more important.

[ilevkivskyi]

@gvanrossum

I think we should note the use cases but not act on it immediately -- there are other tasks that IMO are more important.

OK, I will focus now on PEP 560 plus related changes to typing. Then later we can get back to Intersection, this can be a separate (mini-) PEP if necessary.

Btw, looking at the milestone "PEP 484 finalization", there are two important issues that need to be fixed soon: https://github.com/python/typing/issues/208 (str/bytes/unicode) and https://github.com/python/typing/issues/253 (semantics of @overload). The second will probably require some help from @JukkaL.

[JukkaL]

I agree that now's not the right time to add intersection types, but it may make sense later.

[Kentzo]

(been redirected here from the mailing list)

I think the Not type needs to be added in addition to Intersection:

Intersection[Any, Not[None]]

Would mean anything but None.

[rinarakaki]

How about the expression Type1 | Type2 and Type1 & Type2 alternative to Union and Intersection respectively.

example:

x: int & Const = 42

[ethanhs]

@rnarkk these have already been proposed many times, but have not been accepted.

[JelleZijlstra]

The Not special form hasn't been proposed before to my knowledge. I suppose you could equivalently propose a Difference[Any, None] type.

What's the use case for that? It's not something I've ever missed in a medium-sized typed codebase at work and in lots of work on typeshed.

[Kentzo]

@JelleZijlstra My specific case was to specify Any but None.

Difference and other set-alike operators can be expressed using Union, Intersection and Not.

[gvanrossum]

I don't think Not[T] fits in with the rest of the type system; it sounds more like something you'd want to do at runtime, and then specifically only for "not None".

[schferbe]

This keeps coming up, in particular when people have code that they want to support both sets and sequences -- there is no good common type, and many people believe Iterable is the solution, but it isn't (it doesn't support __len__).

@gvanrossum So what is the solution? (see also my stackoverflow question)

[JukkaL]

Have you tried defining a custom protocol which subclasses the relevant protocols? Or you can explicitly define all the methods you care about in a custom protocol.

[schferbe]

It seems that protocols are not available in typing under 3.6. The following code works and has no warnings in mypy.

from typing_extensions import Protocol

class SizedIterable(Protocol):

    def __len__(self):
        pass

    def __iter__(self):
        pass

def foo(some_thing: SizedIterable):
    print(len(some_thing))
    for part in some_thing:
        print(part)

foo(['a', 'b', 'c'])

Thanks!

[reinhrst]

Cross-posting from python/mypy#3135, as a use case for Intersection that I don't believe is possible right now: class-decorators that add methods.

class FooBar(Protocol):
   def bar(self) -> int:
     return 1

T = TypeVar("T", bound=Type)
def add_bar(cls: T) -> Intersection[FooBar, T]:
   def bar(self) -> int:
       return 2
  cls.bar = bar

@add_bar
class Foo: pass

Foo().bar()

Now I also understand that mypy doesn't support class decorators yet, but having a way to describe them correctly, and having mypy support them seem 2 different issues.

[ivan-klass]

This is also highly needed for mixin classes to annotate self-type, for cases where mixin can only be mixed to some type or even also require another mixins.

class Base:

def do(self, params: dict):
    raise NotImplementedError

class Mixin:

def _auxilary(self, params: dict) -> dict:
    return dict(params, foo='bar')

def do(self: Intersection['Mixin', Base], params: dict):
    super().do(self._auxilary(params))  # so IDE shows no warning about `.do` or `._auxilary`

[gvanrossum]

A work-around for mixins is to use a common ABC.

[0x1ee7]

Any workaround for this @gvanrossum

Cross-posting from python/mypy#3135, as a use case for Intersection that I don't believe is possible right now: class-decorators that add methods.

class FooBar(Protocol):
   def bar(self) -> int:
     return 1

T = TypeVar("T", bound=Type)
def add_bar(cls: T) -> Intersection[FooBar, T]:
   def bar(self) -> int:
       return 2
  cls.bar = bar

@add_bar
class Foo: pass

Foo().bar()

Now I also understand that mypy doesn't support class decorators yet, but having a way to describe them correctly, and having mypy support them seem 2 different issues.

[DrPyser]

Any update on this? Is it still on the road map? Mixins are a core feature of the language, I don't think this is an edge case. I know it can be worked around by defining base classes, but it means adding unnecessary boilerplate and overhead just to please type checkers.

Thanks for all the great work!

[ilevkivskyi]

Is it still on the road map?

If you are asking about the mypy team, then it is unlikely the support will be added this year. There are many other features we need to support, so we will not have time for this.

[DrPyser]

Okay. Sad. Thanks for the answer.

[wiml]

I also have been really missing an Intersection type (and was surprised it didn't already exist). I've been writing a nontrivial application using mypy and generating stubs for a sizable codebase it uses, and it's a fairly pythonic thing to say things like "This method returns a file-like object that is also usable as a FnordHandle", or whatever.

I'm not up to date on my type theory but it seems to me that any typechecker that supports protocols already supports the necessary semantics — Intersection[A, B, ...] would have the same behavior as defining a protocol that subclasses from A, B, but adds no other attributes, right?

[ilevkivskyi]

@wiml You can easily define an "intersection" protocol:

class Both(FileLike, SomethingElse, Protocol):  # assuming both bases are protocols
    pass

The hard part is to support general intersection types like Intersection[Union[A, B], Tuple[C, D], Callable[[E], F]].

[pdmccormick]

In lieu of a true intersection construct, what about a more limited Extends or Implements one, which captures the idea of a type that multiply inherits from a list of base classes?

class FileSomethingProtocol(FileLike, SomethingElse, Protocol, OtherThingsToo): pass

def helper(obj: Extends[FileLike, SomethingElse, Protocol]):
    ...

obj = FileSomethingProtocol()
helper(obj)    # obj extends all the required bases

[ilevkivskyi]

In lieu of a true intersection construct, what about a more limited Extends or Implements one, which captures the idea of a type that multiply inherits from a list of base classes?

This was proposed at early stages of PEP 544 discussions but was rejected, since it is easy to define a combining protocol or class, see https://www.python.org/dev/peps/pep-0544/#merging-and-extending-protocols

I think mypy will have general intersection types at some point, since it is a very natural feature in a language with multiple inheritance (also TypeScript has intersections). But not this year, we already have lots on our roadmap.

[flying-sheep]

Just ran into this:

import matplotlib.axes as a
def plot_into(..., ax: Intersection[a.Axes, a.SubplotBase]):
    ax.set_frame_on(False)
    ax.set_xticks([])
    ax.set_yticks([])
    ax.get_subplotspec().subgridspec(...)
    ...

For us, having Intersection would be an advantage as we use sphinx-autodoc-typehints and Intersection[...] would keep us from having to create, export, and document protocol classes (or ABCs).

(also I’m not sure if we can define a protocol as inheriting from two non-protocols)

[ysangkok]

@flying-sheep PEP-544 protocols are independent from the implementations, see https://gist.github.com/ysangkok/3668fc669c0aa8479a932405795fd4d3

[mjr129]

Shouldn't the typing library be abstracted from whether or not a particular linter (mypy) handles a particular feature? The annotations act as a hint to the developer, if nothing else.

PyCharm and IntelliJ already appear to support "Intersection" (though I'd note they don't properly support Union at present!):

class A:
    alpha = 1

class B:
    beta = 2

def foo( x : object ):
    assert isinstance( x, A ) and isinstance( x, B )
    print( x.alpha )
    print( x.beta )
    print( x.gamma ) # ERROR! Cannot find reference 'gamma' in 'A | object | B' 

[JukkaL]

PyCharm and IntelliJ already appear to support "Intersection" (though I'd note they don't properly support Union at present!)

Intersections are potentially much easier to support without proper support for all other PEP 484 features. Mypy also supports a limited form of intersection types, but it really is quite limited. Does PyCharm support arbitrary intersection types correctly, such as an intersection between a tuple, a callable and a type variable?

I think that it's important that we show that all typing features can be supported together. Each additional feature may increase the complexity of implementation exponentially, and having a standard that is effectively impractical to fully support would be unfortunate.

[mjr129]

Does PyCharm support arbitrary intersection types correctly, such as an intersection between a tuple, a callable and a type variable?

Not as far as I'm aware - there's really no way of specifying such a construct since isinstance( x, TypeVar( y ) ) isn't a valid call.

[Dreamsorcerer]

My use case is to have stricter type checking on an aiohttp Application. These objects support a dict-like interface to provide a shared config across the application. So, I think what I'm looking for is something like:

class MyAppDict(TypedDict):
    api_client: ClientSession
    db: Pool

MyApp = Intersection[Application, MyAppDict]

Perhaps a class MyApp(Application, MyAppDict) could also work, except that TypedDict can't be used as a subclass like that.

[flying-sheep]

The problem we’re trying to solve is that it’s currently impossible to say “I accept anything that subclasses/implements A and B” if one of A or B is a concrete class.

Therefore another possibility would be to treat classes as protocols.

Let’s say the syntax is Protocol[ConcreteClass] or so. Then we could e.g. do

from typing import Protocol
from sqlalchemy_mixins import SmartQueryMixin

class DeclarativeBase(Protocol):
    metadata: MetaData
    __tablename__: str
    __table__: Table
    __mapper__: Mapper

class SmartBase(DeclarativeBase, Protocol[SmartQueryMixin]):
    pass

[ghost]

Another potential use case: Often when we annotate that something is a np.ndarray, we also want to specify how many dimensions that array has. Intersection[Sequence[float], np.ndarray] specifies a one-dimensional numpy array of floats. Intersection[Sequence[Sequence[float]], np.ndarray] specifies a two-dimensional array, etc.

[eric-wieser]

@charles-staats: Numpy itself might also benefit from this (https://github.com/numpy/numpy/issues/16547#issuecomment-359276703) to allow Intersection[np.Array[float], np.Shaped[N,M]]

[johnthagen]

Now that Python 3.10 introduced using | for Union shorthand, we could start thinking about using & for intersection shorthand like TypeScript does.

def fun(a: FooClass & BarMixin):
    ...

[intgr]

@johnthagen Would be very welcome, yes. But is anyone stepping up to implement it in mypy?

[NeilGirdhar]

@intgr I just want to add as motivation for anyone who's thinking about it, I think intersections would solve many existing issues.

https://github.com/python/mypy/issues/9424 would be solved if mypy would do the correct thing of intersecting int with U rather than trying to narrow (neither type is narrower than the other). Similarly, the duplicate issue https://github.com/python/mypy/issues/5720 would be solved by intersecting int with T.

[OneHoopyFrood]

This would also solve typing issues with libraries that add Mixins. In particular, I am using transitions which creates dynamic mixin methods for each "trigger" you implement.

For obvious reasons, these are not reflected for static typing analysis. I am happy to implement a protocol detailing the mixin values, but when type hinting self on a method that uses one such mixin, I get type errors for methods belonging to my natural class. If I could type hint to an Interface between the natural class and the protocol though, it'd work peachy keen!

[rmitsch]

Is this in active development? I encountered a number of situations in which this would have been useful. Would be invaluable for proper typing imo.