[red-knot] Add a benchmark involving realistic code that creates large unions

[AlexWaygood]

Code that creates large unions (explicitly or implicitly) is known to be hard for type checkers to analyze in a performant way. Both mypy and pyright have had lots of issues regarding performance for these, and both have implemented several optimizations to deal with them. We should add at least one benchmark (probably several) that measures how we do on this.

Common themes that come up in this area are:

• Unions involving Literal types (Literal[1, 2, 3] desugars to Literal[1] | Literal[1] | Literal[3] from the type checker's perspective, so "medium-sized Literal types" quickly end up creating huge unions)

• Enums. A similar issue to Literal types. If you have an enum like this:

  class A(enum.Enum):
    X = 1
    Y = 2
    Z = 3

  Then in some cases, x: A can desugar to x: Literal[A.X] | Literal[A.Y] | Literal[A.Z]

• Pydantic. Pydantic uses some big unions, and features in performance bug reports in both the mypy and pyright issue trackers.
• Unions involving protocols
• Unions involving recursive type aliases
• Unions involving TypedDicts

References

Here are some references that are worth looking at (and from which we might be able to derive benchmarks). The great thing about all of these is that they are performance issues that we know real users encountered when their type checker was checking real code. I've tried to exclude anything specific to recursive type aliases, since that feels somewhat out of scope for this issue:

Mypy

• https://github.com/python/mypy/issues/9169
  • fixed-ish by:
  • https://github.com/python/mypy/pull/9192
  • https://github.com/python/mypy/pull/9394
• https://github.com/python/mypy/issues/12225
• (enum-related) https://github.com/python/mypy/issues/12408
  • fixed-ish by https://github.com/python/mypy/pull/12519
• https://github.com/python/mypy/issues/12526
  • linked PRs:
  • https://github.com/python/mypy/pull/12541
  • https://github.com/python/mypy/pull/12659
• (enum-related) https://github.com/python/mypy/issues/13821
  • fixed by https://github.com/python/mypy/pull/14277
• (pydantic-related) https://github.com/python/mypy/issues/14034
  • fixed by https://github.com/python/mypy/pull/15104

Pyright

• https://github.com/microsoft/pyright/issues/7617
  • Fixed by https://github.com/microsoft/pyright/commit/aff916f0cdebd68fc6faa416a8fb428d36a06554
• !! https://github.com/microsoft/pyright/issues/7143
  • Fixed by https://github.com/microsoft/pyright/pull/7154
• (pydantic-related) https://github.com/microsoft/pyright/issues/4781
  • Fixed by https://github.com/microsoft/pyright/commit/c7ab8045bae66a19b69ce013de6f8fb642ec403c

[dangotbanned]

We recently added a TypedDict-heavy feature in altair.

Not sure if this helps as real-world examples, but sharing as red-knot came up during review

• https://github.com/vega/altair/pull/3536#discussion_r1761294081

mypy performance here has a lot of room for improvement

[hauntsaninja]

Nice! I'll add altair to mypy_primer. You may also be interested in https://github.com/python/mypy/issues/17231#issuecomment-2379375287 , I think it makes mypy significantly faster on your workload.