Document typeshed better: a mypy upgrade suddenly triggers new errors due to typeshed changes

[davidism]

I just updated MyPy in Jinja, and had to ignore the return type of sum in one of my functions because MyPy started saying it wasn't compatible with the TypeVar I was using. https://github.com/pallets/jinja/commit/a24df26d54fa2ccbe9bdaa0bb9419075a00e2699

I see a recent change related to the return type of sum: https://github.com/python/typeshed/pull/7578. Is this a bug in typeshed/mypy, or is there a better way I should be annotating that function in Jinja?

V = t.TypeVar("V")

def sync_do_sum(
    environment: "Environment",
    iterable: "t.Iterable[V]",
    attribute: t.Optional[t.Union[str, int]] = None,
    start: V = 0,  # type: ignore
) -> V:
    if attribute is not None:
        iterable = map(make_attrgetter(environment, attribute), iterable)

    return sum(iterable, start)  # type: ignore[no-any-return, call-overload]

src/jinja2/filters.py:1295: error: Returning Any from function declared to return "V"  [no-any-return]
src/jinja2/filters.py:1295: error: No overload variant of "sum" matches argument types "Iterable[V]", "V"  [call-overload]
src/jinja2/filters.py:1295: note: Possible overload variants:
src/jinja2/filters.py:1295: note:     def [_SumT <: _SupportsSum] sum(Iterable[_SumT]) -> Union[_SumT, Literal[0]]
src/jinja2/filters.py:1295: note:     def [_SumT <: _SupportsSum, _SumS <: _SupportsSum] sum(Iterable[_SumT], _SumS) -> Union[_SumT, _SumS]
Found 2 errors in 1 file (checked 25 source files)

[srittau]

The "correct" way to do this is to add a your own protocol and constrain V to that:

class _SupportsSum(Protocol):
    def __add__(self, __x: Any) -> Any: ...

V = TypeVar("V", bound=_SupportsSum)

But we should probably add SupportsSum to _typeshed to make it a bit easier to use (and to not add it to the global namespace).

[AlexWaygood]

But we should probably add SupportsSum to _typeshed to make it a bit easier to use (and to not add it to the global namespace).

I'd prefer SupportsAdd as a name, but I agree.

[davidism]

Is there any way to make this type of error more discoverable? I don't think most people would know to look at typeshed for this error. I wouldn't have known to use something from _typeshed unless I had posted here, I regularly forget that typeshed exposes some types, because it's not mentioned in Python's typing documentation or in mypy/typing-extensions/mypy-extensions.

[davidism]

If anyone involved in typing is going to be at PyCon sprints, maybe some documentation for typeshed would be a good sprint task.

[srittau]

The typing documentation is distributed over multiple sites, at the moment. There's also a lot of institutional knowledge that's onlyt written down in some issue discussions. https://typing.readthedocs.io is supposed to become a central documentation repository in the future, but there's nothing really there yet, unfortunately.

[AlexWaygood]

If anyone involved in typing is going to be at PyCon sprints, maybe some documentation for typeshed would be a good sprint task.

There's actually a nice mypy PR I reviewed the other day that would improve the situation a little bit:

• https://github.com/python/mypy/pull/6557

The author doesn't seem to be particularly active at the moment, though, so idk if it has a chance of getting merged.

And I agree with @srittau that it's much better to have this information in a centralised place rather than the mypy docs, and https://typing.readthedocs.io/ seems like the best place for this kind of thing.

[JelleZijlstra]

It would also be useful to distribute types like SupportsAdd in a .py package, maybe typing-extensions.

[srittau]

It's been my plan for a long time now to get some of the definition from _typeshed into typing_extensions. (Maybe under typing_extensions.ext?). But time is finite. :(

[AlexWaygood]

Something like this would also be helpful for guiding people to the likely source of the error in cases like this:

• https://github.com/python/mypy/issues/10886