Annotations for Type factories

[vors]

Context: at my company, we have a wildly used framework that at the time of writing didn't consider good static type hints for the framework users as one of the design objectives.

It makes use of the "type factory" pattern that could be illustrated with the following (much simplified) example

# framework code

# `make_int_in_range_class` is a "type factory" method
def make_int_in_range_class(lower: int, upper: int):
    # imagine here some very elaborated machinery that constructs the type dynamically
    class IntInRange(int):
        def __init__(self, v: int) -> None:
            if v < lower or v > upper:
                raise ValueError("not in range")
            self.v = v

        # many more methods like
        def custom_serialization() -> bytes:
            return b"foo"

    return IntInRange

# user code in another file

MyIntInRange = make_int_in_range_class(0, 10)  # Create the `MyIntInRange` Type

def foo(x: MyIntInRange) -> None:  # use `MyIntInRange` type in the annotation 
    print(x)

foo(MyIntInRange(4))  # example usage

When I run mypy on this code I'm rightfully getting

-----------------------------------------------------------------------------
demo.py: note: In function "foo":
demo.py:12:12: error: Variable
"robotypes_toy_generic.demo.MyIntInRange" is not valid as a type  [valid-type]
    def foo(x: MyIntInRange) -> None:
               ^
demo.py:12:12: note: See https://mypy.readthedocs.io/en/latest/common_issues.html#variables-vs-type-aliases
Found 1 error in 1 file (checked 1 source file)

Note that Pyright seems to be more permissive here and doesn't error out, but this seems to be a non-standard behavior from PEPs point of view.

The goal of having the type hint at the first place in this code is 2 fold:

1. Documentation.
2. We could not afford yet to enable globally the check_untyped_defs = True flag, too many errors. But I'd like to remove one obstacle from getting type check coverage in the new code, so it's desirable to have the type hints (however poor they could be). And I'd like to avoid having excessive use of Any or type: ignore[untyped-def].

Ideally, I'd like to have some syntax to tell any type checker that make_int_in_range_class produces a valid type (let's say even Any to make things simple, but maybe it could be some Protocol).

I was not able to find a good way of doing it short of asking ALL USERS to write some typing lie like

if TYPE_CHECKING:
  MyIntInRange = Any
else:
  MyIntInRange = make_int_in_range_class(0, 10)  # Create the `MyIntInRange` Type

This is kind of a sad solution and also we have something like 1000 call sites that would need to be updated like that. So I'm looking for advice on how this could be addressed on the framework level OR if people think it's not too fringy, maybe we could add a new feature in typing for that.

I was imagining that it could be possible to make something like this work

def make_int_in_range_class() -> Type[Any]:

[gvanrossum]

Does def make_int_in_range_class() -> Any not work?

Or what about putting this in your definition:

if TYPE_CHECKING:
    make_int_in_range_class = Any
else:
    def make_int_in_range_class(): 
        ...

[vors]

Hi @gvanrossum , thank you. No, these both don't work with mypy from what I can tell.

The errors are the same

demo.py:26:12: error: Variable "demo.MyIntInRange" is not valid as a type  [valid-type]

I tried the latest mypy version (0.991) as well as our current version.

[hauntsaninja]

Glad my if TYPE_CHECKING suggestion works a little for you. One thing that might make things slightly less painful for your users is an explicit type alias. They'll still have to type-ignore though.

MyIntInRange: TypeAlias = make_int_in_range_class(0, 10)  # type: ignore[valid-type]

def foo(x: MyIntInRange) -> None:  # no error, silently resolves to Any
    print(x)

[vors]

Yes, thank you @hauntsaninja ! I get a lot of mileage from your suggestion from gitter :)

This is indeed better, I like that! However, it still has this problem of "need to update all callsites". I can probably do a one-time migration with some scripting, but I was hoping for an even more elegant way of solving it.

Full transparency: right now I have mypy plugin where I can do a lot of tricks to allow that and I plan to (ab)use the fact that Pyright is smart. However, maintaining a custom mypy plugin is something I hope to move away from long-term. It tends to break type checking in subtle ways in my experience and I hope to have a more type-checker-agnostic code.

[Gobot1234]

In the future you should actually be able to fully type all of this code using something similar to

from typing import Protocol

class AddedInRangeMethods(Protocol):
    def custom_serialization(self: int) -> bytes: ...

def make_int_in_range_class(...) -> int & AddedInRangeMethods: ...

which makes use of IntersectionTypes (#213)

[hauntsaninja]

Gobot1234, the intersection type isn't really the issue here (and also you'd need something like type[int] & type[AddedInRangeMethods] which kind of makes me shudder; type subtyping is a mess anyway). The issue is that neither mypy nor pyright allow using variables in annotations, so at the minimum you need TypeAlias to convince them it's not a variable.

In full generality, dynamically created types are not supported by the Python static type system. For simple cases, individual type checkers may try to be more permissive here, e.g. like Pyright in this case. Note that this may entail allowing unsoundness or false positives e.g. on instantiation.

def foo() -> type[int]: ...

X = foo()  # need to use explicit TypeAlias here to have a hope of any mainstream type checker allowing X in annotations

def bar(x: X) -> None:
    reveal_type(X)

[vors]

Does def make_int_in_range_class() -> Any not work?

Or what about putting this in your definition:

if TYPE_CHECKING:
    make_int_in_range_class = Any
else:
    def make_int_in_range_class(): 
        ...

I'm pretty convinced that this would be an improvement if we can codify it like that.

Clarification about Pyright behavior: if I leave def make_int_in_range_class() unannotated, then it works. If I annotate it wil -> Any, it doesn't.