TypeVarTuple with Callable and default arguments

[cdce8p]

Previous discussion: https://github.com/microsoft/pyright/issues/3775

I've been playing around with TypeVarTuples recently, in particular together with Callable and args. https://peps.python.org/pep-0646/#type-variable-tuples-with-callable

One issue became obvious early on which isn't defined in the PEP itself.

How should default arguments be handled?

A common pattern, especially in async code, is to pass a callable and it's args to a function

from typing import Callable
from typing_extensions import TypeVarTuple

Ts = TypeVarTuple("Ts")

def call_later(cb: Callable[[*Ts], None], *args: *Ts) -> None: ...

That works well if all arguments are required

def func1(x: int, y: int, z: int) -> None: ...
call_later(func1, 0, 0, 0)  # ok

However, what should happen if y and z have default arguments?

def func2(x: int, y: int = 0, z: int = 0) -> None: ...
call_later(func2, 0)

Instinctively, I would think that it should work, too. As @erictraut did point out though, there are at least a few cases where the behavior would need to be further specified / or explicitly forbidden.

[gvanrossum]

@pradeep90 @mrahtz

[pradeep90]

Thanks for the thoughtful use case. Yeah, we didn't address this in the PEP.

For the above case, Pyre currently uses all the parameters, default or not. So, it expects 3 arguments in call_later(func2, 1, 2, 3). [Repro](https://pyre-check.org/play?input=%23%20Pyre%20is%20being%20run%20in%20strict%20mode%3A%20https%3A%2F%2Fwww.internalfb.com%2Fintern%2Fstaticdocs%2Fpyre%2Fdocs%2Ftypes-in-python%23strict-mode%0A%23%20Use%20the%20pyre-unsafe%20header%20to%20run%20in%20unsafe%20mode.%0A%0Afrom%20typing%20import%20*%0Afrom%20pyre_extensions%20import%20TypeVarTuple%2C%20Unpack%0A%0ATs%20%3D%20TypeVarTuple(%22Ts%22)%0A%0Adef%20func1(cb1%3A%20Callable%5B%5BUnpack%5BTs%5D%5D%2C%20None%5D)%20-%3E%20tuple%5BUnpack%5BTs%5D%5D%3A%0A%20%20%20%20...%0A%0Adef%20func2(x%3A%20int%2C%20y%3A%20int%20%3D%200%2C%20z%3A%20int%20%3D%200)%20-%3E%20None%3A%20...%0A%20%20%20%20%0Adef%20apply(f%3A%20Callable%5B%5BUnpack%5BTs%5D%5D%2C%20None%5D%2C%20*args%3A%20Unpack%5BTs%5D)%20-%3E%20tuple%5BUnpack%5BTs%5D%5D%3A%20...%0A%0Adef%20main()%20-%3E%20None%3A%0A%20%20%20%20f%20%3D%20func1(func2)%0A%20%20%20%20x%20%3D%20apply(func2%2C%201)%0A%20%20%20%20y%20%3D%20apply(func2%2C%201%2C%202)%0A%20%20%20%20z%20%3D%20apply(func2%2C%201%2C%202%2C%203)%0A%20%20%20%20reveal_locals()).

So, only one of the possible ways of calling func2 will be accepted right now (i.e., passing all three arguments). The alternative is to track the various possible solutions for Ts and, as Eric pointed out, that makes this complicated feature even more complicated. We could revisit that if users run into this very often.

A workaround is to assign the function to a variable of type Callable[[int], None], so that the type checker knows which signature you intend to use.

@mrahtz If that sounds good to you too, we could mention this limitation in the PEP.

[cdce8p]

Thanks for the response @pradeep90!

We could revisit that if users run into this very often.

From my experience so far, if a codebase uses functions like call_later(cb, *args), they are relied upon all over the place. About 1/3 of the calls in my case didn't pass all arguments (and thus throw errors). At that scale adding type aliases becomes impractical quickly and adds additional complexity to the code. I would prefer if we could find a solution to make it work.

Regarding the issues with an implementation, Eric did first mention that TypeVarTuples don't have a concept of default arguments. I originally though of WithDefault (#1232) in the context of callable types, but I think it could be used here too.

The other case would be

def func1(cb1: Callable[[*Ts], None]) -> tuple[*Ts]:
    ...

def func2(x: int, y: int = 0, z: int = 0) -> None: ...

func1(func2) # Type is ambiguous; could be tuple[int] or tuple[int, int]

If Ts would resolve to [int, WithDefault[int]] and we define that WithDefault[int] means int if evaluated, tuple[*Ts] would always be tuple[int, int]. Doing so could remove the ambiguity and allow a path forward.

[mrahtz]

I think I might be missing something here for no one to have brought it up already, but isn't this what ParamSpec is for? This seems to type-check correctly in Pyright:

from typing import Any, Callable, TypeVar, ParamSpec

P = ParamSpec('P')

def call_later(
    cb: Callable[P, Any],
    *args: P.args,
    **kwargs: P.kwargs,
) -> None: ...

def func1(x: int, y: str, z: float) -> None: ...
call_later(func1, 0, '0', 0.0)  # Ok

def func2(x: int, y: str = '0', z: float = 0.0) -> None: ...
call_later(func2, 0)  # Ok
call_later(func2, 0, '0')  # Ok
call_later(func2, 0, '0', 0.0)  # Ok
call_later(func2, 0, 0)  # Not ok

Or is the point that, sure, it's doable with ParamSpec - but it'd sort of be nice if it worked with TypeVarTuple too?

[cdce8p]

I think I might be missing something here for no one to have brought it up already, but isn't this what ParamSpec is for?

This is only possible if call_later also accepts **kwargs. However, this isn't always the case. Especially with asyncio methods, it's a common pattern to only allow *args. E.g. call_soon, call_later, call_at, ... https://docs.python.org/3/library/asyncio-eventloop.html#scheduling-callbacks

A while back I suggested allowing only P.args in a function signature #1000, but it was basically rejected with the reference to TypeVarTuples as alternative: https://github.com/python/typing/issues/1000#issuecomment-1004459692.

[mrahtz]

This is only possible if call_later also accepts **kwargs.

Oh huh, I didn't realise.

Mulling this over, even with potential future extensions, I don't think TypeVarTuple would be the right construct to handle this situation in general. One of Eric's examples is particular instructive:

def func1(cb1: Callable[[*Ts], None], cb2: Callable[[*Ts], None]) -> tuple[*Ts]:
    ...

The question of whether cb1 and cb2 have the same call signature when default arguments are involved is complicated enough that I think we'd want some mechanism for specifying the desired behaviour more precisely. It seems like one could reasonably want any of the following behaviours:

1. The signature of cb1 and cb2 is the same ignoring default arguments
2. The signature of cb1 and cb2 is exactly the same, including the same default arguments
3. cb1 and cb2 share the same base signature (without default arguments), and cb2's default arguments are a superset of cb1's, or vice-versa

(And as Eric points out, there's still the issue of what the *Ts in the return type should mean.)

Even if we did implement some way of using TypeVarTuples for this, there are so many options for how to interpret the resulting type constraints that I think it would be unintuitive to the reader.

But for the call_later example in particular

def call_later(cb: Callable[[*Ts], None], *args: *Ts) -> None: ...

I'm wondering whether it might be possible to specifying the exact behaviour desired manually with something like:

ArgType = *Ts | *tuple[*Ts, T1] | *tuple[*Ts, T1, T2]  # | Etc
def call_later(cb: Callable[[ArgType], None], *args: ArgType) -> None: ... 

So if we want to use that with

def func2(x: int, y: str = '0', z: float = 0.0) -> None: ...

We could do:

• *args: *tuple[int]
  • ⇒ type checker infers solution of cb: Callable[[*tuple[*Ts, T1, T2]], None], *args: *Ts
• *args: *tuple[int, str]
  • ⇒ type checker infers solution of cb: Callable[[*tuple[*Ts, T1, T2]], None, *args: *tuple[*Ts, T1]
• *args: *tuple[int, str, float]
  • ⇒ type checker infers solution of cb: Callable[[*Ts], None], *args: *Ts

@pradeep90 Am I crazy, or could this work?

[cdce8p]

I'm wondering whether it might be possible to specifying the exact behaviour desired manually with something like:

ArgType = *Ts | *tuple[*Ts, T1] | *tuple[*Ts, T1, T2]  # | Etc
def call_later(cb: Callable[[ArgType], None], *args: ArgType) -> None: ... 

Am I crazy, or could this work?

Not fully certain, but wouldn't we need a constraint and nested TypeVarTuple for that?

T1 = TypeVar("T1")
T2 = TypeVar("T2")
Ts = TypeVarTuple("Ts")
Tu = TypeVarTuple("Tu", *Ts, *tuple[*Ts, T1], *tuple[*Ts, T1, T2])

def call_later(cb: Callable[[*Tu], None], *args: *Tu) -> None: ...

Without Tu, there isn't any connection between the ArgType for Callable and the one for *args.

[hmc-cs-mdrissi]

I think this complexity strongly leans me towards support P.args only / P.kwargs only solution. I'd rather make this legal,

def foo(func: Callable[P, object], *args: P.args):
  ...

then work out remaining implications of typevartuple with defaults. Rules here I'd expect is func must not have any required keyword only arguments and that when foo is called it can not have any keyword arguments passed (besides func to allow foo(func=f).