CarliJoy commented 1 year ago

Feature or enhancement

Add an decorator that copies the parameter specs from one function onto another:

from collections.abc import Callable
from typing import ParamSpec, TypeVar, cast, Any

P = ParamSpec("P")
T = TypeVar("T")

def copy_kwargs(
    kwargs_call: Callable[P, Any]
) -> Callable[[Callable[..., T]], Callable[P, T]]:
    """Decorator does nothing but returning the casted original function"""

    @wraps(kwargs_call)
    def return_func(func: Callable[..., T]) -> Callable[P, T]:
        return cast(Callable[P, T], func)

    return return_func

Alternative names of copy_kwargs could be apply_parameters

Pitch

A quite common pattern in Python is to create a new function enhanced that enhanced a given function original or method and passes arguments using *args, **kwargs. This way the signature of original can be adopted without changing the signature of enhanced. That is especially useful if you enhance a 3rd party function.

A downside of this pattern is, that static type checkers (and IDE) are unable to detect if the correct parameters were used or give type/autocomplete hints.

Adding this pattern allows static type checkers and IDE to give correct parameter hints and check them.

Previous discussion

Discussed with @ambv at a Sprint within Europython 2023

Specification

This function is very simple, so adding it every project that requires it would be very simple. A reason to add it to the standard library is that type checkers can/should check, that the applied parameter spec matches the one of the function.

In example:

# Our test function for kwargs
def source_func(foo: str, bar: int, default: bool = True) -> str:
    if not default:
        return "Not Default!"
    return f"{foo}_{bar}"

@copy_kwargs(source_func)
def kwargs_test(**kwargs) -> float:
    print(source_func(**kwargs))
    return 1.2

kwargs_test("a", 2) # raises a TypeError but does not produce any TypingError atm

But if source_func would be defined as

def source_func(*, foo: str, bar: int, default: bool = True) -> str:
    ...

The type checker would complain.

So I would suggest that type checkers check if the wrapped functions signature is compatible to the sourced function signature. Which is separate discussion from including it into the stdlib.

The documentation should include a hint, that *args and **kwargs should be always added to the applied function.

Related Discourse Threads:

Related Issues:

How should we annotate functions that forward to their superclass? python/typing#1471 by @NeilGirdhar

Linked PRs

gh-121693

CarliJoy commented 1 year ago

The function should actually go to typing instead of functools as so it can be backported to older versions of python.

AlexWaygood commented 1 year ago

The function should actually go to typing instead of functools as so it can be backported to older versions of python.

We occasionally backport typing-adjacent things from other modules in typing_extensions (collections.abc.Buffer and types.get_original_bases have been two recent examples), so this isn't, on its own, a great argument for why this proposed feature should go into typing as opposed to any other module

CarliJoy commented 1 month ago

We occasionally backport typing-adjacent things from other modules in typing_extensions (collections.abc.Buffer and types.get_original_bases have been two recent examples), so this isn't, on its own, a great argument for why this proposed feature should go into typing as opposed to any other module

I added it to typing as in opposite to wrap nothing besides the typing is changed (like in a cast). In the linked discourse you will find ppl, wanting to use functools.wrap for this purpose. I hope that putting the function in typing makes the differance clearer.

CarliJoy commented 1 month ago

I am wondering if it makes sense to add a skip_first argument, to get it properly working when applied to function.

See mypy play, pyright play

@overload
def copy_kwargs(
    kwargs_call: Callable[P, Any], keep_first: Literal[False] = False
) -> Callable[[Callable[..., T]], Callable[P, T]]:
    ...

@overload
def copy_kwargs(
    kwargs_call: Callable[Concatenate[TFirstSource, P], Any],
    keep_first: Literal[True],
) -> Callable[
    [Callable[Concatenate[TFirstTarget, P2], T]],
    Callable[Concatenate[TFirstTarget, P], T],
]:
    ...

def copy_kwargs(
    kwargs_call: Callable, keep_first: bool = False
) -> Callable[[Callable], Callable]:
    """Decorator does nothing but returning the casted original function"""

    def return_func(func: Callable[..., T]) -> Callable[P, T]:
        return cast(Callable[P, T], func)

    return return_func

This can be used for something like this:

class Foo:
    def upstream_method(self, a: int, b: float, *, double: bool = False) -> float:
        return 1.0

class Bar:
    def __init__(self):
        self.foo = Foo()

    @copy_kwargs(Foo.upstream_method, keep_first=True)
    def enhanced(
        self, a: int, b: float, *args: Any, double: bool = False, **kwargs: Any
    ) -> str:
        return str(self.foo.upstream_method(a, b, *args, double=double, **kwargs))

Otherwise at least MyPy and PyRight complains about self as it is of course a different type for both methods.

Ravencentric commented 1 month ago

How would a case like this be covered? Where you're extending the wrapped function with your own params? For example, consider this wrapper around httpx.Client:


from collections.abc import Callable
from functools import wraps
from typing import Any, ParamSpec, TypeVar, cast

from httpx import Client

P = ParamSpec("P")
T = TypeVar("T")

def copy_kwargs(kwargs_call: Callable[P, Any]) -> Callable[[Callable[..., T]], Callable[P, T]]:
    """Decorator does nothing but returning the casted original function"""

    @wraps(kwargs_call)
    def return_func(func: Callable[..., T]) -> Callable[P, T]:
        return cast(Callable[P, T], func)

    return return_func

class APIWrapper:
    @copy_kwargs(Client)
    def __init__(self, api_url: str = "https://example.com/api/", ratelimit: int = 2, **kwargs: Any) -> None:
        self.api_url = api_url
        self.ratelimit = ratelimit
        self.kwargs = kwargs

    def post(self) -> str:
        with Client(**self.kwargs) as session:
            session.post(self.api_url)

        return "POST"

APIWrapper(api_url="https://example.org/api")  # Unexpected keyword argument "api_url" for "APIWrapper"

CarliJoy commented 1 month ago

The decorator is called copy_kwargs and not copy_and_modify. So no, it isn't possible to add a new parameter using this the decorator. Note: Using this in class like described by you would violate the Liskov substitution principle (which is not enforced for __init__ in Python type checkers, but still). Also currently is not possible to "combine" signatures of two Parameter Specs together.

But using the decorator for __init__ is possible: But you need the version with the skip_first argument and also refer to the __init__ method and not the class itself (@copy_kwargs(Client.__init__)).

See a similar example working on MyPy Play

I guess I will add the version with skip_first to the PR as this pattern is rather common especially with classes.

Back to your question. You can use Concatenate to add a positional only argument (not a keyword argument, see below). For a keyword argument, you can try combine call protocol with ParamSpecs. I did not invest enough time to try to get this to work.

def copy_kwargs_and_add_str_argument(
    kwargs_call: Callable[Concatenate[TFirstSource, P], Any],
) -> Callable[
    [Callable[Concatenate[TFirstTarget, P2], T]],
    Callable[Concatenate[TFirstTarget, str, P], T],
]:
    """Decorator does nothing but returning the casted original function"""

    def return_func(func: Callable[..., T]) -> Callable[P, T]:
        return func

    return return_func  # type: ignore

See working on MyPy Play

Please open a new Issue or Discourse Thread (there might be already one, see links in description, use search first!) to suggest changes that can combine / extent signatures. Open a Stackoverflow question to ask about combining ParamSpec and call Protocols (you can sent me a link).

Further discussion please only about non-modifying copies (besides handling the general case of self) of ParameterSpec's.

Ravencentric commented 1 month ago

Thank you for the detailed example! My apologies for going off-topic initially, this will be my last off-topic message just to add some of the discussion links I found relating to my usecase: https://discuss.python.org/t/precise-typing-of-kwargs-but-without-a-typeddict/49902 https://discuss.python.org/t/extract-kwargs-types-from-a-function-signature/34238 https://discuss.python.org/t/dynamically-building-paramspecs-from-callables/17442

python / cpython

Add a stdlib decorator that copies/applies the ParameterSpec from one function to another #107001

Feature or enhancement

Pitch

Previous discussion

Specification

Related Discourse Threads:

Related Issues:

Linked PRs