Keyword args not supported

[DonkeyKong1]

First, I really like this. I was getting pretty confused when I kept getting "plum.function.NotFoundLookupError" and couldn't find issue with the simple class I'd made. Randomly, I forgot to provide a keyword for the arg and it worked. Am I correct that keyword args are not supported?

[wesselb]

Hey @DonkeyKong1! You're right that dispatch interacts a bit strangely with keyword arguments. In particular, arguments which are not given a default value must also be given in non-keyword form. E.g., consider

@dispatch
def f(x: int):
     return x

Then f(2) will work, whereas f(x=2) breaks.

You can use default values for arguments, and, in that case, you can give these arguments in keyword form. See https://github.com/wesselb/plum#keyword-arguments-and-default-values.

I hope this clarifies! :) Please let me know if you have any other questions.

[astanziola]

Hi! Sorry to post on an old issue, but is there any fundamental reason for not supporting keyword arguments without default values? I'm using plum to dispatch over operators with many type combinations of mandatory inputs, and it would make the code much clearer to be able to call them using keyword arguments.

[wesselb]

Hey @astanziola!

The design of Plum is heavily inspired by how multiple dispatch works in the Julia programming language. In Julia, keyword arguments are also not considered in the dispatch process. I believe you can find some discussion online around this point.

More fundamentally, multiple dispatch is based on the idea that arguments are identified by position and type. Keyword arguments, on the other hand, are identified by name. I agree that one could consider a dispatch process which also considers keyword arguments, but, up to this point, we've tried to follow the Julia programming language.

[astanziola]

Thank you for clarifying @wesselb , I had no idea that Julia was following this rule for the multiple dispatch system!

[teffland]

Hi, are there any plans in the future to support this? I really like dispatch, but this is a bit of an issue for downstream usability, since in python its very common to call functions with named args for both readability and to stay robust to irrelevant upstream signature updates.

Is there something inherently preventing this behavior? Would be interested in making a contribution to allow this behavior or at least display a more useful error message to the user.

[wesselb]

@teffland Currently, there are no plans to dispatch on keyword arguments, but obviously future plans can change. :) I agree that calling a function with named arguments is very common in Python. It would be very nice to support this.

Is there something inherently preventing this behavior?

That's a good question, and I'm inclined to answer "not necessarily", but how this would work would have to be spelled out in detail. For example, consider the following scenario:

@dispatch
def f(x: int, y: float):
    ... # Method 1

@dispatch
def f(y: float, x: int):
   ... # Method 2

Then calling f(1, 1.0) would call method 1 and calling f(1.0, 1) would call method 2.

What's unclear to me is what f(x=1.0, y=1) would do. For method 1, I suppose that the call would be equivalent to f(1.0, 1), which wouldn't match. For method 2, the call would be equivalent to f(1, 1.0), which also wouldn't match. The strange thing is that the arguments are switched around between the two methods because the names don't line up.

Perhaps this a poor example, but what's going wrong is the following. In Python, there are two ways to say that a particular argument should have a value:

1. by position, or
2. by name.

Once you name things, the position becomes irrelevant. And once you position an argument, the name of the argument as written in the function doesn't matter. Hence, in some sense, it's an either-or situation where you have to choose to designate arguments by position or designate arguments by name.

Currently, the whole multiple dispatch system has been designed around arguments-designated-by-position (and type), which is, as argued above, somehow incongruent with designating arguments by name. Why? Once you name things, the position becomes irrelevant; and once you position something, the name of the argument becomes irrelevant.

Therefore, if we were to support naming arguments, how precisely this would work would have to be spelled out in detail.

[teffland]

@wesselb thanks for the detailed response. I've given it some consideration.

I suppose one way to deal with this could be to take each function definition and convert it into multiple possible signatures. These signatures could be composed of one ordered and one named component, for the positional and named parts respectively. As in, Signature = Tuple[Tuple[Type],Dict[str,Type]].

Since python requires positional args come before named ones, a function with n parameters could be converted into n+1 total signatures, each with a different partition of arguments in the ordered vs named subsets. The 0th would have 0 elements specified by position and all in the named set, the 1st would be the first element specified by position and the rest in the named set, etc. As follows:

def f(x:int, y: int, z: int):
   # 4 possible signatures
   # 0. ((,),{x:int, y:int, z:int})
   # 1. ((int,), {y:int, z:int})
   # 2. ((int,int), {z:int})
   # 3. ((int,int,int),{})

Then when, matching signatures it would partition the call signature the same way.

One other smaller detail to consider would be matching named args with defaults vs w/o defaults. I guess for the above signatures, you could only put params in the "named" signature if they don't have defaults, and just would check that the input named arguments are a superset of the "named" portion of the signature. That would take care of it I believe. So, continuing the above example, if f were instead written as:

def f(x:int, y: int, z: int, a: int = 0):
   # 4 possible signatures
   # 0. ((,),{x:int, y:int, z:int})
   # 1. ((int,), {y:int, z:int})
   # 2. ((int,int), {z:int})
   # 3. ((int,int,int),{})

then it would still have the same signature. (Note that this would mean this definition is incompatible with the first definition in the same registry. I would argue this is a good restriction, as allowing two implementations of functions that only differ by optional parameters is inherently ambiguous.)

Let me know what you think. Would be happy to take a crack at a PR sometime.

[wesselb]

Hey @teffland! Apologies for the very slow reply.

What you're proposal is in fact already happening, but only for arguments with defaults. That is,

def f(x: int, y: int = 1, z: int = 2):
   # Generates three signatures:
   # 1. ((int,), {y: int = 1, z: int = 2})
   # 2. ((int, int), {z: int = 2})
   # 3. ((int, int, int), {})

Extending it to also include all position arguments is an interesting proposal! I might see potential issues. Consider the definitions

@dispatch
def f():
    print("Doing something with no ints")

@dispatch
def f(x: int):
    print("Doing something with one int")

@dispatch
def f(x: int, y: int):
    print("Doing something with two ints")

Then the third one would generate the methods ((), {x: int, y: int}) and ((int,), {y: int}), overwriting the definitions of the first two! What are your thoughts on potential conflicts like this?

[gabrieldemarmiesse]

I played a bit in plume and I would like to provide a small improvement: When the signature of all overloads are matching (same number of arguments and in the same order) then using keyword argument shouldn't be ambiguous.

I feel this use case is quite common and even if it's not all use cases of plum, it would remove some astonishment that can happen as a first time user of plum. It's quite confusing for beginners right now.

If the signature of all overloads aren't matching (different arguments or a different order) then we can fall back to the existing behaviour and fail.

[wesselb]

@gabrieldemarmiesse could you give an example? I'm not sure that I fully understand this use case.

[gabrieldemarmiesse]

When I'm talking about a function where all overloads have matching signatures are matching, I'm thinking about this:

from plum import dispatch

@dispatch
def foo(x: int, y: int):
    print("you have called in int version of foo")

@dispatch
def foo(x: str, y: str):
    print("you have called in str version of foo")

foo(1, 2)
foo("1", "2")
# this should be  possible but currently fail
# with plum.resolver.NotFoundLookupError
foo(x=1, y=2)

Interestingly, this type of restriction (same number of arguments and always in the same order) is also the restriction that is enforced by @overload, so I believe it's very common.

This common pattern should be supported, while this pattern:

@dispatch
def f(x: int, y: float):
    ... # Method 1

@dispatch
def f(y: float, x: int):
   ... # Method 2
f(x=1, y=2)

should fail with a discriptive error message saying that if you want to use keyword arguments, then you should have matching signatures (and optionally use @overload)

[wesselb]

@gabrieldemarmiesse, I see! If I understand correctly, the idea is to tie argument names to positions, e.g. x should always be the first argument and y the second, and to resolve things that way. I think that should be possible! Perhaps this shouldn't be the default behaviour, as requiring arguments to be named in a specific ways might be undesirable in certain scenarios, but we could allow the user to opt into the behaviour:

from plum import Dispatcher

dispatch = Dispatcher(named_positional=True)

...

Very cool idea! :)

[gabrieldemarmiesse]

Perhaps this "named_positional" should be inferred automatically from the list of signatures? If it's possible then the UX would be better as users wouldn't have to think about it. I'll try to find out if that is possible.

[wesselb]

@gabrieldemarmiesse Right! That sounds even better. I would be fully on board with that. :)

[repst]

I explored integrating plum dispatch into a large existing code base. Since the user would have to know that a function is dispatched in order to call it correctly (e.g. can't assign a positional arg as a keyword arg or vice versa), I was not comfortable making dispatched functions user-facing. Instead, I used the following workaround that hides the dispatched functions from the user and allows dispatching on both positional and keyword arguments:

from plum import dispatch

def user_facing(x, y, z="default"):
    # assign all args and kwargs as positional args in _private
    return _private(x, y, z)

@dispatch
def _private(x: int, y: int, z: tuple):
    pass

@dispatch
def _private(x: str, y: str, z: str):
    pass

Not sure if this is well-known or not.

[wesselb]

Hey @repst! Thanks for sharing this. I think that’s a neat little trick which finds an interesting middle ground between user friendliness and flexibility. :)

[Splines]

Unfortunately with the suggestion by repst, the user won't get nice IntelliSense and see what dispatched versions of a method there are available and what types. Would really appreciate being able to use keyword arguments with the otherwise amazing plum ;)

[leycec]

@leycec hath been indirectly summoned. Mu-hah! Everyone had better things to do tonight, yet @leycec showed up anyway. So, I'm prompted to start blathering due to @Moosems of growing @salve-org infamy lamenting over at salve-org/salve#74 that:

if multiple dispatch was nicer than it is in plum, I'd actually use it here. The main issue is that it doesn't allow keyword arguments alongside normal positional arguments.

B-b-but Plum is nice, @Moosems! That aside, keyword argument support would be the chef's kiss. As I see it, the core issue here is that callable arguments are flexible by default – that is, an argument may be flexibly passed either by position or by name. While essential, this flexibility also introduces ambiguities in dynamic typing systems like @beartype and Plum.

As an incremental path to supporting ambiguously flexible arguments, the first milestone for Plum should probably be to...

Support Keyword-only Arguments for a Tastier Plum Experience

The neat-o thing about keyword-only arguments is the disambiguity. They eliminate the ambiguities introduced by flexible arguments. Since a keyword-only argument cannot (by definition) be passed positionally, Plum can definitely be generalized to dispatch on keyword-only arguments without breaking backward compatibility. Whether this is trivial or not is anyone's best guess. It's probably not... but it's probably a lot more trivial than refactoring Plum to support flexible arguments passed by name all in one go.

Thankfully, keyword-only arguments are great! Arguments passed by name tend to be a lot more readable than arguments passed by position. Honestly, the only reason to pass arguments by position in the modern era is:

• Callables accepting only a single parameter. It's obvious what the parameter is, so it doesn't need a name. Conversely, calls to callables accepting two or more parameters should usually be passed by name. The more parameters, the more that "usually" becomes "absolutely or else a giant bear named Ursula eats your code readability."
• Efficiency. It turns out that passing parameters by name is a lot slower than passing parameters by position. Indeed, passing parameters by name is so much slower that there's kinda no point in even memoizing (i.e., caching) callable calls that accept parameters by name. This is why the @beartype codebase internally forces all calls to a memoized callable to pass parameters positionally.

Once Plum supports keyword-only arguments, it should then be a lot easier to chart a sane path towards the final milestone: refactoring Plum to additionally support flexible arguments passed by name.

If I had this nebulous thing called "time," I'd happily throw down the gauntlet and start chucking around pull requests onto the Plum codebase. Sadly, here we are. My gauntlet is fake.

[Moosems]

For the record: I do think plum is awesome. However, not being able to do:

x = 5

my_str = "Hello World!"
print_multiple(x, special_str=my_str)

other_str = b"Bytes string!"
print_multiple(x, special_str=other_str)

Is a pretty big deal breaker when you want to hand this off to a user who doesn't want to learn the rules of some other project used inside the library they want to use once and forget.

[PhilipVinc]

(Typing this from my phone while traveling, so excuse all typos that will emerge)

I was recently thinking about that and I thought that there might be one way to make plum work with keyword arguments through an opt in mechanism:

The main issue is that the dispatch logic of plum works by identifying positional arguments, and keyword-addressing positional arguments might break because the first argument in one method might be named as the second in another one.

So, a possible solution is to oblige all methods to have the same positional argument names.

This would be opt-in and the limitation would be in exchange of support of dispatch on keyword args.

A simple, limited way, would be to have something like @dispatch.abstract(keyword_dispatch=True) record all positional argument names in the abstract definition, and subsequent registrations would fail if the use different names for the positional arguments

For example:

@dispatch.abstract(keyword_dispatch=True)
def myfun(a, b, c, **kwargs):
  Pass

@dispatch
def myfun(a: int, b:float, *, some_kw=1)
  ...

myfun(1, 2.0) # works
myfun(b=2.0, a=1) # as above 

@dispatch 
def myfun(a :int, c: int):
  Pass

# this errors because the positional arguments do not have the correct names 

Something to solve would be how to add new positional argument names that were not defined in the abstract, but I'm sure it could be worked out...

[PhilipVinc]

The advantage of this is that by enforcing consistent naming of positional arguments, i think we get a 1to1 mapping between some keyword arguments and positional arguments and therefore we can keep using our model of MD based on positional arguments even if the user specified some keyword arguments.

The only counter-argument is that it would be harder to know, without looking at the original abstract definition, what keyword arguments are dispatched upon and which ones are not. But given that this feature has been asked by so many people, and it would be a step in the pythonization of plum, I think it's a price that might be worth paying.

And maybe we could even further improve on this idea...

[wesselb]

@PhilipVinc, I really quite like this idea!! Using @dispatch.abstract to define the mapping between keyword and positional arguments is also clever.

The only counter-argument is that it would be harder to know, without looking at the original abstract definition, what keyword arguments are dispatched upon and which ones are not.

We could extend the error messages to make this really clear.

@leycec and @Moosems, what would you think about the approach that @PhilipVinc proposes? Would an abstract definition that defines the mapping between keyword and positional arguments be a concession you'd be willing to make?

[leycec]

@PhilipVinc: Plum currently doesn't enforce consistent parameter naming, huh? Yeah. That's even lower-hanging fruit than keyword-only arguments. Pluck that Plum! :yum:

A one-to-one mapping from keyword to positional parameters definitely seems like the way. I briefly considered the exact same strategy for @beartype. But then laziness took over and dominated me. Now, I just run with the ad-hoc crotchety argument detection scheme @beartype has been using for what seems like a decade now... and probably is. Stellar stuff, @PhilipVinc!

[Moosems]

To be honest, I don’t understand exactly what’s going on here. Why does myfun not work with a and c only? I would recommend taking a look at this: https://stackoverflow.com/a/23228684

I’m considering making a side implementation for fun sometime soon that effectively stores the functions to a dictionary and if the inputs don’t match anything in the dict, it just throws an error. I don’t see why this is quite so complicated, but I can’t help but remember the developer learning curve that states that just when you think you know what’s going on and try it yourself, you realize you have no idea what’s actually happening.

[Moosems]

Here's my super quick and dirty implementation:

from inspect import FullArgSpec, getfullargspec

from beartype.door import is_bearable
from beartype.typing import Any, Callable

class Dispatcher:
    funcs: dict[str, list[tuple[Callable, FullArgSpec]]] = {}

    def __init__(self, func: Callable) -> None:
        self.name = func.__name__

        arg_spec: FullArgSpec = getfullargspec(func)

        # Keeps a single coherent list
        arg_spec.args.extend(arg_spec.kwonlyargs)
        for index, arg in enumerate(arg_spec.args):
            if arg not in arg_spec.annotations:
                arg_spec.annotations[arg] = Any  # Especially here

        if self.name in Dispatcher.funcs:
            Dispatcher.funcs[func.__name__].append((func, arg_spec))
            return

        Dispatcher.funcs[func.__name__] = [(func, arg_spec)]

    def __call__(self, *args, **kwargs) -> Any:
        funcs: list[tuple[Callable, FullArgSpec]] = Dispatcher.funcs[self.name]
        func: Callable | None = None
        positional_error: bool = False
        keyword_error: bool = False
        lookback_func: Callable | None = None
        extra_args: list[str] = list(kwargs.keys())
        for current_func, arg_spec in funcs:
            lookback_func = func

            # Start by checking that the positional args play nice
            continue_check = False
            args_len = len(args)
            used_args: list[str] = []
            for index, arg in enumerate(arg_spec.args):
                if not index < args_len:
                    break

                used_args.append(arg)

                if not is_bearable(args[index], arg_spec.annotations[arg]):
                    continue_check = True
                    break

            if continue_check:
                continue

            # Now check that the kweyword args play nice
            if not kwargs:
                unused_args = [
                    arg for arg in arg_spec.args if arg not in used_args
                ]
                if unused_args:
                    if lookback_func:
                        func = lookback_func
                        break

                    raise Exception(f"Args {unused_args} not in any function")

            continue_check = False
            for kwarg in kwargs:
                if kwarg not in arg_spec.args:
                    if arg_spec.varkw and extra_args:
                        continue
                    continue_check = True
                    break

                if not is_bearable(kwargs[kwarg], arg_spec.annotations[kwarg]):
                    continue_check = True
                    break

                if kwarg in extra_args:
                    extra_args.remove(kwarg)

            if continue_check:
                continue
            func = current_func

        if not func:
            raise Exception(
                f"Keyword arguments {extra_args} don't match any function!"
            )

        return func(*args, **kwargs)

# TODO: get the type checker to play nice with function overloading
@Dispatcher
def test(a) -> None:  # type: ignore
    print("Yuh Any")

@Dispatcher
def test(a: int) -> None:  # type: ignore
    print("Yuh int")

@Dispatcher
def test(a: float) -> None:  # type: ignore
    print("Yuh float")

@Dispatcher
def test(a: list) -> None:  # type: ignore
    print("Yuh list")

@Dispatcher
def test(a: list, b: int) -> None:  # type: ignore
    print("Yuh dual int")

@Dispatcher
def test(a: list, b: str) -> None:  # type: ignore
    print("Yuh dual str")

@Dispatcher
def test(a: list, *, b: list) -> None:  # type: ignore
    print("Yuh dual list")

@Dispatcher
def test(x: list, y: str, **kw) -> None:  # type: ignore
    print("Yuh kwargs")

test("")
test(5)
test(5.5)
test([5])
# test(c=[5])  # Should break
test([], 5)
test(a=[], b=5)
test([], "")
test(a=[], b="")
test(a=[], b=[])
test(x=[], y="", z=0)
# test(a=[], c="")  # Should break

Certainly not perfect or optimized, but it works fairly well although there's probably bugs in there I didn't catch in my basic tests. I don't know how you got the type checker to be nice to the function overloading but it looks like it has something to with this line but I don't care enough to implement that whole Function system for this mock idea. What do ya'll think of this as a rough draft?

[Moosems]

@wesselb soft ping

@dispatch
def f(x: int, y: float):
    ... # Method 1

@dispatch
def f(y: float, x: int):
   ... # Method 2

Then calling f(1, 1.0) would call method 1 and calling f(1.0, 1) would call method 2.

What's unclear to me is what f(x=1.0, y=1) would do.

It should break. I would never expect that to work at all. It doesn't match either of the signatures whatsoever. I get what you're saying, I just don't think this should work. If keyword enters the ring, it should dominate over positional. If I had something like:

@dispatch
def f(a: str, x: int, y: float):
    ... # Method 1
@dispatch
def f(a: str, y: float, x: int):
   ... # Method 2

And I run f("", x=1.0, y=1) I would also expect that to break. If I ran f("", 1.0, 1) I would expect that to run, however. I would also expect f("", 1.0, x=1), f("", 1, y=1.0), f("", x=1, y=1.0), and f("", y=1.0, x=1) to work. Does that make sense at all?

Also, nice seeing you here @gabrieldemarmiesse!

[wesselb]

@Moosems, I agree that a system like this could work: first resolve the positional arguments, then try to populate the keyword arguments.

The big issue with this is the following. Consider a generic function add(a, b) with an implementation add(a: int, b: int). Now another packages comes along, which provides a matrix type Matrix. This package wants to be able to add martices, so it implements add(x: Matrix, y: Matrix) or even add(b: Matrix, a: Matrix). Then add(obj1, b=obj2) will work for integers, but not for matrices. For keyword arguments to really work well without the possibility of confusing situations like this, it seems that you need consistent naming. This is what @PhilipVinc proposes.

If you search online, there's a fair bit of discussion about why Julia doesn't dispatch on named arguments. From this comment it seems that also there one of the objections is the lack of consistent naming.

[Moosems]

I don't follow that example. Could you produce a minimal example showing this more thoroughly? Why would this exclude matrices? I would expect that to exclude the integers instead.

[PhilipVinc]

Mojo does not allow dispatch on keyword arguments. It does something very similar to what plum does

[Moosems]

@PhilipVinc I'm fairly sure it does as the following produces no errors for me:

fn x(*, a: Int) -> None:
    pass

fn x(*, a: Bool) -> None:
    pass

def main():
    x(a=True)

[leycec]

Wierdo @beartype guy again. Is dispatching really this computationally expensive? I've never peeked under Plum's hood before, because I'm a polite guy. But @Moosems hypothetical __call__() implementation has me flustered and wildly waving my hands around. Even seemingly innocuous lines like...

                # This...
                unused_args = [
                    arg for arg in arg_spec.args if arg not in used_args
                ]

            # ...and this.
            for kwarg in kwargs:
                if kwarg not in arg_spec.args:

...are going to be shockingly slow. Both of those snippets exhibit average-case quadratic time complexity O(n**2) for n the number of parameters. And this time complexity is happening on every dispatch passed excess positional parameters or any keyword parameters (respectively).

Obviously, this isn't necessarily reflective of actual Plum behaviour. Plum is fast and good. But these sorts of horrifying call-time inefficiencies are why @beartype dynamically generates highly optimized code at decoration time rather than recomputing the same decision problems over and over and oooover again on every call (i.e., dispatch in Plum's case).

Obviously, Plum isn't going to metastasize into a freakishly muscular dynamic code generator anytime soon. Code generation is icky and non-trivial. It's also fun and intersects with parsing! I love it. Still, the impetus and the advantage is for Plum to eventually "go there." A useful long-term goal might be to consider refactoring Plum to eventually, slowly, and responsibly mutate into a dynamic code generator like @beartype.

This has been yet another ignorable dispatch from the wierdo @beartype guy. :smiling_face_with_tear:

[wesselb]

@Moosems This is the example a little more detail.

Author A makes a package math, which contains math functions that everyone is intended to use and extend. In particular, it defines a function add and implements add(a: int, b: int). Some user B of math uses this package to write add(some_object, other_object) in their code.

Then author C comes along and invents the package matrix, which defines a class Matrix. matrix imports math.add and implements add(a: Matrix, b: Matrix), so matrices can be added. The intention is that the code which user B wrote now also works for matrices, without modification! That is, it will now also work whenever some_object = Matrix() and other_object = Matrix(). Magic. :) This is a common multiple dispatch design pattern, which allows code to really easily and flexibly be extended. (Basically all of Julia works like this.)

The problem now arises if user B instead wrote add(a=some_object, b=other_object). In that case, if author C implemented add(x: Matrix, y: Matrix), then add(a=some_object, b=other_object) will break, because the keywords don't match up! In other words, to really benefit from the composibility and extensibility that multiple dispatch promises, if we allow keyword arguments, then it is really important that the naming is consistent.

---

@leycec, multiple dispatch unfortunately is expensive, but I believe that Plum currently does avoid O(n^2) costs where n is the number of parameters. Performance is currently optimising by caching as much as possible. If caching is not possible, dispatch becomes more expensive.

Interestingly, discussions online on why Julia doesn't dispatch on keyword arguments also appeal to computational complexity issues.

[leycec]

@wesselb: Super human-readable example. Thanks so much for the demystification. And... I can't believe "demystification" is actually a word. I typed that out expecting a sea of red. If only Firefox was so accepting of everything else I blast out on this ancient keyboard that predates the sinking of Atlantis.

So. Enforcing consistent nomenclature with the hammer of decoration-time exceptions is still Step 1, huh? Superb! @PhilipVinc is still right about everything. That's a quantum physicist for you, huh? :smile:

multiple dispatch unfortunately is expensive...

...heh. I've been thinking about this all day. Clearly, nobody except me wants to go there. Nonetheless, the low-hanging fruit here is unreal, @wesselb. Like, seriously. I am usually kidding, but I'm suddenly not kidding. This isn't simply paper-worthy; this isn't simply dissertation-worthy; this is career-worthy. Because...

I'm confident that multiple dispatch can efficiently be done with O(1) constant-time complexity. The approach? The exact same as @beartype. Basically, you split multiple dispatch into two phases:

1. Decoration-time. In this first phase, the @plum.dispatch decorator would dynamically generate a new function performing multiple dispatch in O(1) time over all decorated callables.
2. Call-time. In this second phase, the aforementioned function generated by @plum.dispatch would perform multiple dispatch in O(1) time.

Practical hyper-boring example or it didn't happen. Consider this excruciating example in which a boring user dispatches over a list containing either strings or integers: ^{...so, so very boring}

from plum import dispatch  # <-- boring boilerplate

@dispatch
def fun_with(funky_funcs: list[str]) -> str:
    '''
    So boring user-defined function #1.
    '''
    return ''.join(funky_funcs)

# Alias this function so that @leycec can exhibit O(1) multiple dispatch code.
fun_with_list_strs = fun_with  # <-- work with me here, people

@dispatch
def fun_with(funky_funcs: list[int]) -> int:
    '''
    So boring user-defined function #2.
    '''
    return sum(funky_funcs)

# Alias this function so that @leycec can exhibit O(1) multiple dispatch code.
fun_with_list_ints = fun_with  # <-- you will be assimilate!

Given the above, the @plum.dispatch decorator could act by dynamically replacing the fun_with() function(s) with a new fun_with() function exhibiting multiple dispatch in O(1) time. Behold! Multiple fast dispatch:

from random import getrandbits  # <-- ...heh

def fun_with(funky_funcs: list[str | int]) -> str | int:
    '''
    omg. wtf. lol. Insert emoji heads exploding, because heads are about to explode.
    '''

    # If the passed parameter is *NOT* a list, raise an exception.
    if not isinstance(funky_funcs, list):
        die_if_unbearable(funky_funcs, list)
    # Else, the passed parameter is a list.
    #
    # If this list is empty, arbitrarily defer to the first user-defined function.
    elif not funky_funcs:
        return fun_with_list_strs(funky_funcs)
    # Else, this list is non-empty.

    # Pseudo-random 32-bit integer.
    random_int = getrandbits(32)  # <-- exactly what @beartype does, yo

    # Pseudo-random item of this list.
    list_item = funky_funcs[random_int % len(funky_funcs)]  # <-- still what @beartype does

    # If this item is a string, defer to the "list[str]" variant of this function.
    if isinstance(list_item, str):
        return fun_with_list_strs(funky_funcs)
    # If this item is an integer, defer to the "list[int]" variant of this function.
    elif isinstance(list_item, int):
        return fun_with_list_ints(funky_funcs)
    # Else, this item is invalid. In this case, raise an exception.
    die_if_unbearable(list_item, str | int)

That's... absolutely O(1) multiple dispatch with negligible constants. That's also verbatim what @beartype does to deeply type-check lists in O(1) time.

In theory, Plum 3.0.0: Now With More Plum could leverage the same technique to deeply dispatch across arbitrary parameters in O(1) time as well. In practice, dynamic code generation is non-trivial. It makes migraines where previously no migraines existed. Probably nobody except a definite autist with no job prospects living in an uninsulated shack in the mosquito-infested woods of Canada wants to go there.

Still, going there is feasible. More importantly, going there yields tremendous gains with respect to theoretical scalability and real-world performance. I'm convinced that O(1) multiple dispatch is humanity's eventual future. Not today. Not tomorrow. Possibly not ever. But it's possible! Somebody, please make this possible miracle occur. Emoji Bob is pleading with you. :pleading_face:

[leycec]

Guh! Sorry about that tangent. I didn't mean to be quite so distracting. O(1) is a personal obsession that really has no relevance or bearing on this feature request. I'd still :heart_eyes: to (eventually) build out a private API in the @beartype codebase to dynamically generate O(1) multiple dispatch code, which Plum could then possibly defer to. But that's several years out. Okay. That's several decades out. Until then...

@PhilipVinc's 1:1 mapping from parameter names to indices is absolutely the way, @wesselb. Thanks again for this gripping discussion. So much QA fun! I leave the rest in your capable paws. :bear:

[Moosems]

The problem now arises if user B instead wrote add(a=some_object, b=other_object). In that case, if author C implemented add(x: Matrix, y: Matrix), then add(a=some_object, b=other_object) will break, because the keywords don't match up! In other words, to really benefit from the composibility and extensibility that multiple dispatch promises, if we allow keyword arguments, then it is really important that the naming is consistent.

Im still not entirely sure why that would break, just treat the keywords separately. a != x and b != y. Check that the keywords line up, then check that the types match the annotations, then check the positionals, and if all that works you’ve got a winner, no?

[wesselb]

@leycec and @Moosems, apologies for the slow reply! I didn't have much spare time last week.

I'm confident that multiple dispatch can efficiently be done with O(1) constant-time complexity. The approach? The exact same as https://github.com/beartype. Basically, you split multiple dispatch into two phases:

This is neat!! If I understand correctly, after defining all methods, you're proposing to dynamically generate an algorithm that performs dispatch for that particular collection of methods in O(1). Solving this problem in full generality sounds incredibly hard, but for many scenarios it's quite feasible to imagine what a near-optimal dispatch algorithm should look like.

Practical hyper-boring example or it didn't happen.

The example is O(1) in the number of elements in the list, but it runs linearly in the number of methods. In this particular example, however, that is easily resolved:

...

_type_to_method = {
    str: fun_with_list_strs,
    int: fun_with_list_ints,
}

...

        try:
            return _type_to_method[list_type(list_item)]
        except KeyError:
            die_if_unbearable(list_item, str | int)

Currently, I think Plum already runs O(1) in terms of the number of elements in a list, because it critically uses is_bearable everywhere! The primary gain is in the runtime w.r.t to the number of methods. If caching is not possible, then the current algorithm there actually runs in O(n^2) time (don't tell anyone...), but in practical scenarios it is near linear.

Of course, we don't necessarily have to solve the general case for this idea to give us the speed-ups we so badly want. After the methods are defined, we can check whether a dynamic algorithm can easily be generated. If so, e.g. in the case that all methods are of the form list[T], then go ahead! If not, fall back to the current behaviour. I think this is very possible. :)

Guh! Sorry about that tangent. I didn't mean to be quite so distracting.

No, please don't apologise! I think this is actually a very feasible and super fun idea. The next-in-line component of Plum to be improved definitely is the resolution algorithm, so ideas like this are super welcome.

I'd still 😍 to (eventually) build out a private API in the https://github.com/beartype codebase to dynamically generate O(1) multiple dispatch code, which Plum could then possibly defer to.

Haha, we would absolutely love this :P, but this would be a huge and incredibly difficult undertaking. However, if someone can do it, then it certainly would be the almighty @leycec!

---

Im still not entirely sure why that would break, just treat the keywords separately. a != x and b != y. Check that the keywords line up, then check that the types match the annotations, then check the positionals, and if all that works you’ve got a winner, no?

@Moosems, I don't quite follow you here. Are you proposing to automatically convert the keyword arguments to positional arguments and then perform dispatch that way? I think this would be incredibly error prone. For example, if we considered matmul instead of add, would you then do this?

matmul(a=matrix, b=another_matrix) -> matmul(matrix, another_matrix)

matmul(b=matrix, a=another_matrix) -> matmul(matrix, another_matrix)

I think you have to be really careful here, because the order of the arguments matters (matrix multiplication is not commutative). Normal Python functions also don't automatically convert keyword arguments to positional arguments in the case that keywords don't match.

It's possible that I am misunderstand what you are saying.

If we want keyword support, then it really seems to me that the keyword arguments need to be named consistently. However, if methods are defined by different code writers across different packages, then you really cannot guarantee that keywords are named consistently. Therefore, I'm becoming increasingly convinced that @PhilipVinc's idea is the way to go.

[Moosems]

No, thats not what I'm proposing. Let's take the following function:

def foo(bar: int, *, baz: float = 3.14, **kwargs) -> None:
    pass

Or even:

def foo(bar: int, *args, baz: float = 3.14, **kwargs) -> None:
    pass

These can be called in one of a few ways:

foo(1)  # Calls function 1
foo(bar=1)  # Calls function 1
foo(1, 2)  # Calls function 2
foo(bar=1, 2)  # doesn't work as expected
foo(2, bar=1)  # doesn't work as expected
foo(1, baz=3.14, spam=3)  # Calls function 1
foo(1, 2, baz=3)  # Calls function 2 and can only be 2
foo(1, 2, baz=3, spam=3)  # Calls function 2 and can only be 2
foo(1, 2, 3.14)  # Calls function 2 and can only be 2
foo(1, 2, 3.14, spam=3)  # Calls function 2 and can only be 2

In these functions we have a few things going on.

1. We have a positional (until given as a keyword) argument named bar. In the first function, this can only be positional. If it is given as a kwarg, all works fine! In the second example, however, if it is given as a keyword argument, that negates the ability to add extra positional arguments.
2. baz can be given positionally in the second example, but must be given as a keyword argument in the first.
3. Extra keyword arguments are allowed in both.

I would hope none of this causes issues as all works as promised based on my prep code tests for this comment. Now, your concern is if I override one function with one set of names with another function that takes other keyword arguments. Let's take some example functions her as well:

@dispatch
def foo(bar: int, *, baz: float = 3.14, **kwargs) -> None:
    pass

@dispatch
def foo(bar: int, *args, baz: float = 3.14, **kwargs) -> None:
    pass

@dispatch
def foo(spam: float, *, eggs: int = 3.14, **kwargs) -> None:
    pass

@dispatch
def foo(spam: float, *args, eggs: int = 3.14, **kwargs) -> None:
    pass

If I ran the following, I would expect it to work without issues:

# Now matches to the first registered function that fits
foo(1)  # Matches function 1
foo(bar=1)  # Matches function 1
foo(1, 2)  # Matches function 2
foo(1, baz=3.14, spam=3)  # Matches function 1
foo(1, 2, baz=3)  # Matches function 2
foo(1, 2, baz=3, spam=3)  # Matches function 2
foo(1, 2, 3.14)  # Matches function 2
foo(1, 2, 3.14, spam=3)  # Matches function 2
foo(3.14)  # Matches function 3
foo(spam=3.14)  # Matches function 4
foo(3.14, 2)  # Matches function 4
foo(3.14, 2, eggs=1)  # Matches function 4
foo(3.14, qux=1)  # Matches function 3
foo(3.14, 2, eggs=3, qux=1)  # Matches function 4

Now lets analyze the add function. We'll take the following:

class Matrix: ...

@dispatch
def add(a: int, b: int) -> int:
    return a + b

@dispatch
def add(a: Matrix, b: Matrix) -> int:
    return a + b

@dispatch
def add(x: Matrix, y: Matrix) -> int:
    return a + b

All of this is simple enough. Now lets imagine running it.

add(1, 2)  # Works
add(1, b=2)  # Works
add(a=1, b=2)  # Works
add(Matrix(), Matrix())  # Works
add(Matrix(), b=Matrix())  # Works
add(a=Matrix(), b=Matrix())  # Works
add(x=Matrix(), y=Matrix())  # Works
add(a=Matrix(), y=Matrix())  # Should NOT work
add(x=Matrix(), b=Matrix())  # Should NOT work

Yes, people should be careful naming their keyword arguments, but thats not Plum's problem. If I, as developer C, choose to deviate from a & b and use x && y, I would expect that one could not interchange the keyword names. If there are more cases you would like me to explain or logic out, I'm more than willing to do so, I simply don't understand what all this fuss is about.

Let's even consider the folllowing:

matmul(a=matrix, b=another_matrix) -> matmul(matrix, another_matrix)

matmul(b=matrix, a=another_matrix) -> matmul(another_matrix, matrix)

Moving around the kwargs should have no effect unless a * is present, thats the whole point of kwargs. I particularly like the following naming conventions:

• Positional argument
• Keyword argument
• Keyword only argument
• argument (function definition)
• parameter (function call) (Yes, these are generally swapped but since the rest of this used argument I don't care to rewrite it so I'll just live with it)

Positional arguments stay positional unless given as a keyword parameter which converts all positional and keyword arguments to keyword only arguments. With keyword arguments, they can be mapped as positional arguments unless given as a keyword parameter in which case all following keyword arguments become keyword only arguments. If a * is given, all keyword arguments become keyword only arguments. With keyword only arguments, they must be given as a keyword parameter.

What this means is someone could not use a and b with author C's add function because the names never line up but that's not Plum's problem. That's a problem for author C to deal with. He should probably specify that in documentation and make it known that he chose to break away from conventions and why but, again, not Plum's problem.

[wesselb]

@Moosems, thanks for typing this out in such detail! I'm in near-complete agreement with everything you've written. In particular, the way methods are matched is very reasonable, and basically equal to how Python does it, if I'm not mistaken.

Yes, people should be careful naming their keyword arguments, but thats not Plum's problem. If I, as developer C, choose to deviate from a & b and use x && y, I would expect that one could not interchange the keyword names.

This is the pain point. In a system that does not dispatch on keyword arguments, inconsistent argument names is not an issue. Technical issues aside, the question is therefore whether the convenience of dispatching on keyword arguments is worth the making dispatch sensitive to the names of arguments, and this I'm not sure about. When you extend a function, you would not only have to implement it for the right types, but also with the right argument names, and there would be no mechanism in place that prevents you from incorrectly naming the arguments.

I can tell that you strongly believe that Plum shouldn't worry about this, but I'm not so sure. Robustness by design is really nice thing. I realise that this is Python, but I'm particularly inclined to stick to how Julia does things, since multiple dispatch there just works so super well. What do others think about this trade-off?

[Moosems]

@wesselb Maybe a decorator flag that is on by default which gives errors if all the old kwargs cannot be found but leaves it up to the user when off?

Maybe something like:

class Matrix: ...

@dispatch
def add(a: int, b: int) -> int:
    return a + b

@dispatch(same_kwargs=True) # Is true by default, I'm just being verbose
def add(a: Matrix, b: Matrix) -> int:
    return a + b

@dispatch(same_kwargs=False)
def add(x: Matrix, y: Matrix) -> int:
    return a + b

[wesselb]

@Moosems Doesn't look too unreasonable!

Generally, it seems that the desired algorithm for supporting keyword arguments is this:

1. For all methods, match all given arguments to function arguments exactly like how Python does it.
2. Eliminate the methods where the given arguments are not instances of the types.
3. From the resulting methods, find the most specific one. If there is none, raise a look-up error. If there is more than one, raise an ambiguity error.

Suppose that for step 3, suppose that the resulting methods are the following:

def algorithm(x: Number, config1: Config1, config2: Config2):
    pass 

def algorithm(x: Number, config2: SubConfig2, config1: SubConfig1):
    pass

Since SubConfig1 is a subtype of Config1 and SubConfig2 is a substype of Config2, I think people would expect this to be automatically narrowed down to the second method. However, this definitely shouldn't happen, as it would be in contradiction to how dispatch words for positional arguments.

Hence, it seems to me that the pain point manifests itself in two ways:

1. Dispatch becomes sensitive to argument names. Consequently, arguments need to be named consistently.
2. The use of keyword arguments suggests independence of argument order, as algorithm(x, config1=config1, config2=config2) is equivalent to algorithm(x, config2=config2, config1=config1). However, the algorithm that determines the most specific method is sensitive to the argument order. Consequently, arguments typically used as keyword arguments (like configuration parameters) need to be defined in the right order.

None of these problems are insurmountable, but the whole system does become significantly more brittle.

[Moosems]

Keywords are positional until they are specified as keyword only or a keyword argument is named previously in the order, but, yes, I can see the issue. I think the ball is in your court here and you make the call :)

[Moosems]

I think it would be best if I described my wishful use case for plum. I have a package named Salve that allows user to use IPC.request(). I don't like the look of having a bunch of IPC.request_xyz() so I currently have the .request() method default all the values for any given use case. I could imagine a simpler implementation of this with plum where I describe each action in its own dispatch function and that's the hope I have. Probably not even possible, but I can dream! The docs and the code in case you want to see what I mean

[Moosems]

That's what ultimately drives the design I'm pushing for

[wesselb]

@PhilipVinc, @leycec, what would you say is the best way forward?

@Moosems, thanks for linking your use case. :) In this specific case, unless I'm misunderstanding, I believe that you're only dispatching on command: COMMAND. The docs seem to always specify the values for the command as keyword arguments. Therefore, in this particular scenario, you could approach it in this way:

@dispatch
def request(command: Literal["COMMAND1"], *, option1: bool, ...):
    ... 

@dispatch
def request(command: Literal["COMMAND2"], *, option1: bool, ...):
    ... 

Clearly this isn't optimal, because it only dispatches on command, but it might suffice for this particular scenario.

[Moosems]

That would work amazingly :). Keyword arguments would just be the cherry on top, ya know?

[wesselb]

@Moosems Definitely agree that it would be nice to be able to dispatch on keyword arugments too! In your case, the keyword arugments seem to be the same for every call, so, if I'm not mistaken, you could use the wrapper method approach:

def request(
    self,
    command: COMMAND,
    file: str = "",
    expected_keywords: list[str] = [""],
    current_word: str = "",
    language: str = "Text",
    text_range: tuple[int, int] = (1, -1),
    file_path: Path | str = Path(__file__),
    definition_starters: list[tuple[str, str]] = [("", "before")],
) -> None:
    return _request(command, file, expected_keywords, current_word, language, text_range, file_path, definition_starters)

@dispatch
def _request(command: Literal["COMMAND1"], file: str, ...):
    ...

@dispatch
def _request(command: Literal["COMMAND2"], file: str, ...):
    ... 

[PhilipVinc]

And I believe @Moosems 's use case would be satisfied by my proposal?

@wesselb I'm unsure what exactly you were proposing before.

I was thinking a bit more about my proposal of forcing consistent naming of the positional arguments and I think it has the following extra benefits:

• We can, at method definition time, raise informative errors telling the user that his n-th argument is not consistent with what is expected and that he should name that argument 'that way'.
  • I wonder if there's a way to disable an argument in some way? Like add a default None value
• We can extend it and make it that if we define a new method with N+1 more positional argument, all future methods with >N positional arguments must name the N+1-th like that new method. If the ordering of importing files/packages is changed, as all must be consistent, no error will be raised and the logic will still work. If the naming is inconsistent an error will be raised from a different location, but it would error anyway....

Also:

• This approach allows us to dispatch only on some keyword arguments (only on the positional as keyword ones), while the keyword-only arguments remain non-dispatched upon.
  • This means that the wrapper method approach will no longer be needed.

Overall, I think this approach (optionally) hides what arguments are dispatched upon from the end-user, but allows us to give informative errors to whom defines the methods...

Plus, the implementation should be relatively easy...

[Moosems]

@wesselb Kind of. I more only use certain keyword arguments based on the command. I provide default values for each such that there aren't errors raised and if the user forgets a value, they get the provided one

[Moosems]

For example: every command but the editorconfig command requires file

[wesselb]

@PhilipVinc, I think this sounds great. I'm on board! The extension mechanism sounds sensible, but perhaps we should first see if that's something that's really necessary. I like the sound of using @dispatch.abstract to define the positional arguments once and for all.

---

For example: every command but the editorconfig command requires file

@Moosems, in this case, unless I'm mistaken, you require different keyword arguments for the commands, but you don't really require to dispatch on these keywords. I.e., this is allowed and should work as expected:

@dispatch
def request(command: Literal["highlight"], *, file: str, some_option: bool, ...):
    ... 

@dispatch
def request(command: Literal["editorconfig"], *, other_option: bool, ...):
    ... 

(You can of course also set default values without things going wrong.)