`TypeError: Test.__call__() got an unexpected keyword argument 'a'`

[sylvorg]

Hello!

With the following code:

from rich.traceback import install

install(show_locals=True)

from plum import dispatch

class Test:
    @dispatch
    def __call__(self):
        pass

    @dispatch
    def __call__(self, *args, **kwargs):
        pass

test = Test()
test(a=1)

I get the following traceback:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /mnt/wsl/sylvorg/sylvorg/sylveon/siluam/oreo/./test15.py:20 in <module>                          │
│                                                                                                  │
│   17 │   │   pass                                                                                │
│   18                                                                                             │
│   19 test = Test()                                                                               │
│ ❱ 20 test(a=1)                                                                                   │
│   21                                                                                             │
│                                                                                                  │
│ ╭───────────────────────────── locals ─────────────────────────────╮                             │
│ │ dispatch = <plum.dispatcher.Dispatcher object at 0x7f4713206350> │                             │
│ │  install = <function install at 0x7f4713d0bce0>                  │                             │
│ │     Test = <class '__main__.Test'>                               │                             │
│ │     test = <__main__.Test object at 0x7f47134f7950>              │                             │
│ ╰──────────────────────────────────────────────────────────────────╯                             │
│                                                                                                  │
│ /nix/store/0rqp0565hq7xhg1n0ld36icv3031f87s-python3-3.11.8-env/lib/python3.11/site-packages/plum │
│ /function.py:484 in __call__                                                                     │
│                                                                                                  │
│   481 │   │   pass                                                                               │
│   482 │                                                                                          │
│   483 │   def __call__(self, _, *args, **kw_args):                                               │
│ ❱ 484 │   │   return self._f(self._instance, *args, **kw_args)                                   │
│   485 │                                                                                          │
│   486 │   def invoke(self, *types):                                                              │
│   487 │   │   """See :meth:`.Function.invoke`."""                                                │
│                                                                                                  │
│ ╭───────────────────────────── locals ──────────────────────────────╮                            │
│ │       _ = <__main__.Test object at 0x7f47134f7950>                │                            │
│ │    args = ()                                                      │                            │
│ │ kw_args = {'a': 1}                                                │                            │
│ │    self = <plum.function._BoundFunction object at 0x7f47132bc0d0> │                            │
│ ╰───────────────────────────────────────────────────────────────────╯                            │
│                                                                                                  │
│ /nix/store/0rqp0565hq7xhg1n0ld36icv3031f87s-python3-3.11.8-env/lib/python3.11/site-packages/plum │
│ /function.py:368 in __call__                                                                     │
│                                                                                                  │
│   365 │   def __call__(self, *args, **kw_args):                                                  │
│   366 │   │   __tracebackhide__ = True                                                           │
│   367 │   │   method, return_type = self._resolve_method_with_cache(args=args)                   │
│ ❱ 368 │   │   return _convert(method(*args, **kw_args), return_type)                             │
│   369 │                                                                                          │
│   370 │   def _resolve_method_with_cache(                                                        │
│   371 │   │   self,                                                                              │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │        args = (<__main__.Test object at 0x7f47134f7950>,)                                    │ │
│ │     kw_args = {'a': 1}                                                                       │ │
│ │      method = <function Test.__call__ at 0x7f471383c540>                                     │ │
│ │ return_type = typing.Any                                                                     │ │
│ │        self = <multiple-dispatch function Test.__call__ (with 2 registered and 0 pending     │ │
│ │               method(s))>                                                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: Test.__call__() got an unexpected keyword argument 'a'

It doesn't seem to be happening with the following blocks either:

from rich.traceback import install

install(show_locals=True)

from plum import dispatch

class Test:
    @dispatch
    def __call__(self, *args):
        pass

    @dispatch
    def __call__(self, *args, **kwargs):
        pass

test = Test()
test(a=1)

from rich.traceback import install

install(show_locals=True)

from plum import dispatch

class Test:
    @dispatch
    def __call__(self):
        pass

    @dispatch
    def __call__(self, **kwargs):
        pass

test = Test()
test(a=1)

Thank you kindly for the help!

[sylvorg]

Interestingly, this does not seem to be working either, failing with the same message:

from rich.traceback import install

install(show_locals=True)

from plum import dispatch

class Test:
    @dispatch
    def __call__(self, **kwargs):
        pass

    @dispatch
    def __call__(self):
        pass

test = Test()
test(a=1)

[wesselb]

Hey @sylvorg, what's going on is the following:

Firstly, it is important to know that Plum ignores keyword arguments in determining whether functions are the same or not:

from plum import dispatch

@dispatch
def f(x: int):
    return 1

@dispatch
def f(x: int, **kw_args):  # Overwrites the above, because keyword arguments are ignored in registering functions!
    return 2

@dispatch
def f(x: int, *, a=1, **kw_args):  # Overwrites above both two!
    return 3

assert f(1) == 3

@dispatch
def f(x: int, *, b=1):  # Overwrites all above methods! Now only keyword `b` is available!
    return 4

f(1, a=1)  # TypeError: f() got an unexpected keyword argument 'a'

However, Plum does not ignore *args:

from plum import dispatch

@dispatch
def f(x: int):
    return 1

@dispatch
def f(x: int, *args):
    return 2

assert f(1) == 1
assert f(1, 1) == 2

Therefore, in your case, the following is happening:

class Test:
    @dispatch
    def __call__(self):
        pass

    @dispatch
    def __call__(self, *args, **kwargs):  # New definition for `__call__`!
        pass

When you call Test(a=1), keyword arguments are ignored: it effectively looks for Test(). The most specific method matching this is the first definition: def __call__(self). And this definition does not implement keyword arguments, giving you the error.

In the other two cases, the second __call__ overwrites the first __call__ because both have *args or neither have *args.

I hope that makes sense!

[sylvorg]

Hmm... Would it be particularly difficult implementing support for keyword arguments? I would have thought that the keyword name makes the resolution easier, and any keyword names and their values not matching the ones already in the resolved functions will either go to a function with **kwargs (or any other variable keyword argument), or result in an NotFoundLookupError.

[wesselb]

@sylvorg We've had a fair bit of discussion around keyword arguments. Very long story short is that supporting keyword arguments isn't entirely straightforward.

Plum's design basically mimics how multiple dispatch in the Julia programming works, including the treatment of keyword arguments.

[sylvorg]

Hmm... Does this mean none of my plum methods can use keywords arguments? At all? As they would always be overridden?

[wesselb]

@sylvorg Definitely not! It just means that which methods are considered the same (i.e. what would be a redefinition) isn't influenced by keyword arguments. For example, this should work fine:

from plum import dispatch

@dispatch
def f(x: int, *, option1 = None):
    return x

@dispatch
def f(x: str, *, option2 = None):
    return x

>>> f(1, option1="value")  # OK

>>> f(1, option2="value")  # Not OK

>>> f("test", option1="value")  # Not OK

>>> f("test", option2="value")  # OK

Keyword arguments without *, are treated in this way:

@dispatch
def f(x: int, y: str="hey"):
    ...  # Implementation

is converted to

@dispatch
def f(x: int):
    y = "hey"
    ...  # Implementation

@dispatch
def f(x: int, y: str):
    ...  # Implementation

[sylvorg]

Ah; so the names are matched in the case of keyword arguments, not the types. Got it. I just can't accept variable keyword arguments in a plum method, right? Those would be overridden, so just one of them or the last one would work.

[sylvorg]

Actually, could you help me create a patch for my own version of plum that would redirect any unknown keyword arguments to a function with **kwargs? Which file(s) would I have to modify to get that functionality?

[wesselb]

@sylvorg You should be able to use variable keyword arguments. Let me try to put together an example of what might be the desired behaviour. I’ll do that a little later

[wesselb]

Since keyword arguments are ignored in the dispatch, you won't be able to create versions of methods for specific keyword arguments. However, you could emulate this behaviour in a way like this:

from plum import dispatch

@dispatch
def f(x, **kw_args):
    if kw_args:
        return f.invoke(object, object)(x, **kw_args)
    print("Only one argument and no keywords!")

@dispatch
def f(*args, **kw_args):
    print("Fallback:", args, kw_args)

Actually, could you help me create a patch for my own version of plum that would redirect any unknown keyword arguments to a function with **kwargs?

I think this might be considerably hard to get right. If I simple solution like the above suffices, my advice would be to go with that.

[sylvorg]

So basically I should just make sure that, if I use **kwargs, I should put them in every function of the same name, then act based on whether kwargs is empty or not?

[wesselb]

@sylvorg, hmm, if you truly want to achieve dispatch based on which keyword arguments are given, you will unfortunately need to emulate that behaviour.

Perhaps another approach is possible, which has been proposed in other issues:

def f(x, *, option=None):
    return _f(x, option)

@dispatch
def _f(x, option: None):
    ...

@dispatch
def _f(x, option: str):
    ...

The idea is that you convert keyword arguments into positional arguments in a fixed order by using a wrapper function, and then use Plum's dispatch for positional arguments.

Could something like that be OK for your use case?