Open maksbotan opened 3 months ago
Welcome 👋 And thanks a lot for the detailed bug report and reproducer.
I suspect that this might be a very tricky case to solve.
Actually, the reproducer can be simplified even further. Error is triggered without multiprocessing
:
import pickle
class Foo(Exception):
foo: int
bar: str
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
if __name__ == '__main__':
p = pickle.dumps(Foo(foo=42, bar="a"))
f = pickle.loads(p)
print(f)
Traceback:
Traceback (most recent call last):
File "/data/check_mp.py", line 22, in <module>
f = pickle.loads(p)
TypeError: Foo.__init__() missing 2 required positional arguments: 'foo' and 'bar'
Removing Exception
or using positional arguments fixes the problem.
It's documented in the pickle docs that classes which require keyword arguments for their constructor must implement __getnewargs_ex__
to work with pickle: https://docs.python.org/3/library/pickle.html#object.__getnewargs_ex__.
Perhaps we can improve the docs and/or the error message, however.
Oh, but your constructor doesn't require keyword arguments. You just use keyword arguments. Hm.
It's documented in the pickle docs that classes which require keyword arguments for their constructor must implement
__getnewargs_ex__
to work with pickle: https://docs.python.org/3/library/pickle.html#object.__getnewargs_ex__.Perhaps we can improve the docs and/or the error message, however.
First of all, how am I supposed to discover that if I'm using multiprocessing? I forgot that it uses pickle under the hood.
Second, docs talk about __new__
:
You should implement this method if the
__new__()
method of your class requires keyword-only arguments.
My class does not have __new__
explicitly defined, so reading the docs I do not assume that this sentence is relevant to my case 🤔
Possibly related: https://github.com/python/cpython/issues/89275 and https://github.com/python/cpython/issues/89275#issuecomment-1093928193.
I don't have a setup to check the current implementation now, but maybe Irit's comment is still valid:
The default
__reduce__
method of Exception returns the arg you pass to the Exception constructor.
So it might happen that keyword arguments are simply ignored (or not remembered as such?).
Yes, the keyword arguments are simply ignored.
BaseException.args
- what is it? Doc says: "The tuple of arguments given to the exception constructor.", However:
class Foo(Exception):
def __init__(self, x, y):
self.x = x
self.y = y
a = Foo(1, y=2)
print(a.args)
> (1,)
So, should we also have a BaseException.kwargs
or BaseException.args
should include all passed arguments: positional, positional-only, keyword, keyword only?
Alternatively, I'd suggest changing the docs to handle those cases, by saying something like "if your custom class exception supports keyword-arguments in the constructor (required or not), then you must implement a custom reduction". That's one way to easily solve the issue (and say that .args
are only the positional arguments (at the moment of the call, not in the signature)).
Or your idea of having .kwargs
would also make sense (so that .args
and .kwargs
would reflect the call). But I'm not sure if this could break something.
Alternatively, I'd suggest changing the docs to handle those cases, by saying something like "if your custom class exception supports keyword-arguments in the constructor (required or not), then you must implement a custom reduction". That's one way to easily solve the issue (and say that
.args
are only the positional arguments (at the moment of the call, not in the signature)).
We still need to update the existing exceptions, like ImportError
:
>>> a = ImportError(name="foo")
>>> a.args
()
>>> b = ImportError("foo")
>>> b.args
('foo',)
>>>
So, we need to decide whether we should add a BaseException.kwargs
and support it or, easier and preferable (in my opinion), state this behavior in the docs and adapt existing exceptions to work well with the existing behavior.
Formally speaking, "The tuple of arguments given to the exception constructor" is correct since "argument" is not used when talking about a signature (where we talk about parameters), so it should indeed refer to what was given to the constructor call (thought we could specify that args only store the positional arguments of the call).
I also prefer first changing the docs before introducing something else.
I encountered this issue along with @maksbotan, and in my opinion, updating the Exception documentation doesn't really help here. My reasoning is as follows:
The error message we received from pickle indicated that some constructor lacked certain arguments but did not specify why. In our real-world scenario, the code was more complex than the simplified example we provided, so initially, I had no idea what was causing the problem. There is no situation where my first instinct would be to scrutinize the Exception documentation. For example, we suspected that the dataclass decorator (which our Exceptions also had) might be causing the issue, so we spent some time refactoring it. After running some tests, we saw that this approach didn't help. We went through several other guesses similarly. Even if this note were in the documentation, it would still take us a considerable amount of time to find it.
In my opinion, this behavior is neither natural nor expected. Sure, there may be a deep technical explanation for why this happens, but it doesn't make sense to argue that this should happen. I would prefer that Exception-derived classes are not treated as exceptions (no pun intended) by the pickling mechanism, rather than simply having this noted somewhere in the documentation.
Bug report
Bug description:
I've tried using an
Exception
subclass with some fields and a simple constructor:When I construct and return this exception from a function invoked through
ProcessPoolExecutor
'smap
method, I get unpickling error:(For simpler reproducer see https://github.com/python/cpython/issues/120810#issuecomment-2182228877)
Traceback on 3.13.0b2 from official Docker image
Same error happens on 3.10, 3.12, and also on MacOS. Different fork method (
fork
,forkserver
,spawn
) all behave in the same way, both on Linux and MacOS.However, if I construct
Foo
with positional arguments, there is no error:Output:
And, most strangely, if I remove
Exception
base class (leaving justclass Foo():
) there is no error with keyword arguments.I've found only one similar issue: #103352, but that change does not fix my issue (tested on latest 3.13 beta).
Currently we workarounded this issue in our code by using positional arguments, but it was very annoying to debug this and getting to solution actually took us a couple of days.
CPython versions tested on:
3.10, 3.12, 3.13
Operating systems tested on:
Linux, macOS