Exception copy error

[321b4b3e-75d5-44c9-b8ef-7555e8fd41ba]

BPO	43460
Nosy	@iritkatriel, @douglas-raillard-arm

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-bug', 'library', '3.11'] title = 'Exception copy error' updated_at = user = 'https://github.com/douglas-raillard-arm' ``` bugs.python.org fields: ```python activity = actor = 'iritkatriel' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'douglas-raillard-arm' dependencies = [] files = [] hgrepos = [] issue_num = 43460 keywords = [] message_count = 3.0 messages = ['388427', '388428', '396600'] nosy_count = 2.0 nosy_names = ['iritkatriel', 'douglas-raillard-arm'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue43460' versions = ['Python 3.11'] ```

[321b4b3e-75d5-44c9-b8ef-7555e8fd41ba]

Instances of subclasses of BaseException created with keyword argument fail to copy properly as demonstrated by:

    import copy

    class E(BaseException):
        def __init__(self, x):
            self.x=x

    # works fine
    e = E(None)
    copy.copy(e)

    # raises
    e = E(x=None)
    copy.copy(e)

This seems to affect all Python versions I've tested (3.6 \<= Python \<= 3.9).

I've currently partially worked around the issue with a custom pickler that just restores __dict__, but:

• "args" is not part of __dict, and setting "args" key in __dict does not create a "working object" (i.e. the key is set, but is ignored for all intents and purposes except direct lookup in __dict__)

• pickle is friendly: you can provide a custom pickler that chooses the reduce function for each single class. copy module is much less friendly: copyreg.pickle() only allow registering custom functions for specific classes. That means there is no way (that I know) to make copy.copy() select a custom reduce for a whole subclass tree.

One the root of the issue:

• exception from the standard library prevent keyword arguments (maybe because of that issue ?), but there is no such restriction on user-defined classes.
• the culprit is BaseException_reduce() (in Objects/exceptions.c) [1]

It seems that the current behavior is a consequence of the __dict__ being created lazily, I assume for speed and memory efficiency

There seems to be a few approaches that would solve the issue:

• keyword arguments passed to the constructor could be fused with the positional arguments in BaseException_new (using the signature, but signature might be not be available for extension types I suppose)

• keyword arguments could just be stored like "args" in a "kwargs" attribute in PyExceptionHEAD, so they are preserved and passed again to \_new__ when the instance is restored upon copying/pickling.

• the fact that keyword arguments were used could be saved as a bool in PyException_HEAD. When set, this flag would make BaseExceptionreduce() only use \_dict and not "args". This would technically probably be a breaking change, but the only cases I can think of where this would be observable are a bit far fetched (if __new or __init have side effects beyond storing attributes in __dict).

[1] https://github.com/python/cpython/blob/master/Objects/exceptions.c#L134

[321b4b3e-75d5-44c9-b8ef-7555e8fd41ba]

The solution based on the signature is something along those lines:

    class E(BaseException):
        def __new__(cls, *args, **kwargs):
            """
            Fix exception copying.

        Turn all the keyword arguments into positional arguments, so that the
        :exc:`BaseException` machinery has all the parameters for a valid call
        to ``__new__``, instead of missing all the keyword arguments.
        """
        sig = inspect.signature(cls.__init__)
        bound_args = sig.bind_partial(*args, **kwargs)
        bound_args.apply_defaults()
        args = tuple(bound_args.arguments.values())
        return super().__new__(cls, *args)

        def __init__(self, x):
            self.x=x

But there are a many shortcomings to that approach:

• What if super().__new() consumes arguments before passing the rest to __init() ? This fix is blind to that since it only cares about __init__ signature

• What if inspect.signature() does not provide a signature (extension modules) ?

• Defaults are "hardcoded" in the args, so the object will always be restored with the defaults of the time it was created. This is a breaking change, as currently the defaults used when restoring the instance are the current ones.

• Also uses more memory for args (and for pickle files), since it contains all the defaults

[iritkatriel]

See bpo-32696, bpo-30005, bpo-29466