Undefined objects can't be copied or pickled by Python > 3.5

nitzmahone commented 3 weeks ago

Description

Due to an ancient erroneous deletion of __slots__ metadata, it's not possible to copy/deepcopy/pickle Undefined objects (all of which we occasionally need to do in Ansible) in Python > 3.5.

The comment above the deletion of __slots__ implies a misunderstanding of how that metadata was/is used. The deletion was always probably technically a bug, but once Python started providing default implementations of __getstate__ and __reduce_ex__ on all objects (which consult that metadata at runtime), it broke the default copyability/pickle-ability of Undefined (and derived) objects.

We're working around this with a monkeypatch that restores the missing __slots__ metadata. We've also submitted PR #2026 to zap the problematic deletion and tests that verify copy/deepcopy/pickle functionality on all the built-in Undefined types.

Repro

(on any Python > 3.5 with Jinja installed):

$ python -c 'import copy; import jinja2; copy.copy(jinja2.Undefined())'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib64/python3.11/copy.py", line 92, in copy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle 'Undefined' object

Expected

The default runtime-provided implementations of __getstate__ and __reduce_ex__ that back copy.copy, copy.deepcopy and pickle should work properly with all Undefined derived types.

Environment

Python version: >= 3.6
Jinja version: anything from at least the past 16 years

davidism commented 3 weeks ago

I've been confused by that code as well. If the comment isn't accurate and everything still works, I'm fine with a PR for that.

nitzmahone commented 3 weeks ago

Cool- there's another small related issue + fix inbound that we tripped over while writing the tests for this one (my colleague is filing a separate issue + PR right now).

pallets / jinja