python / cpython

The Python programming language
https://www.python.org
Other
63.41k stars 30.37k forks source link

Race condition in WeakKeyDictionary/WeakValueDictionary #89967

Closed colesbury closed 4 weeks ago

colesbury commented 2 years ago
BPO 45809
Nosy @colesbury
Files
  • issue45809-repro.patch: Patch to make reproduction easier
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-bug', 'library', '3.9', '3.10', '3.11'] title = 'Race condition in WeakKeyDictionary/WeakKeyDictionary' updated_at = user = 'https://github.com/colesbury' ``` bugs.python.org fields: ```python activity = actor = 'colesbury' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'colesbury' dependencies = [] files = ['50440'] hgrepos = [] issue_num = 45809 keywords = ['patch'] message_count = 2.0 messages = ['406357', '406358'] nosy_count = 1.0 nosy_names = ['colesbury'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue45809' versions = ['Python 3.9', 'Python 3.10', 'Python 3.11'] ```

    Linked PRs

    colesbury commented 2 years ago

    The issue described bpo-7105 (and maybe bpo-7060) still exists due to a race condition in WeakKeyDictionary. This shows up as test failure that looks like:

      test test_weakref failed -- Traceback (most recent call last):
        File "Lib/test/test_weakref.py", line 1960, in test_threaded_weak_value_dict_deepcopy
          self.check_threaded_weak_dict_copy(weakref.WeakValueDictionary, True)
        File "Lib/test/test_weakref.py", line 1940, in check_threaded_weak_dict_copy
          raise exc[0]
        File "Lib/test/test_weakref.py", line 1897, in dict_copy
          _ = copy.deepcopy(d)
        File "Lib/copy.py", line 153, in deepcopy
          y = copier(memo)
        File "Lib/weakref.py", line 189, in __deepcopy__
          for key, wr in self.data.items():
      RuntimeError: dictionary changed size during iteration

    The cause is that the check of "self._iterating" and the call to "_atomic_removal" are not performed atomically together. By the time _atomic_removal() is called, an iteration might have already started.

    https://github.com/python/cpython/blob/ec382fac0db6d9159c2d3496a70b7a605545957e/Lib/weakref.py#L109-L114

    colesbury commented 2 years ago

    The attached patch (bpo-45809-repro.patch) introduces artificial delays to make reproduction of the underlying issue easier.

    To reproduce the issue:

    patch -p1 \< bpo-45809-repro.patch ./python -m test test_weakref -m test_threaded_weak_value_dict_deepcopy -v