mailgun / expiringdict

Dictionary with auto-expiring values for caching purposes.
Apache License 2.0
347 stars 76 forks source link

RuntimeError with Python3.5 #25

Open oz123 opened 7 years ago

oz123 commented 7 years ago

I am seeing a very annoying bug when using this library with Python 3.5:


In [1]: import sys
   ...: 
   ...: # Python 3.5 has a broken 'optimised' version of OrderedDict which can corrupt itself at high load.
   ...: if sys.version_info.major == 3 and sys.version_info.minor == 5:
   ...:     from test import support
   ...: 
   ...:     py_coll = support.import_fresh_module('collections', blocked=['_collections'])
   ...:     OrderedDict = py_coll.OrderedDict
   ...: else:
   ...:     from collections import OrderedDict
   ...:     

In [2]: from expiringdict import ExpiringDict

In [3]: ExpiringDict(max_len=60, max_age_seconds=4)
Out[3]: ExpiringDict()

In [4]: d=ExpiringDict(max_len=60, max_age_seconds=4)

In [5]: d["soc3"] = {"soc":14}

In [6]: d["soc2"] = {"soc":11}

In [7]: d
Out[7]: ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/.virtualenvs/hagenv2g/lib/python3.5/site-packages/IPython/core/formatters.py in __call__(self, obj)
    670                 type_pprinters=self.type_printers,
    671                 deferred_pprinters=self.deferred_printers)
--> 672             printer.pretty(obj)
    673             printer.flush()
    674             return stream.getvalue()
~/.virtualenvs/hagenv2g/lib/python3.5/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    366                 if cls in self.type_pprinters:
    367                     # printer registered in self.type_pprinters
--> 368                     return self.type_pprinters[cls](obj, self, cycle)
    369                 else:
    370                     # deferred printer

~/.virtualenvs/hagenv2g/lib/python3.5/site-packages/IPython/lib/pretty.py in _ordereddict_pprint(obj, p, cycle)
    829             p.text('...')
    830         elif len(obj):
--> 831             p.pretty(list(obj.items()))
    832 
    833 def _deque_pprint(obj, p, cycle):

~/.virtualenvs/hagenv2g/lib/python3.5/site-packages/expiringdict/__init__.py in items(self)
    111         """ Return a copy of the dictionary's list of (key, value) pairs. """
    112         r = []
--> 113         for key in self:
    114             try:
    115                 r.append((key, self[key]))

RuntimeError: OrderedDict mutated during iteration
kophy commented 7 years ago

In items and values, the loop is like:

for key in self:
    try:
        r.append(self[key])

self[key] may trigger expiration--delete while iterating. Solution is to iterate over the copy of all keys:

for key in list(self.keys()):
    try:
        r.append(self[key])
oz123 commented 7 years ago

@kophy Because this package seemed abandoned, I forked it and modified it. I also added some things to a similar package called ttldict. You can find updated packages in pypi:

https://pypi.python.org/pypi/ttldict/

The code is tested on all Python3 versions.

feel free to star the project in https://github.com/mobilityhouse/ttldict.

And BTW, sorry for posting here so late about the work done in parallel...