search

piskvorky / sqlitedict

Persistent dict, backed by sqlite3 and pickle, multithread-safe.
Apache License 2.0
1.17k stars 131 forks source link

Maximum recursion depth exceeded while trying to store nested dictionaries #118

Open kuriringohankamehameha opened 4 years ago

kuriringohankamehameha commented 4 years ago

It seems that sqlitedict fails to pickle relatively nested dictionaries successfully. Sometimes, I get this error when trying to store a dictionary (around 5 MB) with a nesting level of around 3-4 in Python3.6.

File "/home/user/.local/lib/python3.6/site-packages/sqlitedict.py", line 253, in __setitem__
    self.conn.execute(ADD_ITEM, (key, self.encode(value)))
  File "/home/user/.local/lib/python3.6/site-packages/sqlitedict.py", line 100, in encode
    return sqlite3.Binary(dumps(obj, protocol=PICKLE_PROTOCOL))
RecursionError: maximum recursion depth exceeded while calling a Python object

Is it just me, or can anyone else verify that this happens during pickling? Any workarounds for this issue until then?

Thanks

piskvorky commented 4 years ago

There might be some recursion limitation in pickle. It's suspicious that would happen at a nesting level of 3-4 levels though; I'd expect any such limits to be orders of magnitude deeper.

If you post a minimal reproducible example, and a full stack trace, we'll have a look.