piskvorky / sqlitedict

Persistent dict, backed by sqlite3 and pickle, multithread-safe.
Apache License 2.0
1.17k stars 131 forks source link

splitting by key into multiple sqlite files #91

Closed notestaff closed 5 years ago

notestaff commented 5 years ago

To improve concurrent access, it would help to split the sqlite database file into multiple files, with the key hash determining which file stores the full key-value mapping for the key. Then concurrent writes would be ok as long as the keys are assigned to different sqlite database files. You'd initialize sqlitedict not with a file but with a directory, and under that directory would create the sqlite db files.

piskvorky commented 5 years ago

That sounds useful but out of scope for sqlitedict, sorry. Perhaps you can implement that user-side?