search

piskvorky / sqlitedict

Persistent dict, backed by sqlite3 and pickle, multithread-safe.
Apache License 2.0
1.17k stars 131 forks source link

Retrieving value of a specific key is slow #75

Closed MiladAlshomary closed 6 years ago

MiladAlshomary commented 6 years ago

I have a sqllite dictionary contains 4.2 million key. Each value is a list of pyspark sparsevector

Trying to retrieve the value of a specific key takes around 3 seconds! any known issue related?

piskvorky commented 6 years ago

@MiladAlshomary how large is that value, in elements and kBs? How long does pickling it take? (sqlitedict uses pickle internally to serialize the values)

MiladAlshomary commented 6 years ago

@piskvorky The values are not big. they are less than 1 kbyte. One thing I was wondering about, if the created sqllite table has already an index over they key column?

ownport commented 6 years ago

@MiladAlshomary it should be according to

MAKE_TABLE = 'CREATE TABLE IF NOT EXISTS "%s" (key TEXT PRIMARY KEY, value BLOB)' % self.tablename
piskvorky commented 6 years ago

Sure, it does. A key-value store without an index on the key wouldn't be very useful :-)

If you send a minimal reproducing example, we'll have a look. Thanks.

MiladAlshomary commented 6 years ago

Hi, it seems like the issue is not reproducible. Something went wrong with the db connection I believe. Now its working properly, retrieving the value of a key takes milli seconds

menshikh-iv commented 6 years ago

@MiladAlshomary aha, OK, thanks for the information.