search

piskvorky / sqlitedict

Persistent dict, backed by sqlite3 and pickle, multithread-safe.
Apache License 2.0
1.16k stars 130 forks source link

UnicodeDecodeError #146

Closed dineshbvadhia closed 2 years ago

dineshbvadhia commented 2 years ago

I've been using sqlitedict for years, first with py2.7 and now py3.x, without any problems. However, using sqlitedict within async code the error below occurs when opening. Serialization with the default pickle causes the same error.

with SqliteDict(filename, flag="r", journal_mode="OFF", encode=msgpack_serialize, decode=msgpack_deserialize) as f:

<class 'UnicodeDecodeError'> 'utf-8' codec can't decode byte 0x97 in position 1916: invalid start byte

To confirm it is a sqlitedict problem, the data was saved and opened as json dump and json load and it works fine.

Any thoughts?

piskvorky commented 2 years ago

Hi, do you have some repeatable minimal example?

dineshbvadhia commented 2 years ago

The minimal example works with and without the async code. The above error persists in the larger application.

Is it possible to knock-out the multi-threaded support to see if this narrows the problem area ?

piskvorky commented 2 years ago

A minimal failing example will be a great start, so we can replicate and confirm the issue (and later, its resolution).

dineshbvadhia commented 2 years ago

The minimal example is not failing :(

piskvorky commented 2 years ago

Then it's not a minimal failing example :)

Until we have a reproducible error, with a proper full stack trace, this will be hard to crack.

dineshbvadhia commented 2 years ago

SqliteDict doesn't work with the async framework.

A single-threaded kv store built on top of sqlite3 works with the async framework.

The async framework folks are looking into it.

Don't get your knickers in the twist. When there is something to report, it will be.