My use case is that I'd like to more easily import and explore JSON documents returned from APIs. Obviously, the automatic type detection and table creation provided by sqlite-utils (SU) adds considerable productivity to that effort.
One of the difficulties I've encountered is that SU will serialize dicts/lists to JSON but doesn't deserialize it when retrieved. It turns out that with a bit of modification, it would not be hard to support this use case. This is what I came up with:
import json
import sqlite3
from sqlite_utils import db
db.COLUMN_TYPE_MAPPING['JSON'] = 'JSONB'
class Database(db.Database):
def __init__(self, fpath, *args, **kwargs):
conn = sqlite3.connect(fpath, detect_types=sqlite3.PARSE_DECLTYPES)
sqlite3.register_converter('jsonb', self.json_loads)
super().__init__(conn, *args, **kwargs)
# Keep this on the class so it can be easily customized in a subclass
def json_loads(self, val: bytes):
return json.loads(val)
db = Database(':memory:')
db['users'].insert(
{'id': 1, 'name': 'John Doe', 'preferences': {'theme': 'dark', 'language': 'en'}},
columns={'preferences': 'JSON'},
replace=True,
)
row = db['users'].get(1)
assert isinstance(row['preferences'], dict), row
This could all be made mostly automatic if SU:
Column type detection used 'JSON' instead of 'TEXT' when detecting Python objects that will be jsonified
sqlite3 is setup to handle JSON/JSONB conversion
Considerations:
I'd like to see this made the default but until the next major version bump, could be hidden behind Database(..., jsonb_columns=True)
Whether or not JSON or JSONB columns are used would depend on SQLite version.
My use case is that I'd like to more easily import and explore JSON documents returned from APIs. Obviously, the automatic type detection and table creation provided by sqlite-utils (SU) adds considerable productivity to that effort.
One of the difficulties I've encountered is that SU will serialize dicts/lists to JSON but doesn't deserialize it when retrieved. It turns out that with a bit of modification, it would not be hard to support this use case. This is what I came up with:
This could all be made mostly automatic if SU:
sqlite3
is setup to handle JSON/JSONB conversionConsiderations:
Database(..., jsonb_columns=True)