arbeitsgruppe-digitale-altnordistik / Sammlung-Toole

A new look on Handrit.is data
https://arbeitsgruppe-digitale-altnordistik.github.io/Sammlung-Toole/
MIT License
0 stars 0 forks source link

Use a database for groups instead of pickling them (possibly TinyDB) #103

Closed BalduinLandolt closed 2 years ago

BalduinLandolt commented 2 years ago

The current implementation of groups as dataclasses works (though we might consider pydantic, see #102 ); but the persistence as pickles is not really good. This should be re-written to use an actual database.

SQLite seems to work OK for the manuscripts etc., but I'm not sure a relational database is really the right structure for this kind of one-to-many relationship we have in the groups. Of course it works, but it seems clumsy to me, compared to a NoSQL database.
In terms of NoSQL databases, TinyDB seems to be the equivalent of SQLite in the python world: It also works of a file or in-memory, and it doesn't need a database-server but can be used directly in code.
Like "real" NoSQL databases (MongoDB, etc.), it stores documents as JSON-like structures, so a one-to-many relationship can just be a Document containing an Array (i.e. in this case a Group having a list of IDs).

@kraus-s what do you think, should we give it a shot?

And if it works great, we should benchmark our Person and Text junction tables in SQLite against this one too...

kraus-s commented 2 years ago

I see your point and I agree, that TinyDB could provide an elegant solution. I would like to keep it simple for now and prefer to first try and implement it in SQLite. This shouldn't even be to hard: We can even store a JSON like array in SQLite, i.e. like this: ID | GroupName | GroupContents 0 | A | {PPL: 'ArnAlf001", "ArnAlf002"}

This is perfectly fine from an SQL point of view, because it does not constitute a many-to-many relationship. It is a single, stored object. This way we could also later on mix MSs, Txts and Ppl in one group and display their relationships as a network graph ;)

BalduinLandolt commented 2 years ago

I was hoping to get rid of SQL... ;)
But you're right. probably we should keep it simple for now.

But as I stated in my comment on #105 I'd probably like to keep this a separate database, as it will be our only writeable DB.