Write a Reliable Key/Value Datastore.

Steve-V / tgg-BotSteve

Issue tracking and source hosting for the TGG BotSteve module

11 stars 10 forks source link

Write a Reliable Key/Value Datastore. #79

Closed AstraLuma closed 12 years ago

AstraLuma commented 13 years ago

So, I'm thinking about the pickle datastore. It's great in its simplicity, but it has a significant downsides:

If the bot crashes, the datastore might go with it
In order to administer, you have to take down the bot
In case of corruption, you loose everything in that module
As the dataset grows, we loose load/save performance

Things I like about it:

Simple
Flexible
Automatic

I'm wondering if there's a better way to do it, like using a nonrelational database. CouchDB and MongoDB seem like a good place to start.

AstraLuma commented 13 years ago

Both CouchDB and MongoDB act the same way, each document being key/value object, where values can be rich JSON-like data.

AstraLuma commented 13 years ago

Started a framework for pluggable backends in astronouth7303/tgg-BotSteve/realdb. There's issues in making sure the data gets saved (something about pickle being unavailable in __del__), but I've got the basics in there.

I also documented the storage interface/framework in that branch under STORAGE.md.

AstraLuma commented 13 years ago

Alternatively, there's storing JSON data as flat files. Each module gets a directory. The key is the file name (with some encoding), and the file contents is JSON data.

Pros:

Simple
External, live administration
No data loss from crashing

Cons:

Slow (every call makes at least one file system operation, plus JSON overhead)
Lots of files

Other:

Administratable using normal Unix tools (human readable text)

Full details are in astronouth7303@aaa63f4d06132af0fef27182527e0a2b69b769f5

AstraLuma commented 13 years ago

I've been thinking about this, and I think the Flat JSON store might have potential, if you add lots of caching.

AstraLuma commented 12 years ago

I fixed up the pickle store so it should work in all cases where the interpreter exits cleanly. In cases where Python crashes and burns, it will still loose data. Not sure if there's anything we can do about that.

A few more design issues have been clarified. These are really edge cases:

Indirect changes are not supported; any changes made without setting an item might be lost.
A module using storage items in its atexit callback may run into problems.

If @Steve-V is ok with the design, API, and restrictions (as benevolent dictator), I think the realdb code might be ready? I still feel like it needs more beating, though. There haven't been nearly enough bugs found.