As a plugin author, I'd like access to a persistent datastore

[binary1230]

Hey,

CC @migetman9 @kitsuta

I wanted to continue implementing some functionality into Slackbot that we (Magfest) are currently using in Hubot, and wanted to discuss here the best ways to do it.

So questions are:

1) Should the concept of a persistent data store be in the core bot, or should it be implemented via an external (non-core) plugin?

I do like the idea that there's a concept of a persistent datastore built into slackbot, and @lins05 if you felt the same way, I'd like to pursue its implementation.

There's a couple of ways we could do this:

a) Vanilla bot install has a 'brain' with persistence.

This means that the core bot has something like brain.set() brain.get(). When a plugin calls these functions, it talks to the underlying DB.

For writing to the disk, it could automatically flushes the value out to disk when you call brain.set(). Alternatively we could save at regular intervals.

This is my preferred implementation option.

b) Vanilla bot has a 'brain' but no persistence, but an external plugin or package could override the brain interface to layer a DB on top of any get()/set() calls.

If you didn't configure another plugin/package to act as the brain, then key/value pairs would be lost once you stopped the bot.

c) Entire concept of persistent data store belongs solely in external plugins.

This is fine, though, I really like the idea that the average developer spinning up slackbot has the ability to have persistent storage that "just works" out of the box.

2) What would we use for a DB?

Ideally, this would be something that is really simple, lightweight, transparent to the plugins, and doesn't require an external server and can just run off of a local file. It's less a database than a simple key/value store.

@migetman9 I know originally you were looking at MongoDB but it seems to require an external server, which means more setup for the average user.

Hubot uses Redis, but the python package for running an embedded Redis seems to be slightly jank for installing (maybe I'm just doing something wrong).

I found pickleDB https://pythonhosted.org/pickleDB/ which looks very simple and promising. I think that might be a decent solution.

---

I'm thinking a typical use case for this would be remembering small info from the bot where performance isn't really going to be much of a concern, like access lists, remembering small things for plugins, etc. Therefore, my default inclination would be to do something really simple.

[jtatum]

The reason hubot uses redis is because their main deployment strategy (Heroku and similar) doesn't offer persistent local storage, so local DBs like sqlite and similar would get lost every day. sqlite would be my vote, though, if we wanted to do something simple with local storage. If we did pickle, it wouldn't be too hard to roll our own solution around it. Redis makes sense to me for something a little more powerful that works in a wider variety of deployment scenarios.

[lins05]

Should the concept of a persistent data store be in the core bot, or should it be implemented via an external (non-core) plugin?

I don't think it's a good idea to provide a persistent storage module/helper in the slackbot core library:

• It doesn't conceptually couple with the core logic of slackbot
• It may not be really needed by more than 50% of the users (IMHO)
• Besides, I don't see an indisputable choice for the store: someone may like redis, some may want memached, some may want mysql/pgsql.

So my suggestion is to provide them as external plugins/extensions.

[osteenbergen]

For my bot (repo) I decided to do two things to solve persistence, first I use a borg/singleton class so every plugin can use it and then store a SQLite object in the singleton.

class Borg:
    _shared_state = {}
    def __init__(self):
        self.__dict__ = self._shared_state

Now every plugin/class can do

b = Borg()
b.session_storage = {}
b.persistence_storage = sqlite3.connect("/tmp/database.db", check_same_thread=False)