Decide and implement a persistent storage for state

[shazow]

Options to consider (sorted vaguely in order of current preference):

• flat file (possibly multiple append-only files?)
• BoltDB (native golang)
• badgerdb (native golang)
• leveldb
• sqlite

Things we'd want to store:

• message history (log)
• registered identities (against pubkey)
• whitelisted pubkeys
• admin pubkeys
• identity settings (theme? last message seen? etc)
• what else?

Note: Need to make sure persistence doesn't collide if multiple instances are run on multiple ports.

[scrothers]

I personally like the idea of sqlite, but, I could be bias. I've wanted to find a project to learn GoLang with, and this one seems like a no brainer, small but complex and such. I have experience from other SQL providers, so obviously I'm quick to choose sqlite.

However my inner developer says we need to create interfaces and adapters for pluggable storage systems.

So how about an interface "storage" which relies on an adapters directory for storage systems that can be picked from?

This also naturally leads to a configuration file, which I believe should be simple and uncomplex. Maybe XML (though that seems overly complex for the job, but simple enough so that it can be read/understood by anybody) or standard ini style parsing.

The config file "system" should have some type of RFS, for example, if we choose XML, we need to discuss how the sections will lay out in the file. If we choose an INI based mechanism, we need to understand how the ini file sections will lay out.

So, I would say a pre-req to this ticket, is the creation of some type of configuration file and parser before this can be implemented.

[shazow]

sqlite is my least favorite choice right now as relational schema-enforced is not beneficial for the problem at hand and adds a fair bit of code complexity to setup, maintain, and query.

I don't believe there is a need for a config file just yet, as long as the command-line flags remain simple enough.

I can confidently say that we are definitely not touching XML for anything. :)

[scrothers]

What is your opinion on interfacing the data storage layer then? I would love to attempt to tackle this :)

[wpovell]

@scrothers Can you explain what that is? I've never heard of it and google hasn't been too enlightening.

[scrothers]

http://en.wikipedia.org/wiki/Adapter_pattern

In this case, I believe it would look something like this (excuse my Go noobishness):

Class "storage" has several functions, which would be: whitelistadd, whitelistdel, opadd, opdel, etc.

This storage class is the "interface", but each function simply calls into the adapters, for example, one such adapter might be "storage_flatfile", which appends and reorders text files and exposes common functions to the interface, like "put, get, delete" or similar.

That way you can create more adapters later, say a "storage_dynamodb" and a "storage_sqlite" which expose the same "put, get, delete" functions that the interface can consume as a unified API for the rest of the application.

But that's it essentially, again, not a Go expert, so how it may be written or used in Go is a mystery to me.

[wpovell]

Oh, nice! This would just be a combination of all the aforementioned methods, leaving the decision to the user?

[scrothers]

That's the idea :) but with less rigid structure and with less inter-dependancy on a specific adapter. The hip word I believe is "de-coupling".

[shazow]

Decoupling makes sense when you have something to decouple, it's still too early to even know what we want to store or what we'll do with it. Further, depends which storage backend we choose will dictate how simple/complex the usage of it is and how we implement the feature.

You're welcome to tackle and propose an interface, but I suspect it's going to change a lot before it's put into use. :/

[jazzychad]

my gut says flat-file is gonna be the preferred way to start - we already have configs/keys loading from files using commandline flags. plus this doesn't add any more dependencies for ppl to install their own ssh-chat on a server somewhere. simple and effective. but, after things settle down and are more stable, then creating an adapter layer would be the way to go, methinks.. then we can build adapters for other storage drivers

[kevinvangelder]

:+1: for flat file, not that I know anything Go related.

[chadtummy]

Of the suggested options, I'm inclined to think either flatfile or leveldb would be the best options.

My understanding is that BoltDB is based on LMDB, which is rather slow at nonsequential writes. If this is the case, it does not strike me as the best option for a data storage format that will eventually be used for things like username registrations (because returning to the username to add more SSH keys to access ssh-chat from multiple computers might be important), for instance.

SQLite is great and has its uses, but it seems a bit hefty for this, and there are some legal issues in places that do not recognize a right to reliquish copyright.

[Juliaria08]

I think that it could be implemented in configuration so all storage methods are available (maybe letting the switching of the options at compile-time?)