a universal redis API for the data replicator

[GlenDC]

When KDS asked to provide a data replication feature which only uses one Redis interface, it was also proposed that this redis interface has to be exposed by an actual redis server. That was never a hard requirement. The only hard requirement is that all commands/controls happen via a single interface, a redis-protocol API, such that the user only has to use a redis client, know the right commands and how to use those, and that it's it. It allows the user to use the data replication feature from any language, as long as a redis client is available. Great.

So far we did solved this quite cumbersome by having a redis server, which both the user and the server use. Making the redis server somehow the controlling piece here, in-between rivine and the user. While it kinda worked, it has several disadvantages:

• we have NO control over what commands there are and how they are used:
  • Right now we use HGET <namespace> <key> to get a key in a namespace, where is the shortID of a transaction. Given this setup there is noway to allow the user to use the longID instead, which might be required due not having the longID yet. Currently that would allow using the rivinec CLI to get a transaction, to than know the shortID...
    • this also forces us currently to use HGET, instead we could for example choose to implement namespaces as a thing that you have to SELECT such that getting data is done using a normal GET <key>;
    • ideally <key> should be able to be both the long and short ID, where the shortID is returned together with the returned should a long ID be used, so that next time the shortID is known as well;
  • we get into hacked APIs within APIs: PUBLISH replication subscribe:food/PUBLISH replication unsubscribe:food, while really what we want would be something like: SUBSCRIBE food/UNSUBSCRIBE food;
    • this is also an issue should we one day want to use something like 0-db instead of redis (which I'm sure we'll want), as that database does not support channels at all, and thus relying on it to play the controller (between rivine and client) is not going to happen, and it never will;
  • having control over what happens when commands get run, would also solve our last missing API command, to allow people to submit/register data using that same redis client. This would be possible as having our own API exposed as a redis-protocol would mean we could allow REGISTER <data> and it would return the longID as key;
• we limit ourselves to what database we can use, as we rely on features that stuff like 0-db does not have (channels);
• we get a really weird flow of Rivine <=> db <-> user, while really that flow should be db <-> Rivine <-> user <->;

On top of these advantages it doesn't make our life easier at all. It doesn't save us time at all.

Therefore I propose that the datastorage module exposes (over TCP) a redis-protocol (see: https://redis.io/topics/protocol) server, which exposes the following API:

SELECT <namespace> // -> void
GET <shortID> // -> (data []byte)
GET <longID> // -> (data []byte [, shortID []byte])
                        // second result param is only returned if available
REGISTER <data> // -> (longID []byte)
SUBSCRIBE <namespace> [startTime|startBlockHeight] // start replicating the given namespace
UNSUBSCRIBE <namespace>

As you can see the API is relatively small, and our codebase won't really change too much (I mean you will have/want to refactor, but you will be able to reuse most of your logic, just better, cleaner and easier structured).

While the redis protocol is easy enough to implement, you don't have to do so. There are libs available that do what we need I think (e.g. https://github.com/phuongdo/go-redisproto), and starting a TCP server in Golang is as easy as eating candy. So the only work you'll have is to define those commands and link them to handlers (hopefully a good help makes this easy as well).

Further you're database will be able to become a lot simpler. For Redis you could still to choose a hashmap if desired, but we wouldn't need the channel stuff there anymore. Details need to be worked out though.

---

An alternative API could also be:

GET <namespace> <shortID> // -> (data []byte)
GET <namespace> <longID> // -> (data []byte [, shortID []byte])
                        // second result param is only returned if available
REGISTER <namespace> <data> // -> (longID []byte)
SUBSCRIBE <namespace> [startTime|startBlockHeight] // start replicating the given namespace
UNSUBSCRIBE <namespace>

I leave it up to you to decide.

[LeeSmet]

So if I get this right:

• When starting the datastore, start a tcp server which listens for messages in redis protocol to controll it.

• Remove controlling functions (SUBSCRIBE/UNSUBSCRIBE) from the database as this now only serves to store data.

• Maybe implement a database using boltDB to remain in line with other modules and we seem to want to expose fetching the data from the redis api in the datastore as well.

Do we attach the tcp listener to the datastore specifically, or do we make a "global" redis api which functions similarly to the current http api and which could eventually be extended to work with all modules?

[GlenDC]

Do we attach the tcp listener to the datastore specifically

this