Turn LucidMQ into a distributed System

bdkiran commented 1 year ago

Utilize raft to make LucidMQ fault tolerant and distributed.

Library: https://github.com/tikv/raft-rs

Here is an example of someone actually using the library: https://www.hyperledger.org/blog/2019/01/11/floating-the-sawtooth-raft-implementing-a-consensus-algorithm-in-rust

gedw99 commented 1 year ago

What properties are you designing for ?

For example in a cluster with a flapping router between the cluster or between the client and server.

too many messaging systems ignore the failure cases and don’t make it clear when guarantees of linearity will break down.

don’t want to be negative just realistic

bdkiran commented 1 year ago

@gedw99 Hi this is a problem that still needs scoping and glad you brought up some questions.

Points to clarify

What do you mean by flapping router? My research has turned in information about a failure mode in networking. I assume you're talking about a design pattern. Is there a resource in mind that I can check out so we're on the same page or an alternate term to search?

Goals of turning LucidMQ into a distributed system

Handle Failure to increase uptime

Guarantees provided by a correct implementation of raft should handle leader failures. Are you saying that raft will not provide consistency across the nodes? It's my understanding that consistency will be upheld since all updates to the commitlog will only be done by a single leader and propagated to the followers in the order that is determined by the leader.

Partition Storage Across Many Nodes

This is a problem that I have questions about. Kafka solves this by topic partitions. Not a fan of this model because:

The abstraction forces the application developers to handle where data should be placed in what partition.
Linearizability is not guaranteed across a whole topic(if partitioning is used) I would like to solve both of these issues in some way but still thinking about what the correct abstraction is and how it will work.

gedw99 commented 1 year ago

I don’t know of any standard term for it.

RAFT is one solution and NATS uses this for it’s distributed broker.

if you want to break free of raft there is CRDT . The rust implementation has been going on for years.

Raft and crdt are very different approaches to a similar problem domain.

you need crdt if:

your app needs to run offline and then sync (with potential changes in the online version), or there is no central authority to determine order of operations . otherwise, you can just use simple states & actions, as long as you make sure they happen in the same order for all clients.

It’s an interesting opportunity perhaps for a message broker because the data lives inside the services and those devices can be anywhere. So it can do p2p. It really changes the ball game… :)

https://github.com/automerge/automerge Is the most well known one . By the famous “ turning the database inside out” Martin Klepman. Worth a read …

https://github.com/rust-crdt/rust-crdt

https://github.com/y-crdt/y-crdt

—-

I am going to also throw into this space SurrealDB . It’s a rust db that is sort of like a Kafka too in that it has live queries. in a message broker you subscribe to data changes. In surreal you do the same via their SQL like query language. It’s pretty neat.

they use kv which under the hood uses raft to been the kv in sync but you can still get stale data because of network partitioning / lag.

Surreal also give you http rest , graphql or a web socket to make their SQL calls.

SurrealDB has a pluggable store . They use tikv .

mig it was me and I am totally biased I would use SurrealDB with a CRDT store. Totally biased but why ?

you get a message broker hat has a SQL dialect. So now there is no code to write . You get a fast data base. Nats and Kafka are both evolving to be a database but lack a query language.

bdkiran commented 3 months ago

Raft implementation here without gRPC dependency. Looks like we can use our own protocol https://github.com/ditsing/ruaft

integrandio / lucidmq