Discovery and clustering

[pires]

For clustering to be achieved automatically, a discovery mechanism needs to be implemented. Multicast-based discovery mechanisms are always prefered because it requires zero-configuration, but since most, if not all cloud providers block multicast traffic I propose the adoption of a gossip protocol like the one implemented by Serf.

By doing so, there's only one requirement in order to form a cluster, at least one node must exist and it must be reachable by any new nodes. Every new node (member) does a full state sync with the existing member and begins gossiping its existence to the cluster.

Gossip allows to keep an eventually-consistent view of the cluster at any given node.

Thoughts?

[manishrjain]

Gossip protocol is definitely the right way to go. As per our discussion, a new node would be provided IP address of an existing node, so it could jump start the protocol.

Few questions/requirements:

• What happens when the entire cluster is brought up together, and we don't have any existing IP address? I suppose rolling start would be required in such a scenario?
• The implementation of Gossip should allow us to pass meta information, like runtime and load statistics about the server, so we can make better judgements about shard reassignment.
• Instead of taking only 1 IP address, take a list so multiple servers could be checked.

We'll also have to determine what our communication protocol would be. I looked at grpc.io, but it's focused around protocol buffers. We're using Flatbuffers, which aren't supported. So, I see no big benefit in going that direction. We should stick to the standard net/rpc package provided by Go. Byte array based request and response should be sufficient, because Flatbuffers.

[manishrjain]

Btw, server failure detection isn't really required from Gossip protocol. MultiRaft is going to tackle that part.

[pires]

What happens when the entire cluster is brought up together, and we don't have any existing IP address? I suppose rolling start would be required in such a scenario?

No. Each new node will always connect to at least an existing node. And after connection is established, the topology will be shared periodically. So, let's say node A is the first in the cluster and nodes B and C connect simultaneously. A will tell B and C there are no more nodes connected, but after a configurable amount of time, A will tell B that C exists and C that B exists. B and C will most probably connect to each other and share topologies themselves. If Z has connected to B, B will infect everyone it's connected to, and so on.

The implementation of Gossip should allow us to pass meta information, like runtime and load statistics about the server, so we can make better judgements about shard reassignment.

As I said in the channel, this is definitely one advantage of going with serf. It provides a cluster messaging facility where you can do this kind of stuff. It also includes network tomography, so one node could eventually know the nearest-K nodes.

We should stick to the standard net/rpc package provided by Go. Byte array based request and response should be sufficient, because Flatbuffers.

I'd say we should stick with whatever allows for any payload format. Narrowing it to net/rcp may be too much. Also, By default [gRPC] uses protocol buffers, Google’s mature open source mechanism for serializing structured data (although it can be used with other data formats such as JSON).

Btw, server failure detection isn't really required from Gossip protocol. MultiRaft is going to tackle that part.

Node failure is important in terms of cluster membership management. We're not talking about sharding/replication and its consensus here.

[manishrjain]

No. Each new node will always connect to at least an existing node.

I think you misunderstood my question. My question is specifically regarding the case when you bring up the cluster the first time, so in your example, bringing up A, B, and C together. In that case, there's no previously existing node; so does that mean, we have to do rolling start?

Regarding grpc, as I mentioned above, it uses protocol buffers, but we use Flatbuffers. I don't want to mix up the two, unless absolutely needed. Which is why I propose using net/rpc, instead.

Node failure is important in terms of cluster membership management.

I think there's some sort of overlap here between Raft and Gossip, that we should be careful of, design wise. Probably more of a note for me at this point, than for you.

[pires]

bringing up A, B, and C together. In that case, there's no previously existing node; so does that mean, we have to do rolling start?

Here's what I said:

So, let's say node A is the first in the cluster and nodes B and C connect simultaneously.

So A is the existing node, while only B and C connect together.

Regarding grpc, as I mentioned above, it uses protocol buffers, but we use Flatbuffers. I don't want to mix up the two, unless absolutely needed. Which is why I propose using net/rpc, instead.

I don't support going with gRPC as well. Neither choosing net/rpc. Tbh, I want to adopt an existing solution and not implement this from scratch.

[manishrjain]

What new technologies would existing solution bring in? How hard would it be to write our own solution with net/rpc and flatbuffers?

[pires]

What new technologies would existing solution bring in?

Not sure I follow the question, but basically something like serf would allow for:

• Gossip discovery
• Cluster membership management
• Cluster messaging
• Network tomography

How hard would it be to write our own solution with net/rpc and flatbuffers

The discovery process should be pretty doable. The membership management and messaging would be tricky. The network tomography should be hard.

[manishrjain]

I don't support going with gRPC as well. Neither choosing net/rpc.

What I mean is, if you don't use either of these for communication, what do you use? That's what I mean when I ask about the new technologies that your solution would bring in. I'm not asking about the functionality the library would bring in -- that's clear a separate discussion.

[pires]

As I said above

I'd say we should stick with whatever allows for any payload format. Narrowing it to net/rcp may be too much.

I just meant that I don't feel like narrowing the messaging transport in a decision right now. Deciding to go with net/rpc may be enough for anyone not picking it just because they we're thinking about something else. serf leverages on memberlist, which uses pure TCP and UDP for messaging exchange.

[manishrjain]

memberlist seems closest to the minimum we need to implement Gossip. But, it lacks the ability to send various stats along with the Gossip pings. To send stats / custom messages, we'd have to call SendTo, on top of Gossip periodically; which will increase the amount of communication required unnecessarily.

I think it seems like writing our own bare bones Gossip impl would give us the right customizability, without any of the unneeded features at this point at the extra cost they might come in.

[jcmartins]

Why not Raft Protocol ? https://github.com/hashicorp/raft Like Influxdb does.

[pires]

Raft is only meant for consensus, not discovery or cluster management. However, serf and memberlist use that Raft implementation. On Jan 19, 2016 20:13, "Joao Martins" notifications@github.com wrote:

Why not Raft Protocol ? https://github.com/hashicorp/raft Like Influxdb does.

— Reply to this email directly or view it on GitHub https://github.com/dgraph-io/dgraph/issues/12#issuecomment-172972578.

[manishrjain]

We're using RAFT only. When a new node wants to join, it can talk to any member of the cluster. That member can convey the master for the entire cluster. The new node then talks to the master, and joins the cluster. This is sufficient for now.