Consider having etcd cluster not managed and controlled by glusterd

[prashanthpai]

Currently, the life-cycle of etcd idaemons are strictly coupled and controlled by glusterd2 process which makes the entire setup very fragile IMHO. Is it a good idea for glusterd2 to just use existing etcd cluster which is externally managed ?

[atinmu]

@prashanthpai - I did feel it at the time of writing it. In case the setup goes into a faulty state it might become very difficult to recover to sane state. We already have a roadmap to have external etcd integration support but default is considered to be gd2 controlling it. The only drawback I see here the ease of installation and forming the cluster with ease. Gluster's one of the core strength has been the ease in install and forming the cluster. Although I havent thought deep about it, I do have a feeling that you'd definitely need to perform few more additional steps here to setup a cluster. Can you sketch out a design on this and then we discuss this further?

[prashanthpai]

Sure. I still have to look into pros and cons of each approach after testing out the current implementation thoroughly especially around peer attach and detach functionalities.

[prashanthpai]

I went through the etcd official documentation on clustering architectures. AFAICT, this is what we need:

For all our development process:

For large production clusters:

workers == etcd proxies.

Some notes/thoughts:

• etcd process (spawn or restart) lifecycle shouldn't be controlled by glusterd2. glusterd2 should only be etcd clients consuming the key/value store.
• A node running etcd server (non-proxy) that is participating in RAFT consensus mechanism can have glusterd instance co-located on the same node.
• glusterd will always talk to the local etcd proxy. The proxy is brought up in readwrite mode. The proxy will keep track of all peers participating in consensus cluster and glusterd2 doesn't have to.
• If one the real etcd servers fail, the proxy can be promoted to 'server' to participate in consensus.

@kshlm @atinmu What do you guys think ?

It'll be good if we can discuss about how each glusterd2 operation (peer add or remove, volume create etc) would look like w.r.t etcd interaction in an externally managed etcd cluster.

[prashanthpai]

Regardless of whether the deployment has etcd server embedded or is external, one thing is certain - not all gluster cluster nodes will be etcd cluster members. A small odd number will be part of consensus, the rest will only be etcd clients or proxies.

[prashanthpai]

Closing this. Elasticetcd supports external etcd instance.