jepsen

[glycerine]

Kyle Kingsbury/Aphyr did Jepsen testing of zookeeper. That could probably be a good baseline for testing the zetcd functionality.

https://aphyr.com/posts/291-jepsen-zookeeper

https://github.com/jepsen-io/jepsen/blob/master/zookeeper/src/jepsen/zookeeper.clj

[xiang90]

@glycerine Ah. Interesting. But I think jespen will try to kill zk process. We need to modify it to kill both etcd and zk proxy. right?

[glycerine]

I haven't played with jepsen myself, so I'm not really sure. It would make sense though. There is certainly network partitioning simulated.

But it does point out an assumption I was making that seems worth discussing.

I assumed that zetcd just loaded etcd as a library; thus my mistake when first testing. It's pretty inconvenient to run 2 process when you only need one anyway. What's the thinking behind doing a separate proxy process? Seems simpler to have zetcd embed etcd, or provide a flag that makes etcd present a zookeeper port; either way only one type of process need run in triplicate.

[heyitsanthony]

@glycerine there are several reasons for having a separate process:

• load balancing -- tying zetcd with the etcd process means there's a forced 1:1 relationship between etcd and zetcd instances. This precludes scaling up the number of zetcd instances to multiplex more clients on an etcd backend; it forces more core etcd instances which isn't the way to scale etcd. The proxy is no longer stateless.

• multi-tenancy -- decoupling the zetcd proxy from etcd makes it possible to cleanly support multiple zk instances at once on a single backend or spin-up/teardown instances at will without disrupting the core etcd cluster. (there's no custom prefixing in zetcd to do this yet but it won't be terribly difficult to add; it'll probably be supported through the planned etcd naming/prefixing proxy, however).

• fault isolation -- If the etcd cluster is serving both zk and etcd API's, then a bug in the zetcd server (resource leaks, crashes, etc) will affect the backend too. It's more reliable (albeit, more expensive) to separate components than bundle them into a single process. If a zetcd proxy crashes, it's less disruptive to the cluster to spin up a fresh one than restarting a crashed etcd member.

Simpler is not necessarily better in this case. There may be support for an embedded version in the future (this will depend on a pass-through client; there is an open issue on the etcd tracker for that), but for the time being it's not a priority for the reasons outlined above.

Having written an entire dissertation on model checking (you may have noticed zetcd supports a rudimentary form of cross-checking, which is a very powerful validation technique), I am very critical of jepsen's exaggerated claims and technology in general. If zetcd is going to have simulated network partition tests, it'll probably be done through the etcd functional-tester instead of going down the clojure rabbit hole.

[glycerine]

The fault isolation, e.g. due to the difference in maturity of the two projects, makes sense to me.

I think you capture the spirit of the idea (leverage existing zookeeper tests/clients) in #14, so I'll close this out.