andrewjstone
commented
7 years ago The intention of that section was to allow plugging in different protocol implementations, not to allow different APIs. We always intended the API to be a disjoint set of what zookeeper provides with some new primitives baked in, and other things intentionally left out. e.g instead of allowing ephemeral, auto-incrementing nodes which are often used for leader election, we'd provide a leader election primitive instead. The goal was to provide a ready-made, opinionated system to allow users to safely coordinate their systems, not to build a toolkit, as the toolkit approach makes it harder to use and debug the different combinations of setups in production.
In practice, isolating even the protocol hasn't been perfect. While most of the VR specific code lives in /src/vr, there are artifacts of the fact that Haret uses VR littered throughout the codebase For instance the namespace manager knows about VrCtxs and the 3 different start modes (startup, recovery, reconfiguration) of replicas. Since implementing a consensus system like VR on top of a lightweight process architecture requires a management layer like the namespace manager (using gossip in this case), in order to start replicas on different nodes and learn of new consensus groups after partition, it was easy to fall into this trap. Ideally this management layer would be agnostic of the consensus protocol as well, but I haven't spent the time to go back and fix it. It isn't high on the priority list right now, although it may help provide cleaner, more structured code.
I'm actually in the middle of a major refactoring of the FSMs that I hope to open a PR for in the next week or two. It's possible that all this code could live in it's own VRR library, but I'm not sure how useful it would be outside of Haret. I'm also hesitant to independently version it at this early state, when I'm the primary developer, as it just adds another layer of management for me. The reality of adding another consensus protocol at this juncture is very remote, so taking the time right now to do this is not a priority.
As far as decomposition of the system, a bunch of things are already in their own libraries. Haret relies on rabble for the cluster system and lightweight processes and vertree for the trie based backend.
It is possible to also abstract out the front end API, but it is harder, as the API is heavily tied to the capabilities of the backend. It is also useless in and of itself.
It appears that zetcd is an independent proxy process that sits in front of etcd. That doesn't require splitting up the code at all, but it does require features, such as subscriptions, that aren't yet built into Haret. It also will either require emulating other non-native features such as ephemeral nodes, or not implementing them altogether. That all seems doable, but again isn't really a priority for me right now. My chief goal is building a correct and stable system. After some stability it will be much more actionable to talk about extension and different front end APIs.
In summary, I'm not fully opposed to this idea, but feel it is a bit of a distraction at this early time. If however you see specific parts of the code that you feel are not properly abstracted and should be split out into their own libraries, I am definitely willing to consider that.
erickt
jrgarcia
In the why.md document, there is a discussion about how haret was designed to isolate off the protocol from the client-facing and data-storage parts of the system. Would there be any interest in formalizing this into multiple libraries? I'm personally interested in exploring a Zookeeper client wire-compatible frontend a la zetcd, so having a looser coupling between the subsystems would make this a bit easier to do.