Allow integration with Serf instead of Zmq

[rothgar]

Serf is a service discovery and orchestration tool built on the gossip protocol for p2p communication that is distributed and fault tolerant. It would allow for more robust systems that are extremely scaleable. It also could help with message broadcasting and alerting. I have not used it very much but it seems like it could be a better solution for huge server deployments that don't have a single point of failure. Just wanted to open this issue to start a conversation about it's possible uses with Salt.

[techhat]

Serf looks like an interesting project, and I'm definitely going to have to dive even deeper into the docs. But from what I can tell, this is not architecturally a good fit. However, I think we would be open to examining use cases that demonstrate situations where Serf is more appropriate than 0MQ. Do you have any in particular?

[techhat]

If you guys can make it fit, I'm all for it. I apologize if I sounded too dismissive of it earlier.

You've probably seen salt-ssh by now, which replaces 0MQ with SSH, but not in a way that would make sense for Serf. However, it did lay the foundation for modularizing the transport layer, which is good for Serf. If you guys want to take a look at the (very pre-alpha) RAET stuff that's been getting checked into Salt lately, it will likely give you additional insights on how to add a Serf layer.

[rothgar]

I am not a dev at all and am just starting with salt (coming from a Puppet and Ansible). Serf seemed like it would be a better fit for Salt than the other two products which is why I opened the ticket here. No real use case. I just like the idea of a fully distributed CM, orchestration, and monitoring system.

[whiteinge]

After playing around a bit with Serf I think a lightweight wrapper such as the serf-master project would be a very natural way to use Salt modules via Salt's masterless salt-call interface. It would take a little glue, a new Serf execution module, and testing but I think it would work out -- especially as Salt internals move more and more to event-centric operations. Serf is a very cool project and has a few features Salt does not get with ZeroMQ (distributed network, no single point of failure, automatic event replays).

@armon, OOC does the SWIM protocol support many clients connecting to only a single host? I realize that would sacrifice the distributed nature of the protocol, kill efficiency, and introduce a single point of failure. It looks to me like the answer is no -- and the workflow I mentioned above does not require that, I ask because I'm still thinking over all the contact points Salt and Serf could integrate.

[armon]

@whiteinge The internal gossip protocol doesn't really "connect" to any node if that makes sense. Messages are generally passed via randomized gossip, so usually just fire-and-forget UDP packets. The TCP anti-entropy also uses a random node. So there is no way to force communicating through a single node. You can have every node do an initial join with a well known hosts, which is fairly common. Load is usually not an issue with that, since thousands of nodes aren't booting at once.

[armon]

This maybe relevant for Salt integration, but Serf 0.5 (releasing soon) introduces a request / response mechanism called "user queries". More here: https://github.com/hashicorp/serf/issues/139. I think this is a big step from simple events, that definitely makes it more powerful and simpler to integrate.

[whiteinge]

@armon gotcha. Thanks for the explanation! And thanks for the link to the next version -- that does indeed look useful.

@thedrow, we don't need a centralized architecture for using Serf with the salt-call workflow I mentioned above. Since Salt in master->minions mode currently works with a centralized architecture I just wanted to know if it was possible. :-)

[whiteinge]

@thedrow Sorry for the slow reply. I wouldn't expect the the communication core of Salt to be replaced with Serf since that kind of central topology is useful / desired for many use-cases. That said, Salt is already flexible enough in it's current state to work with Serf in a manner similar to the "serf-master" project you linked above with minimal glue -- you don't need the master->minion(s) setup in order to use most of Salt. And there is definitely also a use-case to not have that central topology, like you're interested in. Salt can support both. :-)

[rothgar]

@thedrow I can totally understand the desire/need for a centralized architecture in some cases. But I think many implementations could benefit from distributed communication. Questioning the centralized topology was the exact reason why I wanted to start this conversation about Serf and Salt.

[whiteinge]

Sorry for the slow reply here. I've been swamped lately but Serf integration has still been on my mind -- it came up in a meat-space conversation just yesterday, in fact.

@thedrow Both centralized and decentralized topologies have good use-cases. Salt, as a core philosophy, is flexible. Salt works now in both centralized and decentralized modes. Adding Serf integration would be a great usability boost for using the decentralized mode at-scale. It won't have the blazing immediacy of ZeroMQ, but not all infrastructures care about speed. Serf integration would be great to have in Salt core.

[rio]

Why would Serf be a better alternative for ZeroMQ + ZRE Protocol? I know this issue has been started a while back so I don't know what is implemented and what not. We've just been looking at Salt comming from Puppet and are switching our internal service communication to the decentralized Pyre (ZeroMQ + ZRE) so I'm just curious what the thoughts behind it all is.

[rothgar]

@Rio I'm not familiar with ZeroMQ +ZRE or how it differs from the gossip protocol used by Serf. I only recommended Serf for its decentralized nature of not needing a master (which pyre also provides). Someone else may know the pros/cons of implementations of different epidemic protocols. It sounds like Salt would hopefully be flexible with different message systems in the future.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.