omit swarm-agent provisioning on nodes that specify `--cluster-store` and `--cluster-advertise`

[dweomer]

... if the value of --cluster-store matches the value for --swarm-discovery.

Problem

As @vieux spells out in the video linked to in this article and @allingeek was trying to tell me a little over a week ago, swarm agents on clustered nodes are superfluous and cause the system to duplicate efforts with regards to cluster participation.

Proposal

For nodes created with --engine-opt cluster-store ..., --engine-opt cluster-advertise ..., --swarm, and --swarm-discovery ... machine should either

• refuse to create the node while emitting a helpful message as to why, or
• simply not provision the agent container while emitting an informative message as to why

  Caveats

• This mostly overlaps with #2302 but not completely.
• This only applies when the values for the engines --cluster-store URL and the --swarm-discovery URL are equal, unless such is invalid configuration (I haven't thought that part through. Maybe a swarm maintainer could chime in here)

[nathanleclaire]

swarm agents on clustered nodes are superfluous and cause the system to duplicate efforts with regards to cluster participation.

I don't understand the issue. What's the duplicated effort exactly?

libnetwork membership and swarm cluster membership are decoupled today AFAIK, I wasn't aware that you could have both without setting them up individually. In Swarm's case it's by running the agent, in libnetwork's by setting the Docker daemon setting. They could theoretically be bundled, but then that would be out of the scope of what Machine is intended to manage (requiring upstream changes to both). @vieux @madhu maybe you can help explain?

[allingeek]

Libnetwork performs the same node registration step as swarm join. So swarm managers will see non-libnetwork enabled engines that use swarm join and libnetwork engines.

If you use libnetwork you don't need swarm join.

Jeff

On Friday, March 11, 2016, Nathan LeClaire notifications@github.com wrote:

swarm agents on clustered nodes are superfluous and cause the system to duplicate efforts with regards to cluster participation.

I don't understand the issue. What's the duplicated effort exactly?

libnetwork membership and swarm cluster membership are decoupled today AFAIK, I wasn't aware that you could have both without setting them up individually. In Swarm's case it's by running the agent, in libnetwork's by setting the Docker daemon setting. @vieux https://github.com/vieux @madhu https://github.com/madhu maybe you can help explain?

— Reply to this email directly or view it on GitHub https://github.com/docker/machine/issues/3183#issuecomment-195605853.

[nathanleclaire]

How does the master become aware of the non-joined members? It happens automatically just by using the same KV store for manage and for --cluster-store?

If that's the case, comparing the two and skipping the join if it's not needed seems reasonable, although if it's simply implicit behavior we're relying on without a backwards compat commitment from upstream I'm hesitant.

[allingeek]

Yes. Libnetwork does the exact same thing as swarm join with respect to engine registration. Give it a shot.

— Jeff

[nathanleclaire]

Cool thanks for the speedy response @allingeek

[dweomer]

Thinking about the the values for --cluster-store and --swarm-discovery being different: makes sense to allow for such as it might be a way to

• attain greater scale
• trash your overlay networks and start over without trashing your entire cluster
• simply maintain separation of these two KVS for the corresponding parts of your infra (stability?)

What are the other potential use-cases?

[dweomer]

Ah there is a trick, @vieux mentions it in the video after 4:35. The paths in the KVS are different by default for swarm and libnetwork. If you pass --discovery-opt kv.path=docker/docker to swarm then the two systems should be able to share /nodes. I have yet to get this to work as passing a value for --discovery-opt via --swarm-opt isn't working for me.

[danhenke]

@dweomer worked for me like this --swarm-opt discovery-opt="kv.path=docker/nodes" .. Note that path is docker/nodes and not docker/docker as per your comment above.

[dweomer]

@danhenke thanks for confirming that!