Inbuilt incompatibility with port mapping (such as in Docker)

[therealbill]

The decision to make the cluster bus a static "base port +10000" makes it impossible to use in a default Docker setup. It is better that this is configurable, and even better would discoverable.

For discoverability it would be reasonable enough to add the cluster bus port to the hello message and have the initial handshake after cluster meet be to connect, issue hello, then use the port returned to connect for cluster administration. This isn't very dissimilar to the sentinel+client method of Redis. It isn't leaking anything a potential invader can't already figure out so there is no leakage. Ideally this would be how to introduce a member to a cluster.

However, that alone does not solve the underlying problem. We need the equivalent of sentinel announce-port, and it needs to be a run-time configurable. With those additions deploying a Disque cluster would be pretty simple to do on a Docker or Docker swarm. As it stands, Disque has the same problems you run into trying to run Redis master/slave with Sentinel, but the additional one of making the assumption/insistence that the cluster communicates on base+10000. Since you solved in for Sentinel, I know you can solve it here fairly easily. ;)

Summary:

• Yay if made discoverable!
• Make cluster bus port configurable rather than calculated

Cheers, Bill

[justincase]

Some previous references on this topic.

When this will be fixed in Redis Cluster I'll also back port the fix in Disque. The fix would be to make Disque instances able to report an IP/port pair different than the one the nodes will sense via automatic detection using the getpeeraddr system call.

• http://stackoverflow.com/questions/30904033/unable-to-get-docker-containers-running-disque-to-establish-a-cluster

• 85

[therealbill]

The solution in that example is not a good one really as it limits you to one instance per Docker host.

[therealbill]

Ultimately to run properly in a Docker environment Disque needs the same thing Redis needs. See this blog post describing a "post-start initialization state" for the details. The problems are fundamentally identical.

[antirez]

Hello guys, this feature was implemented in Redis Cluster (unstable branch) and is now under testing. It will be merged into Disque as well and I guess it will be identical. This will break nodes.conf here as well since we need to save the bus port additionally... However here the interface with the client is the more clean HELLO command so there will be no changes for clients AFAIK. Please test the feature in Redis unstable if you want, I did not tested it enough so far so feedbacks are appreciated.

[therealbill]

@antirez So are you saying the Redis Cluster (unstable) method is to use the HELLO command to discover what the cluster bus port is instead of a port offset? If so, will that be runtime configurable? In order to fully solve this problem (for both) it must be reconfigurable at run time so it can be set up after container launch whereupon you know what port to announce.

Since it has been a while since developer day here is the desired sequence of events:

1. You configure and start the container
2. The server starts up on it's configured port
3. The Docker port mapper assigns a different port as the accessible port
4. An external tool queries Docker for the port mapping, then issues a config set to set the bus announce port
5. cluster meet <IP> <port>

Does the method used by the Redis Cluster unstable branch follow this sequence? Or is the method there different, and if so what is it? I'd be happy to test it.

[aclemmensen]

I'm trying to set up a Disque cluster in Mesos and I'm facing the same problem with ports. For me it'd be optimal if the cluster comms port could be configured instead of an increment of 10k. That would allow me to map to a random port at runtime and then use my service discovery mechanism to resolve the actual value and introduce the cluster members to one another.