TODO: Add cluster feature when scale the docker image under stack or service

[vowstar]

• [x] EMQ configured by docker ENV settings
• [x] Plugins load by docker ENV settings
• [x] Plugins configured by docker ENV settings
• [ ] Service discovery under stack or service(e.g. docker-compose case)
• [ ] Cluster feature when change scale container number up, automatically join cluster
• [ ] Automatically leave cluster when scale down container number

[je-al]

I have a somewhat working setup for all of these.

3.- preprocessing of the configuration files through python / regex to generate go-lang text template files which then I process at startup using gucci. Variable names get mapped by ENV files by translating them, dots get replaced by __, the whole thing gets uppercased and default values are set to whatever was originally on the config file.

4-5.- there's a "special" env var which should be set to the service name, then basically nslookup to get all the IPs (does require that the service's endpoint mode is set to dnsrr when on swarm), and a while trying to cluster join.

6.- very raw trap setup.

Also, just because it's cleaner, I'm including supervisord. A proper solution (i.e. a pod running in k8s) would consist of at least three containers

I'm tidying up the thing anyways, so I could fork the repo on github if you're interested.

edit^2: I went ahead and did it anyways, they're available as feature branches at https://github.com/je-al/emq-docker

[vowstar]

Hi, thanks for your efforts to make clustering. I noticed that you use nslookup to get all the IPs, but I found a problem that nslookup depends on busybox under alpine, it only support UDP method to get IP list. So it only can get 14 IPs at most. When swarm mode scale node number above 14, the IP will change every time and incomplete. This is because alpine's busybox use resolv unit to do dns query, but resolv under uclibc only support UDP method that the buffer only NS_PACKETSZ=512bytes. every where NS_PACKETSZ is 512 I used to use UDP to discovery service(at my old docker image https://github.com/devicexx/docker-images/blob/master/library/emqttd/latest/default/start.sh) but it not stable when run opt/emqttd/bin/emqttd_ctl cluster join 'emqttd@'${REMOTE_IP} many times, without success when node above 3. Then I tried method such like you use, using DNS to do service discovery.

I think your script can be optimized, do not use busybox's nslook up under apline due to 14 IPs limit, you can try install bind-tools by apk add --no-cache bind-tools and use nslookup which bind-tools provides, that is no problem. The easiest way is using ubuntu and ubuntu have no IP limits in nslookup(It's depends on glibc).

I'm also learning and studing how to make emq-cluster stable, when I success I'll open source it. Thank you again and very glad to talking with you, I looking forwards to discussion cluster problem again.

[vivobg]

@vowstar Please have a look at the EMQ 2.3 beta. It adds cluster autodiscovery. You might also find my comment to a similar issue useful: https://github.com/emqtt/emq-docker/issues/13#issuecomment-318831416

[vowstar]

@vivobg OK, thanks