Open tukusejssirs opened 2 years ago
the configuration you used is correct but you also need to setup mongodb/redis persistence. mqemitter is used to share messages between aedes instances and mongo to persist qos > 0, retained messages and subscriptions so you also need that.
@robertsLando, thanks for reply! :pray:
I don’t need persistence nor QOS > 0. Do I still need to setup persistence
? All I need to have a cluster and to enable the communication between cluster nodes/brokers.
As for the main question: as I understand it, the config in the OP is not yet a cluster config, is it? If it is not a cluster yet, how to make a cluster config?
aedes-cli
in Docker container?You should setup both mqemitter and persistence in order to run clusters
Then follow this guide to provide the load balancing: https://pspdfkit.com/blog/2018/how-to-use-docker-compose-to-run-multiple-instances-of-a-service-in-development/ or https://sandny.com/2019/11/03/nginx-load-balancer-with-docker-for-nodejs-services/
(there are a lot)
You will have N instances of aedes broker running, mqemitter allows them to share messages (so if you have a client A connected to instance 1 and a client B connected to instance 2 client A will be able receive messages published to instance 2 and vice versa). Persistence is needed to share the same state between all aedes instances
Thanks again. The linked site help me a lot, however, the ports (port
, wsPort
, wssPort
, tlsPort
) collide. Only once container instance runs successfully, others fail with EADDRINUSE
(obviously). See below my current config. I think the culprit is that currently I need to use network_mode
set to host
(all I actually need is to access the broker from on host itself, as my app is not yet running in Docker).
Is there a way to change the ports per container instance? Or how could I work around the issue?
aedes.yml
:version: '3.7'
services:
aedes:
# Note: I needed to remove `container_name`, as each container instance needs to have different name.
# container_name: aedes
image: moscajs/aedes:latest
restart: always
stop_signal: SIGINT
deploy:
# Scale = 3
replicas: 3
network_mode: host
command: -c /data/dockerConfig.js
volumes:
- /some/path/mqtt:/data
mongo:
container_name: mongo
restart: always
network_mode: host
logging:
driver: none
image: mvertes/alpine-mongo
volumes:
- mongo_data:/data/db
ports:
- 27017:27017
volumes:
mongo_data:
name: mongo_data
/some/path/mqtt/dockerConfig.js
module.exports = {
protos: ['tcp', 'ws'],
host: 'localhost',
port: 1883,
wsPort: 3000,
wssPort: 4000,
tlsPort: 8883,
key: null,
cert: null,
rejectUnauthorized: true,
credentials: null,
brokerId: 'aedes',
concurrency: 100,
queueLimit: 42,
maxClientsIdLength: 23,
heartbeatInterval: 60000,
connectTimeout: 30000,
stats: false,
statsInterval: 5000,
persistence: {
name: 'mongodb',
options: {
url: 'mongodb://localhost:27017/aedes'
}
},
mq: {
name: 'mongodb',
options: {
url: 'mongodb://localhost:27017/aedes'
}
},
verbose: false,
veryVerbose: false,
noPretty: false
}
Update
I understand I can append --port
to command
in Docker compose file, but how can I make the ports different in each container instance? Also I need to use both tcp
and ws
protocols; how could I define them both use --port
and --protos
(I think it is not yet possible)? I think I need to do it similarly to this, but I have no idea how to make it. I understand that it is related more to Docker than aedes-cli
though.
The approach I suggest is to create multiple aedes containers directly in compose file naming them like aedes1
aedes2
and aedes3
... based on how many instances you need, the configuration they share is the same just but then inside nginx you will have something like:
upstream aedes {
server aedes1::1883;
server aedes2::1883;
server aedes3::1883;
}
and
server {
location / {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass "http://aedes";
}
}
You don't have to to expose aedes or mongodb ports in docker-compose (remove network_mode: host
from your docker-compose file).
@robertsLando, sorry for being away for a while. I am in the middle of dockerising my app, therefore now I came back to this issue.
I can successfully run the following aedes
container. According to docker logs aedes
, all three replicas should be running (both TCP and WS), however, I cannot connect to it from another container running in the same network (someNetwork
). It throws ECONNREFUSED
error.
For now, I have not done config in nginx
related to aedes
and its load-balancing. Should I implement that if I want to access it from within someNetwork
only?
What could be wrong with this config?
# docker-compose
aedes:
image: moscajs/aedes:latest
restart: unless-stopped
stop_signal: SIGINT
deploy:
replicas: 3
networks:
- someNetwork
command: -c /data/aedes.js
volumes:
- /some/path/aedes.js:/data/aedes.js:ro
// /some/path/aedes.js
module.exports = {
protos: ['tcp', 'ws'],
host: 'localhost',
port: 8193,
wsPort: 4000,
wssPort: 4001,
tlsPort: 8883,
key: null,
cert: null,
rejectUnauthorized: false,
credentials: null,
brokerId: 'aedes',
concurrency: 0,
queueLimit: 200,
maxClientsIdLength: 23,
heartbeatInterval: 60000,
connectTimeout: 30000,
stats: false,
statsInterval: 5000,
persistence: {
name: 'mongodb',
options: {
url: 'mongodb://mongo:27017/aedes'
}
},
mq: {
name: 'mongodb',
options: {
url: 'mongodb://mongo:27017/aedes'
}
},
verbose: false,
veryVerbose: false,
noPretty: false
}
// Client (in another container); both TCP and WS connection fails with `ECONNREFUSED`
const {connect} = require('mqtt')
const tcpPort = 8193
const tcpClient = connect(`mqtt://aedes:${tcpPort}`)
tcpClient.on('error', e => {
console.log(`MQTT error:`, e)
})
const wsPort = 4000
const wsClient = connect(`mqtt://aedes:${wsPort}`)
wsClient.on('error', e => {
console.log(e)
})
// Sample output:
// - same output for both TCP and WS;
// - IP addresses cycle through the replicas.
Error: connect ECONNREFUSED 172.29.0.6:8193
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1195:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '172.29.0.6',
port: 8193
}
Update: I also tried to run the client code within aedes
container using 127.0.0.1
, but I still get ECONNREFUSED
.
Update 2: When I comment out host: 'localhost'
from aedes.js
(or when I use host: '0.0.0.0'
, it works locally in the aedes
container, but only for WS, never for TCP. I tried it in all instances of aedes
. It still does not work in another container (with and without using a custom Docker network).
Update 3: I believe that the ports are not open outside of the aedes
container. I tested this using nmap
both in the container of my app and on the host system. I tried to add 8193
and 4000
to ports
in the Docker compose file (without mapping host port specification), however, it didn’t solve the issue.
Update 4: I ‘solved’ this issue by modifying the official example. I believe the ports issue is on the Docker image/container level, but I have no idea why it does not work.
First asked on SO.
I need to create a MQTT broker cluster. I want to use
aedes
(as I already use it, albeit without cluster) and Docker. Note that I have never created a cluster before (with nor without Docker).Here I found official example how to create a cluster using
aedes
, but I was thinking about usingaedes-cli
Docker image.Now, I have no idea which is better: should I create separate containers per broker (similarly to this question) or use
mqemitter-redis
/mqemitter-mongodb
(as theaedes-cli
docs suggest).As for creating separate containers, I have no idea how to connect them.
As for using
mqemitter-redis
/mqemitter-mongodb
, I have no idea how to setup them in theaedes-cli
config file.I have no need for data persistance accross broker cluster restarts. All I need is to distribute the messages accross multiple brokers in order to improve the runtime performance.
Below is a working, single-broker configuration.
aedes.yml
:/some/path/mqtt/dockerConfig.js