Bridge configuration on emqx-edge docker containers seems to be either missing or broken, or patchy.
If would be great to be able to specify bridge configuration with environment variables in docker run command and have them override any bridge template ( eg. the default bridge.aws.X configuration in the /opt/emqx/etc/plugins/emqx_bridge_mqtt.conf file
Justification
This feature used to work in previous versions of docker containers, but due to new layout of plugin files and parsing ( and an assumption of the aws namespace of the bridge ) it seems to have mixed results at best.
As many bridges as the user wants could be configured with some improvement in the docker-enttrypoint.sh.
Workarounds
Custom docker build of emqx-edge with following Dockerfile and docker-entrypoint.sh
Dockerfile
FROM emqx/emqx-edge:v3.2.2
COPY docker-entrypoint.sh /usr/bin/docker-entrypoint.sh
ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]
CMD ["/usr/bin/start.sh"]
docker-entrypoint.sh
#!/bin/sh
## EMQ docker image start script
# Huang Rui <vowstar@gmail.com>
# EMQ X Team <support@emqx.io>
## Shell setting
if [[ ! -z "$DEBUG" ]]; then
set -ex
else
set -e
fi
## Local IP address setting
LOCAL_IP=$(hostname -i |grep -E -oh '((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])'|head -n 1)
## EMQ Base settings and plugins setting
# Base settings in /opt/emqx/etc/emqx.conf
# Plugin settings in /opt/emqx/etc/plugins
_EMQX_HOME="/opt/emqx"
if [[ -z "$PLATFORM_ETC_DIR" ]]; then
export PLATFORM_ETC_DIR="$_EMQX_HOME/etc"
fi
if [[ -z "$PLATFORM_LOG_DIR" ]]; then
export PLATFORM_LOG_DIR="$_EMQX_HOME/log"
fi
if [[ -z "$EMQX_NAME" ]]; then
export EMQX_NAME="$(hostname)"
fi
if [[ -z "$EMQX_HOST" ]]; then
case $EMQX_CLUSTER__K8S__ADDRESS_TYPE in
"dns")
DNSAddress="${LOCAL_IP//./-}.${EMQX_CLUSTER__K8S__NAMESPACE}.pod.cluster.local"
export EMQX_HOST="$DNSAddress"
;;
"hostname")
DNSAddress=$(sed -n "/^${LOCAL_IP}/"p /etc/hosts | grep -e "$(hostname).*.${EMQX_CLUSTER__K8S__NAMESPACE}.svc.cluster.local" -o)
export EMQX_HOST="$DNSAddress"
;;
*)
export EMQX_HOST="$LOCAL_IP"
;;
esac
fi
if [[ -z "$EMQX_WAIT_TIME" ]]; then
export EMQX_WAIT_TIME=5
fi
if [[ -z "$EMQX_NODE__NAME" ]]; then
export EMQX_NODE__NAME="$EMQX_NAME@$EMQX_HOST"
fi
# Set hosts to prevent cluster mode failed
# unset EMQX_NAME
# unset EMQX_HOST
if [[ -z "$EMQX_NODE__PROCESS_LIMIT" ]]; then
export EMQX_NODE__PROCESS_LIMIT=2097152
fi
if [[ -z "$EMQX_NODE__MAX_PORTS" ]]; then
export EMQX_NODE__MAX_PORTS=1048576
fi
if [[ -z "$EMQX_NODE__MAX_ETS_TABLES" ]]; then
export EMQX_NODE__MAX_ETS_TABLES=2097152
fi
if [[ -z "$EMQX__LOG_CONSOLE" ]]; then
export EMQX__LOG_CONSOLE="console"
fi
if [[ -z "$EMQX_LISTENER__TCP__EXTERNAL__ACCEPTORS" ]]; then
export EMQX_LISTENER__TCP__EXTERNAL__ACCEPTORS=64
fi
if [[ -z "$EMQX_LISTENER__TCP__EXTERNAL__MAX_CLIENTS" ]]; then
export EMQX_LISTENER__TCP__EXTERNAL__MAX_CLIENTS=1000000
fi
if [[ -z "$EMQX_LISTENER__SSL__EXTERNAL__ACCEPTORS" ]]; then
export EMQX_LISTENER__SSL__EXTERNAL__ACCEPTORS=32
fi
if [[ -z "$EMQX_LISTENER__SSL__EXTERNAL__MAX_CLIENTS" ]]; then
export EMQX_LISTENER__SSL__EXTERNAL__MAX_CLIENTS=500000
fi
if [[ -z "$EMQX_LISTENER__WS__EXTERNAL__ACCEPTORS" ]]; then
export EMQX_LISTENER__WS__EXTERNAL__ACCEPTORS=16
fi
if [[ -z "$EMQX_LISTENER__WS__EXTERNAL__MAX_CLIENTS" ]]; then
export EMQX_LISTENER__WS__EXTERNAL__MAX_CLIENTS=250000
fi
# Fix issue #42 - export env EMQX_DASHBOARD__DEFAULT_USER__PASSWORD to configure
# 'dashboard.default_user.password' in etc/plugins/emqx_dashboard.conf
if [[ ! -z "$EMQX_ADMIN_PASSWORD" ]]; then
export EMQX_DASHBOARD__DEFAULT_USER__PASSWORD=$EMQX_ADMIN_PASSWORD
fi
# Catch all EMQX_ prefix environment variable and match it in configure file
CONFIG="${_EMQX_HOME}/etc/emqx.conf"
CONFIG_PLUGINS="${_EMQX_HOME}/etc/plugins"
CONFIG_MQTT_BRIDGE_PLUGIN="${_EMQX_HOME}/etc/plugins/emqx_bridge_mqtt.conf"
# If Bridge plugin configuration detected in env
# then remove any existing config
if [[ "$(echo $(env) | grep -E 'EMQX_BRIDGE__MQTT__')" ]]; then
echo "Cleaning bridge config"
echo "" > $CONFIG_MQTT_BRIDGE_PLUGIN
fi
for VAR in $(env)
do
# Config normal keys such like node.name = emqx@127.0.0.1
if [[ ! -z "$(echo $VAR | grep -E '^EMQX_')" ]]; then
VAR_NAME=$(echo "$VAR" | sed -r "s/EMQX_([^=]*)=.*/\1/g" | tr '[:upper:]' '[:lower:]' | sed -r "s/__/\./g")
VAR_FULL_NAME=$(echo "$VAR" | sed -r "s/([^=]*)=.*/\1/g")
# Config in emq.conf
if [[ ! -z "$(cat $CONFIG |grep -E "^(^|^#*|^#*\s*)$VAR_NAME")" ]]; then
echo "$VAR_NAME=$(eval echo \$$VAR_FULL_NAME)"
echo "$(sed -r "s/(^#*\s*)($VAR_NAME)\s*=\s*(.*)/\2 = $(eval echo \$$VAR_FULL_NAME|sed -e 's/\//\\\//g')/g" $CONFIG)" > $CONFIG
fi
# Config in plugins/*
for CONFIG_PLUGINS_FILE in $(ls $CONFIG_PLUGINS); do
if [[ ! -z "$(cat $CONFIG_PLUGINS/$CONFIG_PLUGINS_FILE |grep -E "^(^|^#*|^#*\s*)$VAR_NAME")" ]]; then
echo "$VAR_NAME=$(eval echo \$$VAR_FULL_NAME)"
echo "$(sed -r "s/(^#*\s*)($VAR_NAME)\s*=\s*(.*)/\2 = $(eval echo \$$VAR_FULL_NAME|sed -e 's/\//\\\//g')/g" $CONFIG_PLUGINS/$CONFIG_PLUGINS_FILE)" > $CONFIG_PLUGINS/$CONFIG_PLUGINS_FILE
fi
done
# MQTT Bridge configurations
if [[ "$(echo $VAR_FULL_NAME | grep -E '^EMQX_BRIDGE__MQTT__')" ]]; then
echo "$VAR_NAME=$(eval echo \$$VAR_FULL_NAME)"
echo "$VAR_NAME=$(eval echo \$$VAR_FULL_NAME)" >> $CONFIG_MQTT_BRIDGE_PLUGIN
fi
fi
# Config template such like {{ platform_etc_dir }}
if [[ ! -z "$(echo $VAR | grep -E '^PLATFORM_')" ]]; then
echo "$VAR_NAME=$(eval echo \$$VAR_FULL_NAME)"
VAR_NAME=$(echo "$VAR" | sed -r "s/([^=]*)=.*/\1/g"| tr '[:upper:]' '[:lower:]')
VAR_FULL_NAME=$(echo "$VAR" | sed -r "s/([^=]*)=.*/\1/g")
echo "$(sed -r "s@\{\{\s*$VAR_NAME\s*\}\}@$(eval echo \$$VAR_FULL_NAME|sed -e 's/\//\\\//g')@g" $CONFIG)" > $CONFIG
fi
done
## EMQ Plugin load settings
# Plugins loaded by default
if [[ ! -z "$EMQX_LOADED_PLUGINS" ]]; then
echo "EMQX_LOADED_PLUGINS=$EMQX_LOADED_PLUGINS"
# First, remove special char at header
# Next, replace special char to ".\n" to fit emq loaded_plugins format
echo $(echo "$EMQX_LOADED_PLUGINS."|sed -e "s/^[^A-Za-z0-9_]\{1,\}//g"|sed -e "s/[^A-Za-z0-9_]\{1,\}/\. /g")|tr ' ' '\n' > /opt/emqx/data/loaded_plugins
fi
exec "$@"
The crucial lines are:
# If Bridge plugin configuration detected in env
# then remove any existing config
if [[ "$(echo $(env) | grep -E 'EMQX_BRIDGE__MQTT__')" ]]; then
echo "Cleaning bridge config"
echo "" > $CONFIG_MQTT_BRIDGE_PLUGIN
fi
Which detects if any bridge configuration is specified on the environment. In which case it cleans out any default configuration present in the plugins conif dir, assuming ( because bridges exist in custom namespaces ) that any pre-existing configuration will not be relevant.
It also makes an assumption that all necessary bridge configuration vars will be provided by environment.
# MQTT Bridge configurations
if [[ "$(echo $VAR_FULL_NAME | grep -E '^EMQX_BRIDGE__MQTT__')" ]]; then
echo "$VAR_NAME=$(eval echo \$$VAR_FULL_NAME)"
echo "$VAR_NAME=$(eval echo \$$VAR_FULL_NAME)" >> $CONFIG_MQTT_BRIDGE_PLUGIN
fi
Which appends all bridge related configuration key=value pairs to the bridge plugin configuration file.
Missing feature
Bridge configuration on emqx-edge docker containers seems to be either missing or broken, or patchy.
If would be great to be able to specify bridge configuration with environment variables in docker run command and have them override any bridge template ( eg. the default bridge.aws.X configuration in the /opt/emqx/etc/plugins/emqx_bridge_mqtt.conf file
Justification
This feature used to work in previous versions of docker containers, but due to new layout of plugin files and parsing ( and an assumption of the aws namespace of the bridge ) it seems to have mixed results at best.
As many bridges as the user wants could be configured with some improvement in the docker-enttrypoint.sh.
Workarounds
Custom docker build of emqx-edge with following Dockerfile and docker-entrypoint.sh
Dockerfile
docker-entrypoint.sh
The crucial lines are:
Which detects if any bridge configuration is specified on the environment. In which case it cleans out any default configuration present in the plugins conif dir, assuming ( because bridges exist in custom namespaces ) that any pre-existing configuration will not be relevant.
It also makes an assumption that all necessary bridge configuration vars will be provided by environment.
Which appends all bridge related configuration key=value pairs to the bridge plugin configuration file.