Mailcow and nginx-proxy

[daschxx]

I want to use Mailcow behind a Nginx Proxy in Docker [https://github.com/jwilder/nginx-proxy] I added in docker-compose.override.yml:

version: '2.1'
services:

    nginx-mailcow:
      environment:
        - LETSENCRYPT_HOST=${MAILCOW_HOSTNAME}
        - LETSENCRYPT_EMAIL=mail@${MAILCOW_HOSTNAME}
        - VIRTUAL_HOST=${MAILCOW_HOSTNAME}
        - VIRTUAL_PROTO=http
        - VIRTUAL_PORT=80
      networks:
        webproxy:

networks:
  webproxy:
    external:
      name: webproxy

in docker-compose.yml:

nginx-mailcow:
[...]
#      ports:
#        - "${HTTPS_BIND:-0.0.0.0}:${HTTPS_PORT:-443}:${HTTPS_PORT:-443}"
#        - "${HTTP_BIND:-0.0.0.0}:${HTTP_PORT:-80}:${HTTP_PORT:-80}"

but now i get error Messages from Watchdog for rspamd and nginx inside mailcow.

I think the problem is the second network, but I have no idea how to solve it.

Is it possible to run Mailcow behind a Proxy inside Docker?

[stevesbrain]

Take a read of this page - specifically, don't modify the docker compose, instead, modify mailcow conf file, adjust the "behind the scenes" port of nginx in there, and then point your reverse proxy at that port instead.

[daschxx]

When I use 127.0.0.1 in a dockerised nginx proxy, nginx use the docker localhost and not the hosts localhost. My Problem is not to use mailcow behind a proxy runing on host. My problem is runing it behind a proxy runing in another docker-composer instance.

[stevesbrain]

Ah, right - in that case, you'd need to make it part of a network, and specify the host/hostname/ip etc. for it (rather than localhost). Either that, or force your nginx container to use bridged networking.

[daschxx]

Can you explain it a little bit more.

Now i create a docker-compose project with nginx

version: '3'

services:
  server:
    image: nginx:alpine
    restart: always
    volumes:
      - /opt/data/nginx-test/conf.d:/etc/nginx/conf.d
    ports:
      - 80:80
      - 443:443
    networks:
      default:

networks:
  default:
    external:
      name: webproxy

the nginx config in /opt/data/nginx-test/conf.d/default.conf

# mail.domain.org
upstream mail.domain.org {
    server mailcow_nginx-mailcow_1:8080;
}
server {
    server_name mail.domain.org;
    listen 80 ;
    access_log /var/log/nginx/access.log;
    location / {
    proxy_pass http://mail.domain.org;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        client_max_body_size 100m;
    }
}

and the docker-compose.override.yml:

version: '2.1'
services:
    nginx-mailcow:
      networks:
        webproxy:

networks:
  webproxy:
    external:
      name: webproxy

it seams to work, but watchdog sends me every 5 minutes an email:

Watchdog: nginx-mailcow service hit the error rate limit Service was restarted, please check your mailcow installation.

in the docker logs I can’t find any error. (_docker logs -f mailcow_nginx-mailcow1)

Any idea what I make wrong?

[stevesbrain]

Is it possible to run the nginx in the foreground so you can perhaps watch it in real time (the one that's restarting)? I'm on my mobile at the moment, so can't test out your config at all right now!

[daschxx]

You mean docker-compose up nginx-mailcow? Here the logs:

1. run

   nginx-mailcow_1      | fd4d:6169:6c63:xxxx::1 - - [01/May/2018:19:14:32 +0000] "GET /forwardinghosts.php HTTP/1.1" 200 27 "-" "-"
   nginx-mailcow_1      | 172.22.1.15 - - [01/May/2018:19:14:49 +0000] "HEAD /settings.php HTTP/1.1" 200 0 "-" "-"
   mailcow_nginx-mailcow_1 exited with code 0

2. run

   nginx-mailcow_1      | fd4d:6169:6c63:xxxx::1 - - [01/May/2018:19:15:23 +0000] "GET /forwardinghosts.php?host=xxx.xxx.xx.xx HTTP/1.1" 200 19 "-" "curl/7.52.1"
   [...]
   nginx-mailcow_1      | fd4d:6169:6c63:xxxx::1 - - [01/May/2018:19:18:36 +0000] "GET /settings.php HTTP/1.1" 200 247 "-" "-"
   nginx-mailcow_1      | 172.22.1.15 - - [01/May/2018:19:18:37 +0000] "HEAD /forwardinghosts.php HTTP/1.1" 200 0 "-" "-"
   mailcow_nginx-mailcow_1 exited with code 0

3. run

   nginx-mailcow_1      | fd4d:6169:6c63:xxxx::1 - - [01/May/2018:19:19:15 +0000] "HEAD /settings.php HTTP/1.1" 200 0 "-" "-"
   [...]
   nginx-mailcow_1      | fd4d:6169:6c63:xxxx::1 - - [01/May/2018:19:21:59 +0000] "GET /forwardinghosts.php HTTP/1.1" 200 27 "-" "-"
   nginx-mailcow_1      | fd4d:6169:6c63:xxxx::1 - - [01/May/2018:19:22:45 +0000] "HEAD /settings.php HTTP/1.1" 200 0 "-" "-"
   nginx-mailcow_1      | 172.22.1.15 - - [01/May/2018:19:22:45 +0000] "GET /settings.php HTTP/1.1" 200 247 "-" "-"
   mailcow_nginx-mailcow_1 exited with code 0

PS: I use the dev branch. Tomorrow I will try it on a new system with master branch.

[chrisheinle]

Can you please also have a look at: https://stackoverflow.com/questions/50074479/using-mailcow-dockerized-with-a-dockerized-apache2-4 . I am facing a similiar issue but didn't want to open an issue here.

[daschxx]

@chrisheinle: The Url ist wrong and the problem is another, you have to proxy to the mailcow hostname, in default: nginx, for testing, try the http port.

    ProxyPass / http://nginx:8080/
    ProxyPassReverse / http://nginx:8080/

[daschxx]

Today I try a new clean master branch mailcow, its the same problem, I have access to the website, but the mailcow nginx container restarts every 5 minutes.

[stevesbrain]

@daschxx I'll see if I have some time today to try and replicate the setup/scenario you are using!

[daschxx]

I think I found the problem. In data/Dockerfiles/watchdog/watchdog.sh the funktion to get the container ip returns 2 IPs and so the other comands hits an error.

get_container_ip() {
  # ${1} is container
  CONTAINER_ID=
  CONTAINER_IP=
  LOOP_C=1
  until [[ ${CONTAINER_IP} =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]] || [[ ${LOOP_C} -gt 5 ]]; do
    sleep 1
    CONTAINER_ID=$(curl --silent http://dockerapi:8080/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" | jq -rc "select( .name | tostring | contains(\"${1}\")) | .id")
    if [[ ! -z ${CONTAINER_ID} ]]; then
      CONTAINER_IP=$(curl --silent http://dockerapi:8080/containers/${CONTAINER_ID}/json | jq -r '.NetworkSettings.Networks[].IPAddress')
    fi
    LOOP_C=$((LOOP_C + 1))
  done
  [[ ${LOOP_C} -gt 5 ]] && echo 240.0.0.0 || echo ${CONTAINER_IP}
}

[daschxx]

see #1427 and #1409

[zedtux]

Thank you @daschxx, you saved me ! 🎉

[bplein]

My apologies, I'm running into this after moving mailcow behind a newly installed nginx proxy.

The proxy is operating correctly, I am able to use Mailcow UI and Sogo, as well as another unrelated node.js app that is behind the same proxy.

I am using an unmodified docker-compose.yml file (updated to the latest today).

Here are the appropriate sections of my docker-compose.override.yml (I am leaving out my volumes)

version: '2.1'
services:
    portainer-mailcow:
      image: portainer/portainer
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - ./data/conf/portainer:/data
      restart: always
      dns:
        - 172.22.1.254
      dns_search: mailcow-network
      networks:
        mailcow-network:
          aliases:
            - portainer
    nginx-mailcow:
      networks:
        proxy-tier:

networks:
  proxy-tier:
    external:
      name: nginx-proxy

I have set the appropriate section of mailcow.conf as follows:

HTTP_PORT=9080
HTTP_BIND=127.0.0.1

HTTPS_PORT=9443
HTTPS_BIND=127.0.0.1

If I exec a shell in the watchdog, and run the same sections of the get_container_ip() function by hand, I get a single string with my 2 IP addresses in it. So the loop is bombing out and the test is testing against 240.0.0.0

Please advise. Maybe I'm just getting tired :)

here is a snippet from "docker network inspect" on the two networks:

"895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28": { "Name": "mc_nginx-mailcow_1", "EndpointID": "457191b0d2b6abfd7276adbacbb864ffa06ee17ba27f35401646cac696438cb4", "MacAddress": "02:42:ac:16:01:0d", "IPv4Address": "172.22.1.13/24", "IPv6Address": "" },

"895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28": { "Name": "mc_nginx-mailcow_1", "EndpointID": "4ea42c0f18cf1cf573cac38171b25537e544985440fcc613e80a1f2046f94c1b", "MacAddress": "02:42:ac:18:00:05", "IPv4Address": "172.24.0.5/16", "IPv6Address": "" },

[andryyy]

This was fixed a while ago. It just uses one IP by random choice. Which function did you copy? And from which file?

[bplein]

Thank you (as always) for your very quick response! (edited a minute later for clarity)

I enter the running watchdog container by execing bash. Then I copy and pasted, line by line, "get_container_ip" from watchdog.sh, putting in the nginx-mailcow name, from within the running container. Then I echo $CONTAINER_IP. The output for CONTAINER_IP is a string with my two IPs in it, like "172.22.1.13 172.24.0.5".

If I do "watch PS" inside the running container, I can see it test 240.0.0.0 port 8081.

[andryyy]

Do you have two containers with the same name? What is your watchdog image version?

The container name is not mc_nginx-mailcow by default. These are not unmodified files. :-/

[andryyy]

@bplein Could you please try to change the image version in docker-compose.yml for watchdog-mailcow to 1.20? => image: mailcow/watchdog:1.20

Run docker-compose pull watchdog-mailcow; docker-compose up -d then, please. :-)

[bplein]

I changed the project name in mailcow.conf to make it shorter to type.

root@mothra:/fiomirror/mailcow# grep mc mailcow.conf
COMPOSE_PROJECT_NAME=mc

This is a diff from my version of docker-compose.yml and your master branch as of yesterday on the right.

root@mothra:/fiomirror/mailcow# diff docker-compose.yml /fio3200/bill/foo/mailcow-dockerized/docker-compose.yml
383c383
<     enable_ipv6: false
---
>     enable_ipv6: true

All running containers:

root@mothra:/fiomirror/mailcow# dockerps
CONTAINER ID        IMAGE                                    STATUS              NAMES
b75d18ff73ab        mailcow/watchdog:1.19                    Up 11 hours         mc_watchdog-mailcow_1
918ed0153035        mailcow/netfilter:1.19                   Up 11 hours         mc_netfilter-mailcow_1
adcfbb2ae2b5        mailcow/rspamd:1.24                      Up 11 hours         mc_rspamd-mailcow_1
b9892c8410eb        mailcow/acme:1.37                        Up 11 hours         mc_acme-mailcow_1
895314535f60        nginx:mainline-alpine                    Up 11 hours         mc_nginx-mailcow_1
c9e669f252b2        mailcow/phpfpm:1.18                      Up 11 hours         mc_php-fpm-mailcow_1
6f25f9d070c5        mailcow/sogo:1.34                        Up 11 hours         mc_sogo-mailcow_1
80b785ec1921        robbertkl/ipv6nat                        Up 11 hours         mc_ipv6nat_1
fd7ba33033d6        mailcow/unbound:1.1                      Up 11 hours         mc_unbound-mailcow_1
f6ceefa7ba3d        mailcow/clamd:1.13                       Up 11 hours         mc_clamd-mailcow_1
3080ab658a8e        portainer/portainer                      Up 11 hours         mc_portainer-mailcow_1
83e7e44b7e0b        mailcow/postfix:1.21                     Up 11 hours         mc_postfix-mailcow_1
379b6f2045e5        mailcow/dockerapi:1.13                   Up 11 hours         mc_dockerapi-mailcow_1
b724f9190a9e        memcached:alpine                         Up 11 hours         mc_memcached-mailcow_1
97c15bbf8dac        mariadb:10.2                             Up 11 hours         mc_mysql-mailcow_1
73757efdf1bc        redis:4-alpine                           Up 11 hours         mc_redis-mailcow_1
b7632fab7fa7        mailcow/dovecot:1.33                     Up 11 hours         mc_dovecot-mailcow_1
c06668be27fe        sdelements/lets-chat:latest              Up 13 hours         letschat_letschat_1
f041bd0a3ed5        mongo:latest                             Up 14 hours         letschat_mongo_1
061396d2fc7e        jrcs/letsencrypt-nginx-proxy-companion   Up 14 hours         nginx-proxy_letsencrypt-nginx-proxy-companion_1
6befe882eb4e        jwilder/nginx-proxy                      Up 14 hours         nginx-proxy_nginx-proxy_1
f61839513c50        dtandersen/factorio:0.16.51              Up 40 hours         factorio1
5550b457f593        dtandersen/factorio:0.16.51              Up 40 hours         factorio2
3a256281aef5        goofball222/unifi                        Up 2 days           unifi
706db0313d1c        mongo                                    Up 2 days           unifidb

[bplein]

I will re-enable watchdog and step through my debugging process so you can see what I mean from my prior testing, but in detail. Here is my test: The only lines I've changed from the get_container_ip function are the replacement of the container ID for ${1}, and an extra line in the middle showing the value of CONTAINER_IP in the middle of the loop.

root@mothra:/fiomirror/mailcow# docker exec -ti mc_watchdog-mailcow_1 bash
bash-4.3#   CONTAINER_ID=()
bash-4.3#   CONTAINER_IP=
bash-4.3#   LOOP_C=1
bash-4.3#   until [[ ${CONTAINER_IP} =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]] || [[ ${LOOP_C} -gt 5 ]]; do
>     sleep 0.5
>     # get long container id for exact match
>     CONTAINER_ID=($(curl --silent http://dockerapi:8080/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" | jq -rc "select( .name | tostring == \"nginx-mailcow\") | .id"))
>     CONTAINER_ID=($(printf "%s\n" "${CONTAINER_ID[@]}" | shuf))
>     if [[ ! -z ${CONTAINER_ID} ]]; then
>       for matched_container in "${CONTAINER_ID[@]}"; do
>         CONTAINER_IP=$(curl --silent http://dockerapi:8080/containers/${matched_container}/json | jq -r '.NetworkSettings.Networks[].IPAddress')
> echo "testing: my container IP is $CONTAINER_IP"
>         # grep will do nothing if one of these vars is empty
>         [[ -z ${CONTAINER_IP} ]] && continue
>         [[ -z ${IPV4_NETWORK} ]] && continue
>         # only return ips that are part of our network
>         if ! grep -q ${IPV4_NETWORK} <(echo ${CONTAINER_IP}); then
>           CONTAINER_IP=
>         fi
>       done
>     fi
>     LOOP_C=$((LOOP_C + 1))
>   done
testing: my container IP is 172.24.0.5
172.22.1.13
testing: my container IP is 172.24.0.5
172.22.1.13
testing: my container IP is 172.24.0.5
172.22.1.13
testing: my container IP is 172.24.0.5
172.22.1.13
testing: my container IP is 172.24.0.5
172.22.1.13
bash-4.3#   [[ ${LOOP_C} -gt 5 ]] && echo 240.0.0.0 || echo ${CONTAINER_IP}
240.0.0.0
bash-4.3#

[bplein]

Looks like it's not a space but another character between the two IPs.

[bplein]

Here is the output from "watch ps | grep 8081" from inside the running watchdog container:

bash-4.3# watch ps | grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3401 root       0:00 grep 8081
 3729 root       0:00 /usr/lib/nagios/plugins/check_http -4 -H 240.0.0.0 -u / -p 8081
 3401 root       0:00 grep 8081
 3729 root       0:00 /usr/lib/nagios/plugins/check_http -4 -H 240.0.0.0 -u / -p 8081
 3401 root       0:00 grep 8081
 3729 root       0:00 /usr/lib/nagios/plugins/check_http -4 -H 240.0.0.0 -u / -p 8081
 3401 root       0:00 grep 8081
 3729 root       0:00 /usr/lib/nagios/plugins/check_http -4 -H 240.0.0.0 -u / -p 8081
 3401 root       0:00 grep 8081
 3729 root       0:00 /usr/lib/nagios/plugins/check_http -4 -H 240.0.0.0 -u / -p 8081
 3401 root       0:00 grep 8081

[bplein]

Sorry, i missed your request to do 1.20, trying that how (prior tests were with 1.19)

[bplein]

root@mothra:/fiomirror/mailcow# docker-compose pull watchdog-mailcow; docker-compose up -d
Pulling watchdog-mailcow ... done
mc_sogo-mailcow_1 is up-to-date
mc_redis-mailcow_1 is up-to-date
mc_dovecot-mailcow_1 is up-to-date
mc_postfix-mailcow_1 is up-to-date
mc_dockerapi-mailcow_1 is up-to-date
mc_ipv6nat_1 is up-to-date
mc_memcached-mailcow_1 is up-to-date
mc_clamd-mailcow_1 is up-to-date
mc_mysql-mailcow_1 is up-to-date
mc_portainer-mailcow_1 is up-to-date
mc_unbound-mailcow_1 is up-to-date
Recreating mc_watchdog-mailcow_1 ...
mc_php-fpm-mailcow_1 is up-to-date
mc_nginx-mailcow_1 is up-to-date
mc_netfilter-mailcow_1 is up-to-date
mc_acme-mailcow_1 is up-to-date
Recreating mc_watchdog-mailcow_1 ... done
root@mothra:/fiomirror/mailcow# dockerps | grep watch
9468200ee097        mailcow/watchdog:1.20                    Up About a minute   mc_watchdog-mailcow_1
root@mothra:/fiomirror/mailcow# docker exec -ti mc_watchdog-mailcow_1 bash
bash-4.3# ps | grep 8081
  440 root       0:00 grep 8081
bash-4.3# watch ps | grep 8081
  444 root       0:00 grep 8081
  444 root       0:00 grep 8081
  444 root       0:00 grep 8081
  444 root       0:00 grep 8081
  444 root       0:00 grep 8081
  444 root       0:00 grep 8081
  444 root       0:00 grep 8081
  444 root       0:00 grep 8081
  444 root       0:00 grep 8081
  646 root       0:00 /usr/lib/nagios/plugins/check_http -4 -H 240.0.0.0 -u / -p 8081
  444 root       0:00 grep 8081
  646 root       0:00 /usr/lib/nagios/plugins/check_http -4 -H 240.0.0.0 -u / -p 8081
  444 root       0:00 grep 8081
  646 root       0:00 /usr/lib/nagios/plugins/check_http -4 -H 240.0.0.0 -u / -p 8081
  444 root       0:00 grep 8081
  646 root       0:00 /usr/lib/nagios/plugins/check_http -4 -H 240.0.0.0 -u / -p 8081
  444 root       0:00 grep 8081
  646 root       0:00 /usr/lib/nagios/plugins/check_http -4 -H 240.0.0.0 -u / -p 8081
  444 root       0:00 grep 8081

[bplein]

By the way, is there a reason that the script has to determine the IP? I know a simpler pull request was rejected, so I assume that there's a good reason.

If the same ping just used the container name, which is handled by docker DNS, it just works. (maybe there are cases where people have non-standard docker networking and name resolution?)

root@mothra:/fiomirror/mailcow# docker exec -ti mc_watchdog-mailcow_1 bash
bash-4.3# cat watchdog.sh | grep 8081
    /usr/lib/nagios/plugins/check_http -4 -H ${host_ip} -u / -p 8081 1>&2; err_count=$(( ${err_count} + $? ))
bash-4.3# /usr/lib/nagios/plugins/check_http -4 -H nginx-mailcow -u / -p 8081
HTTP OK: HTTP/1.1 200 OK - 238 bytes in 0.001 second response time |time=0.001127s;;;0.000000 size=238B;;;0

[andryyy]

Yes, it was discussed in another thread. But we might go back to not determining the IP after all... there's some stuff that needs to be changed though.

You also have a non-standard setup, that's always problematic and technically not supported. But we will find the issue...

Can you paste this inside a watchdog bash?

get_container_ip() {
  # ${1} is container
  CONTAINER_ID=()
  CONTAINER_IP=
  LOOP_C=1
  until [[ ${CONTAINER_IP} =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]] || [[ ${LOOP_C} -gt 5 ]]; do
    sleep 0.5
    # get long container id for exact match
    CONTAINER_ID=($(curl --silent http://dockerapi:8080/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" | jq -rc "select( .name | tostring == \"${1}\") | .id"))
    # returned id can have multiple elements (if scaled), shuffle for random test
    CONTAINER_ID=($(printf "%s\n" "${CONTAINER_ID[@]}" | shuf))
    echo "Found ${CONTAINER_ID[@]}"
    if [[ ! -z ${CONTAINER_ID} ]]; then
      for matched_container in "${CONTAINER_ID[@]}"; do 
    CONTAINER_IP=$(curl --silent http://dockerapi:8080/containers/${matched_container}/json | jq -r '.NetworkSettings.Networks[].IPAddress')
    echo "${matched_container} has ${CONTAINER_IP}"
        # grep will do nothing if one of these vars is empty
        [[ -z ${CONTAINER_IP} ]] && continue
        [[ -z ${IPV4_NETWORK} ]] && continue
        # only return ips that are part of our network
        if ! grep -q ${IPV4_NETWORK} <(echo ${CONTAINER_IP}); then
          echo "IP did not match network"
          CONTAINER_IP=
        else
          echo "Break"
          break
        fi
      done
    fi
    LOOP_C=$((LOOP_C + 1))
  done
  [[ ${LOOP_C} -gt 5 ]] && echo 240.0.0.0 || echo ${CONTAINER_IP}
}

Run get_container_ip nginx-mailcow after, please. :-)

[bplein]

root@mothra:/fiomirror/mailcow# docker exec -ti mc_watchdog-mailcow_1 bash
bash-4.3# get_container_ip() {
>   # ${1} is container
>   CONTAINER_ID=()
>   CONTAINER_IP=
>   LOOP_C=1
>   until [[ ${CONTAINER_IP} =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]] || [[ ${LOOP_C} -gt 5 ]]; do
>     sleep 0.5
>     # get long container id for exact match
>     CONTAINER_ID=($(curl --silent http://dockerapi:8080/containers/json | jq -r ".[] | {name: .Config.Labels[\"com.docker.compose.service\"], id: .Id}" | jq -rc "select( .name | tostring == \"${1}\") | .id"))
>     # returned id can have multiple elements (if scaled), shuffle for random test
>     CONTAINER_ID=($(printf "%s\n" "${CONTAINER_ID[@]}" | shuf))
>     echo "Found ${CONTAINER_ID[@]}"
>     if [[ ! -z ${CONTAINER_ID} ]]; then
>       for matched_container in "${CONTAINER_ID[@]}"; do
> CONTAINER_IP=$(curl --silent http://dockerapi:8080/containers/${matched_container}/json | jq -r '.NetworkSettings.Networks[].IPAddress')
> echo "${matched_container} has ${CONTAINER_IP}"
>         # grep will do nothing if one of these vars is empty
>         [[ -z ${CONTAINER_IP} ]] && continue
>         [[ -z ${IPV4_NETWORK} ]] && continue
>         # only return ips that are part of our network
>         if ! grep -q ${IPV4_NETWORK} <(echo ${CONTAINER_IP}); then
>           echo "IP did not match network"
>           CONTAINER_IP=
>         else
>           echo "Break"
>           break
>         fi
>       done
>     fi
>     LOOP_C=$((LOOP_C + 1))
>   done
>   [[ ${LOOP_C} -gt 5 ]] && echo 240.0.0.0 || echo ${CONTAINER_IP}
> }
bash-4.3# get_container_ip nginx-mailcow
Found 895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28
895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28 has 172.24.0.5
172.22.1.13
Break
Found 895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28
895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28 has 172.24.0.5
172.22.1.13
Break
Found 895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28
895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28 has 172.24.0.5
172.22.1.13
Break
Found 895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28
895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28 has 172.24.0.5
172.22.1.13
Break
Found 895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28
895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28 has 172.24.0.5
172.22.1.13
Break
240.0.0.0
bash-4.3#

[bplein]

Thank you again. It's a great containerized product, I just want to share the same host with more things (hence the separate proxy).

[andryyy]

Thanks for helping me debugging this!

Can you post the output of this? Again run from within watchdog.

curl --silent http://dockerapi:8080/containers/895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28/json

[bplein]

bash-4.3# curl --silent http://dockerapi:8080/containers/895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28/json
{"Platform": "linux", "State": {"Status": "running", "Pid": 8559, "OOMKilled": false, "Dead": false, "Paused": false, "Running": true, "FinishedAt": "2018-09-09T16:02:57.424548459Z", "Restarting": false, "Error": "", "StartedAt": "2018-09-09T16:02:59.405827884Z", "ExitCode": 0}, "Config": {"Tty": false, "Cmd": ["/bin/sh", "-c", "envsubst < /etc/nginx/conf.d/templates/listen_plain.template > /etc/nginx/conf.d/listen_plain.active && envsubst < /etc/nginx/conf.d/templates/listen_ssl.template > /etc/nginx/conf.d/listen_ssl.active && envsubst < /etc/nginx/conf.d/templates/server_name.template > /etc/nginx/conf.d/server_name.active && envsubst < /etc/nginx/conf.d/templates/sogo.template > /etc/nginx/conf.d/sogo.active && envsubst < /etc/nginx/conf.d/templates/sogo_eas.template > /etc/nginx/conf.d/sogo_eas.active && nginx -qt && until ping phpfpm -c1 > /dev/null; do sleep 1; done && until ping sogo -c1 > /dev/null; do sleep 1; done && until ping redis -c1 > /dev/null; do sleep 1; done && until ping rspamd -c1 > /dev/null; do sleep 1; done && exec nginx -g 'daemon off;'"], "Volumes": {"/etc/nginx/conf.d": {}, "/web": {}, "/dynmaps": {}, "/meta_exporter": {}, "/etc/ssl/mail": {}}, "Domainname": "", "WorkingDir": "", "Image": "nginx:mainline-alpine", "Hostname": "895314535f60", "StdinOnce": false, "StopSignal": "SIGTERM", "Labels": {"com.docker.compose.service": "nginx-mailcow", "com.docker.compose.config-hash": "02751dbb6aa7c51598982ca6ba5ba19dd537a8f146f6ccb2bffb77604c8da0d5", "com.docker.compose.project": "mc", "com.docker.compose.version": "1.22.0", "com.docker.compose.oneoff": "False", "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>", "com.docker.compose.container-number": "1"}, "AttachStdin": false, "User": "", "Env": ["HTTPS_PORT=9443", "HTTP_PORT=9080", "MAILCOW_HOSTNAME=mailhost.plein.email", "IPV4_NETWORK=172.22.1", "TZ=America/Chicago", "VIRTUAL_HOST=plein.email", "VIRTUAL_PORT=9080", "LETSENCRYPT_HOST=plein.email", "LETSENCRYPT_EMAIL=bill@plein.org", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "NGINX_VERSION=1.15.3"], "ExposedPorts": {"9443/tcp": {}, "9080/tcp": {}, "80/tcp": {}}, "OnBuild": null, "AttachStderr": false, "Entrypoint": null, "AttachStdout": false, "OpenStdin": false}, "ResolvConfPath": "/var/lib/docker/containers/895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28/resolv.conf", "HostsPath": "/var/lib/docker/containers/895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28/hosts", "Args": ["-c", "envsubst < /etc/nginx/conf.d/templates/listen_plain.template > /etc/nginx/conf.d/listen_plain.active && envsubst < /etc/nginx/conf.d/templates/listen_ssl.template > /etc/nginx/conf.d/listen_ssl.active && envsubst < /etc/nginx/conf.d/templates/server_name.template > /etc/nginx/conf.d/server_name.active && envsubst < /etc/nginx/conf.d/templates/sogo.template > /etc/nginx/conf.d/sogo.active && envsubst < /etc/nginx/conf.d/templates/sogo_eas.template > /etc/nginx/conf.d/sogo_eas.active && nginx -qt && until ping phpfpm -c1 > /dev/null; do sleep 1; done && until ping sogo -c1 > /dev/null; do sleep 1; done && until ping redis -c1 > /dev/null; do sleep 1; done && until ping rspamd -c1 > /dev/null; do sleep 1; done && exec nginx -g 'daemon off;'"], "Driver": "overlay2", "Path": "/bin/sh", "HostnamePath": "/var/lib/docker/containers/895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28/hostname", "RestartCount": 0, "Name": "/mc_nginx-mailcow_1", "Created": "2018-09-09T04:42:19.019968814Z", "ExecIDs": null, "GraphDriver": {"Data": {"MergedDir": "/var/lib/docker/overlay2/0e1b2061135fd63b671c7342582f03beeb8e42eff74aeb8503ba2fbfd56b43e7/merged", "WorkDir": "/var/lib/docker/overlay2/0e1b2061135fd63b671c7342582f03beeb8e42eff74aeb8503ba2fbfd56b43e7/work", "LowerDir": "/var/lib/docker/overlay2/0e1b2061135fd63b671c7342582f03beeb8e42eff74aeb8503ba2fbfd56b43e7-init/diff:/var/lib/docker/overlay2/58fcda51d93a6bb9f2a9b55d959b846fc1f92f3ad5616d098fded199c3f18abe/diff:/var/lib/docker/overlay2/eab40f426a46980bd4cabf278c2a45bacdcc4f70f1f00aff1623b8b5cc856129/diff:/var/lib/docker/overlay2/1b0189da5d1839864263281affc1c980a9ccd469a24724ec0708ef52642f2761/diff:/var/lib/docker/overlay2/58241d0c82ca23f483b8a491ef3e9cfaa0f553bf2422c6e3f5f2546d262ce138/diff", "UpperDir": "/var/lib/docker/overlay2/0e1b2061135fd63b671c7342582f03beeb8e42eff74aeb8503ba2fbfd56b43e7/diff"}, "Name": "overlay2"}, "Mounts": [{"RW": false, "Propagation": "rprivate", "Destination": "/meta_exporter", "Source": "/fiomirror/mailcow/data/conf/rspamd/meta_exporter", "Mode": "ro", "Type": "bind"}, {"RW": false, "Propagation": "rprivate", "Destination": "/etc/ssl/mail", "Source": "/fiomirror/mailcow/data/assets/ssl", "Mode": "ro", "Type": "bind"}, {"RW": false, "Propagation": "rprivate", "Destination": "/dynmaps", "Source": "/fiomirror/mailcow/data/conf/rspamd/dynmaps", "Mode": "ro", "Type": "bind"}, {"RW": true, "Propagation": "rprivate", "Destination": "/etc/nginx/conf.d", "Source": "/fiomirror/mailcow/data/conf/nginx", "Mode": "rw", "Type": "bind"}, {"RW": false, "Propagation": "rprivate", "Destination": "/web", "Source": "/fiomirror/mailcow/data/web", "Mode": "ro", "Type": "bind"}], "ProcessLabel": "", "NetworkSettings": {"Bridge": "", "Networks": {"nginx-proxy": {"NetworkID": "7157b7968501d2e3f00a52894fdf30a5b932680bc52809aced44423dcd06ed01", "MacAddress": "02:42:ac:18:00:05", "GlobalIPv6PrefixLen": 0, "Links": null, "GlobalIPv6Address": "", "IPv6Gateway": "", "DriverOpts": null, "IPAMConfig": null, "EndpointID": "09c8860efc8773cfa95ae102253e700c4cb377f3e7432e9026dda641a7d6f398", "IPPrefixLen": 16, "IPAddress": "172.24.0.5", "Gateway": "172.24.0.1", "Aliases": ["nginx-mailcow", "895314535f60"]}, "mc_mailcow-network": {"NetworkID": "1b2eb175662d254ee02a97b4ee383de6d58f7ac62d570b7df9f9b5c672569af3", "MacAddress": "02:42:ac:16:01:0d", "GlobalIPv6PrefixLen": 0, "Links": null, "GlobalIPv6Address": "", "IPv6Gateway": "", "DriverOpts": null, "IPAMConfig": null, "EndpointID": "4a12089b40f071c6afaab9bd6fcbf31d09da5c26dbb319b28e00ca7e4a33fde0", "IPPrefixLen": 24, "IPAddress": "172.22.1.13", "Gateway": "172.22.1.1", "Aliases": ["nginx-mailcow", "nginx", "895314535f60"]}}, "SecondaryIPv6Addresses": null, "LinkLocalIPv6Address": "", "HairpinMode": false, "IPv6Gateway": "", "SecondaryIPAddresses": null, "SandboxID": "5f3f74876055c79308fd395065613d176ccb3a2b256fff421b519a7026119b32", "MacAddress": "", "GlobalIPv6Address": "", "Gateway": "", "LinkLocalIPv6PrefixLen": 0, "EndpointID": "", "SandboxKey": "/var/run/docker/netns/5f3f74876055", "GlobalIPv6PrefixLen": 0, "IPPrefixLen": 0, "IPAddress": "", "Ports": {"9443/tcp": [{"HostPort": "9443", "HostIp": "127.0.0.1"}], "9080/tcp": [{"HostPort": "9080", "HostIp": "127.0.0.1"}], "80/tcp": null}}, "AppArmorProfile": "docker-default", "Image": "sha256:336262580e12f5af08e0dff8fa64d20c11b1848343f46d51b9af4f66c28755d3", "LogPath": "/var/lib/docker/containers/895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28/895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28-json.log", "HostConfig": {"CpuPeriod": 0, "MemorySwappiness": null, "ContainerIDFile": "", "KernelMemory": 0, "Memory": 0, "CpuQuota": 0, "UsernsMode": "", "AutoRemove": false, "BlkioDeviceReadIOps": null, "Dns": ["172.22.1.254"], "ExtraHosts": null, "PidsLimit": 0, "DnsSearch": null, "Privileged": false, "IOMaximumIOps": 0, "CpuPercent": 0, "NanoCpus": 0, "Ulimits": null, "CpusetCpus": "", "DiskQuota": 0, "CgroupParent": "", "BlkioWeight": 0, "MemorySwap": 0, "RestartPolicy": {"MaximumRetryCount": 0, "Name": "always"}, "OomScoreAdj": 0, "BlkioDeviceReadBps": null, "VolumeDriver": "", "ReadonlyRootfs": false, "CpuShares": 0, "PublishAllPorts": false, "MemoryReservation": 0, "BlkioWeightDevice": null, "ConsoleSize": [0, 0], "NetworkMode": "mc_mailcow-network", "BlkioDeviceWriteBps": null, "Isolation": "", "GroupAdd": null, "ReadonlyPaths": ["/proc/asound", "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger"], "CpuRealtimeRuntime": 0, "Devices": null, "BlkioDeviceWriteIOps": null, "Binds": ["/fiomirror/mailcow/data/conf/nginx:/etc/nginx/conf.d:rw", "/fiomirror/mailcow/data/web:/web:ro", "/fiomirror/mailcow/data/conf/rspamd/meta_exporter:/meta_exporter:ro", "/fiomirror/mailcow/data/assets/ssl:/etc/ssl/mail:ro", "/fiomirror/mailcow/data/conf/rspamd/dynmaps:/dynmaps:ro"], "CpusetMems": "", "Cgroup": "", "UTSMode": "", "PidMode": "", "Runtime": "runc", "VolumesFrom": [], "CapDrop": null, "DnsOptions": null, "ShmSize": 67108864, "Links": null, "CpuRealtimePeriod": 0, "IpcMode": "shareable", "MaskedPaths": ["/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware"], "PortBindings": {"9443/tcp": [{"HostPort": "9443", "HostIp": "127.0.0.1"}], "9080/tcp": [{"HostPort": "9080", "HostIp": "127.0.0.1"}]}, "SecurityOpt": null, "CapAdd": null, "CpuCount": 0, "DeviceCgroupRules": null, "OomKillDisable": false, "LogConfig": {"Config": {}, "Type": "json-file"}, "IOMaximumBandwidth": 0}, "Id": "895314535f60acd6e581adad1c2d9c61993d48294268f9639de9c17b32d53a28", "MountLabel": ""}
bash-4.3#

[andryyy]

Okay, thanks. :-)

Can you change the image version to 1.21 and pull watchdog-mailcow, up -d?

[bplein]

Looks good!

Health checks are 100% according to "docker logs -f" on the watchdog, and if I exec into a shell on the watch dog, load the get_container_ip function, and test it, I get:

bash-4.3# get_container_ip nginx-mailcow
172.22.1.13

Which is correct.

Vielen Danke, @andryyy !!

[andryyy]

Thanks for testing! :-)