Problems with certificates while running Mailu under nginx-proxy/letsencrypt-companion

[flayks]

I found Mailu a good solution to host my emails as I'm trying to rebuild my server with Docker, but I get several issues about getting it running properly, especially with the certificates. Experimented with TLS_FLAVOR in my .env file, set to cert or mail but the admin only works with notls.

I run my own nginx container for accessing the admin at mails.server.site.com, but I always get a 502 error or a 403 with mail or cert option.

Also, when I run the containers, I get this error from the mailu-front:

mailu-front | Missing cert or key file, disabling TLS mailu-front | 2018/03/30 10:20:37 [notice] 7#7: using the "epoll" event method mailu-front | 2018/03/30 10:20:37 [notice] 7#7: nginx/1.12.2 mailu-front | 2018/03/30 10:20:37 [notice] 7#7: OS: Linux 4.9.20-std-1 mailu-front | 2018/03/30 10:20:37 [notice] 7#7: getrlimit(RLIMIT_NOFILE): 1048576:1048576 mailu-front | 2018/03/30 10:20:37 [notice] 7#7: start worker processes mailu-front | 2018/03/30 10:20:37 [notice] 7#7: start worker process 9 mailu-front | 2018/03/30 10:20:37 [notice] 7#7: start worker process 10 mailu-front | 2018/03/30 10:20:37 [notice] 7#7: start worker process 11 mailu-front | 2018/03/30 10:20:37 [notice] 7#7: start worker process 12

The certificates generated by docker-letsencrypt-nginx-proxy-companion are actually present on /etc/certs/mail.site.com and working, but when I try to symlink anything it doesn't work and I get the same message for the front container.

Did I missed something? I looked at all the issues here about nginx-proxy, reverse proxies or certificates, read the docs carefully, but nothing sounds to work in my situation.

My configs

.env

###################################
# Common configuration variables
###################################

# Set this to the path where Mailu data and configuration is stored
ROOT=/data/tools/mailu

# Mailu version to run (1.0, 1.1, etc. or master)
VERSION=1.5

# Set to a randomly generated 16 bytes string
SECRET_KEY=key

# Address where listening ports should bind
BIND_ADDRESS4=127.0.0.1
BIND_ADDRESS6=::1

# Main mail domain
DOMAIN=site.com

# Hostnames for this server, separated with comas
HOSTNAMES=mail.site.com

# Postmaster local part (will append the main mail domain)
POSTMASTER=admin

# Choose how secure connections will behave (value: letsencrypt, cert, notls, mail)
# TLS_FLAVOR=letsencrypt
# TLS_FLAVOR=notls
TLS_FLAVOR=cert

# Authentication rate limit (per source IP address)
AUTH_RATELIMIT=10/minute;1000/hour

# Opt-out of statistics, replace with "True" to opt out
DISABLE_STATISTICS=True

###################################
# Optional features
###################################

# Expose the admin interface (value: true, false)
ADMIN=true

# Choose which webmail to run if any (values: roundcube, rainloop, none)
WEBMAIL=none

# Dav server implementation (value: radicale, none)
WEBDAV=none

# Antivirus solution (value: clamav, none)
ANTIVIRUS=clamav
[...]

docker-compose.yml

version: "3.6"

networks:
  reverse-proxy:
    external: true
  mail_network:
    driver: bridge

services:
  nginx:
    container_name: mailu-nginx
    image: nginx
    restart: always
    volumes:
      - ./mailu.conf:/etc/nginx/conf.d/default.conf
      - /data/logs/mailu:/logs
    environment:
      VIRTUAL_PORT: 80
      VIRTUAL_HOST: mails.server.site.com
      LETSENCRYPT_HOST: mails.server.site.com
      LETSENCRYPT_EMAIL: email
    networks:
      reverse-proxy:
      mail_network:

  front:
    container_name: mailu-front
    image: mailu/nginx:$VERSION
    restart: always
    env_file: .env
    ports:
      - "$BIND_ADDRESS4:8080:80"
      - "$BIND_ADDRESS4:8443:443"
      - "$BIND_ADDRESS4:110:110"
      - "$BIND_ADDRESS4:143:143"
      - "$BIND_ADDRESS4:993:993"
      - "$BIND_ADDRESS4:995:995"
      - "$BIND_ADDRESS4:25:25"
      - "$BIND_ADDRESS4:465:465"
      - "$BIND_ADDRESS4:587:587"
      - "$BIND_ADDRESS6:8080:80"
      - "$BIND_ADDRESS6:8443:443"
      - "$BIND_ADDRESS6:110:110"
      - "$BIND_ADDRESS6:143:143"
      - "$BIND_ADDRESS6:993:993"
      - "$BIND_ADDRESS6:995:995"
      - "$BIND_ADDRESS6:25:25"
      - "$BIND_ADDRESS6:465:465"
      - "$BIND_ADDRESS6:587:587"
    volumes:
      - "$ROOT/certs:/certs"
    networks:
      mail_network:

[...the rest is untouched]

nginx custom mailu.conf

upstream www-mailu {
    server front;
}

server {
    server_name mails.server.site.com;
    access_log /logs/access.log;
    error_log /logs/error.log;

    location / {
        proxy_pass http://www-mailu/admin;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
        proxy_cache_valid 200 302 1m;
    }
}

[flayks]

Some progress there. I finally found a way to get the certs working with the TLS_FLAVOR=mail (using cp /etc/certs/mail.site.com/fullchaim.pem /data/tools/mailu/certs/cert.pem also for key.pem – not ideal but works hey, a cron script could be good for that, or a symlink?), so I don't get the "mailu-front | Missing cert or key file, disabling TLS" error.

However, I'm still trying to access the admin with a custom nginx reverse proxy via my nginx container in the same file, but it still gives me 502 Bad Gateway, I can't really find a way to get this working.

server {
    access_log /logs/access.log;
    error_log /logs/error.log;

    location / {
        proxy_pass https://localhost:8443/admin;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
    }
}

[kaiyou]

The logic is pretty simple: unless the admin specifically wants TLS disabled (in which case notls should be set), we do not want Mailu to be used at all if TLS is not working properly. This is why we disable TLS completely and display 403 and deny SMTP/IMAP access when the cert and key are not in place.

To answer your latest question, could you provide us with the part of your docker-compose.yml that binds port 8443 for the front to be available to your reverse proxy? When using mail, the front container does not listen over port 443 and does not have TLS enforced for HTTP requests.

[Greek64]

@flayks Your nginx-proxy/letsencrypt-companion configuration seems faulty. The VIRTUAL_PORT,VIRTUAL_HOST,LETSENCRYPT_HOST, and LETSENCRYPT_EMAIL Enviromental Variables should be defined for the front service/container, and not for the nginx container itself. see the documentation here.

Basically, the VIRTUAL_HOST Variable contains the host for which the certificates are generated. Every request to the nginx container containing that hostname is then redirected to the container that has defined that VIRTUAL_HOST Variable (To the port defined in VIRTUAL_PORT).

Also note that in your setup it makes no sense to bind the ports 80 and 443 of the front service/container. As explained above, all http requests are redirected from the nginx service/container to your front service/container in port 80 (as defined by VIRTUAL_PORT). A binding of those ports would effectively "bypass" you reverse proxy, as the mailu pages would be accesible from "mails.server.site.com:80", "mails.server.site.com:443", as well as "yoursystemaddress:8080", "yoursystemaddress:8443". (yoursystemaddress is effectively your $BIND_ADDRESS4 address)

Now, concerning the certificates: In your "cert" directory of your letsencrypt companion you should have following structure:

mail.site.com [Directory]
mail.site.com.chain.pem [symlink]
mail.site.com.crt [symlink]
mail.site.com.dhparam.pem [symlink]
mail.site.com.key [symlink]

The mail.site.com.crt is the cert.pem and mail.site.com.key is the key.pem needed by the "front" container. I suggest "linking" them through docker-compose itself using:

volumes:
   -type: bind
     source: /etc/certs/mail.site.com.crt
     target: /certs/cert.pem
     read_only: true

This ensures that the container fails if the certificates do not exist (and prevents the ominous directories that docker creates when the files do not exist)

[ghost]

has this been resolved? im experiencing something quite similar with traefik/letsencrypt, almost working, however i cant get to admin, i can get to webmail fine .... i've tried the notls and mail in TLS_FLAVOR= and seemingly it just plainly fails

logs -f mailu_admin_1
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
[2018-04-16 17:33:55 +0000] [23] [INFO] Starting gunicorn 19.7.1
[2018-04-16 17:33:55 +0000] [23] [INFO] Listening at: http://0.0.0.0:80 (23)
[2018-04-16 17:33:55 +0000] [23] [INFO] Using worker: sync
[2018-04-16 17:33:55 +0000] [32] [INFO] Booting worker with pid: 32
[2018-04-16 17:33:55 +0000] [33] [INFO] Booting worker with pid: 33
[2018-04-16 17:33:55 +0000] [34] [INFO] Booting worker with pid: 34
[2018-04-16 17:33:55 +0000] [35] [INFO] Booting worker with pid: 35
130.25.212.186 - - [16/Apr/2018:17:35:59 +0000] "GET / HTTP/1.1" 404 233 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"
130.25.212.186 - - [16/Apr/2018:17:36:03 +0000] "GET / HTTP/1.1" 404 233 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"
130.25.212.186 - - [16/Apr/2018:17:36:04 +0000] "GET / HTTP/1.1" 404 233 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"
130.25.212.186 - - [16/Apr/2018:17:36:09 +0000] "GET /admin/ HTTP/1.1" 404 233 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"

docker-compose.yml:

version: '2'

services:
  traefik:
    image: traefik:1.5-alpine
    container_name: traefik
    restart: always
    ports:
      - "80:80"
      - "443:443"
    labels:
      - traefik.enable=true
      - traefik.frontend.rule=Host:mail.${DOMAIN}
      - traefik.port=8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - $ROOT/traefik/traefik.toml:/traefik.toml:ro
      - $ROOT/traefik/acme:/etc/traefik/acme

  front:
    image: mailu/nginx:$VERSION
    restart: always
    env_file: .env
    ports:
    - "$BIND_ADDRESS4:8000:80"
    - "$BIND_ADDRESS4:4443:443"
    - "$BIND_ADDRESS4:110:110"
    - "$BIND_ADDRESS4:143:143"
    - "$BIND_ADDRESS4:993:993"
    - "$BIND_ADDRESS4:995:995"
    - "$BIND_ADDRESS4:25:25"
    - "$BIND_ADDRESS4:465:465"
    - "$BIND_ADDRESS4:587:587"
    volumes:
      - "$ROOT/certs:/certs"

  redis:
    image: redis:alpine
    restart: always
    volumes:
      - "$ROOT/redis:/data"

  imap:
    image: mailu/dovecot:$VERSION
    restart: always
    env_file: .env
    volumes:
      - "$ROOT/data:/data"
      - "$ROOT/mail:/mail"
      - "$ROOT/overrides:/overrides"
    depends_on:
      - front

  smtp:
    image: mailu/postfix:$VERSION
    restart: always
    env_file: .env
    volumes:
      - "$ROOT/data:/data"
      - "$ROOT/overrides:/overrides"
    depends_on:
      - front

  antispam:
    image: mailu/rspamd:$VERSION
    restart: always
    env_file: .env
    volumes:
      - "$ROOT/filter:/var/lib/rspamd"
      - "$ROOT/dkim:/dkim"
      - "$ROOT/overrides/rspamd:/etc/rspamd/override.d"
    depends_on:
      - front

  antivirus:
    image: mailu/$ANTIVIRUS:$VERSION
    restart: always
    env_file: .env
    volumes:
      - "$ROOT/filter:/data"

  webdav:
    image: mailu/$WEBDAV:$VERSION
    restart: always
    env_file: .env
    volumes:
      - "$ROOT/dav:/data"

  admin:
    image: mailu/admin:$VERSION
    restart: always
    env_file: .env
    labels:
      - traefik.enable=true
      - traefik.frontend.rule=Host:admin.${DOMAIN}
      - traefik.port=80
      - traefik.frontend.entryPoints=https
    volumes:
      - "$ROOT/data:/data"
      - "$ROOT/dkim:/dkim"
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on:
      - redis

  webmail:
    image: "mailu/$WEBMAIL:$VERSION"
    restart: always
    env_file: .env
    labels:
      - traefik.enable=true
      - traefik.frontend.rule=Host:webmail.${DOMAIN}
      - traefik.port=80
      - traefik.frontend.entryPoints=https
    volumes:
      - "$ROOT/webmail:/data"

  fetchmail:
    image: mailu/fetchmail:$VERSION
    restart: always
    env_file: .env
    volumes:
      - "$ROOT/data:/data"

[Nuxij]

My setup is a little simpler, I have a Caddyfile like this:

email.example.com {
        proxy / front:80 {
                transparent
        }
}

And then I have it routing to the front container internal to the docker engine.

I've set TLS_FLAVOR=mail and linked in the certs that caddy generates for me. Webmail all works fine this way.

The problem comes when I hit /admin, it appears to be loading and then tries to redirect me to http://front/admin/ui/. I get the feeling it just needs to look at X-Forwarded-For or something.

Not sure if this is directly related to the issue OP has but seems pretty similar.

[kaiyou]

@Joeasaurus it is probably not directly related. Yours is indeed a missing X-Forwarded-For, or else you could setup Caddy to use a specific Host header when making its backend query. The weird par is that transparent should do the job by setting header_upstream properly, could you debug the Caddy request?

[Raphy]

Hey !

I'm experiencing a similar issue.

I'd like to use Mailu in my dedicated server that hosts several applications using Docker with main Traeffik. I'd like to setup Mailu to be compatible with this main Traefik container.

I've set the labels to mailu/front service :

# ...
  front:
    image: mailu/nginx:$VERSION
    restart: always
    env_file: .env
    labels:
        traefik.enable: true
        traefik.backend: mail-domain-tld-http
        traefik.frontend.rule: Host:mail.domain.tld
        traefik.docker.network: traefik
        traefik.port: 80
    ports:
#        - "80:80" # HTTP
#        - "443:443" # HTTPS
        - "110:110" # POP3
        - "143:143" # IMAP
        - "993:993" # IMAPS
        - "995:995" # POP3S
        - "25:25" # SMTP
        - "465:465" # SMTPS
        - "587:587" # SMTP
    volumes:
        - "$ROOT/certs:/certs"
    networks:
        - default
        - traefik # My Traefik network
# ...

Even if I can hit https://mail.domain.tld/webmail/, I have a 403 error.

I can send emails from my mail client, but I can't receive :

imap_1_f70722795ed4 | Nov 28 22:21:54 lmtp(22): Error: SSL context initialization failed, disabling SSL: Can't load SSL certificate: There is no valid PEM certificate.

My main Traefik logs outputs something strange, it obtains the mail.domain.tld certificate but it also obtains for an unset domain admin.mailu :

traefik_1_fb02f7411e84 | time="2018-11-28T21:29:37Z" level=info msg="legolog: [INFO] [admin.mailu] acme: Obtaining bundled SAN certificate"
traefik_1_fb02f7411e84 | time="2018-11-28T21:29:37Z" level=info msg="legolog: [INFO] [mail.domain.tld] acme: Obtaining bundled SAN certificate"

I have tried to let TLS_FLAVOR=mail and TLS_FLAVOR=letsencrypt and both is not working and do the same

Do you have any idea how I can make the things work ?

EDIT : I've found a solution after testing a lot of things and is close to what was said earlier. I've extracted the certificated as files from acme.json of Traefik container with https://github.com/DanielHuisman/traefik-certificate-extractor Then, I've copied the certificates in $ROOT/certs directory (cert.pem and privatekey.pem as key.pem). Set TLS_FLAVOR=mail to use these certificates with mail and everything is working well.

I will use the certificate extractor as daemon to auto refresh the certificates obtained by Traefik. I will try to create symlinks, don't know if it will work with Docker. And don't know once the certificate is refreshed, the mail server will automatically refresh them.

Any ideas ?

[muhlemmer]

Correlated: #565

[Sporesirius]

@outbackdingo are you fixed your issue with traefik and mailu?

[muhlemmer]

@Sporesirius, We have a new documentation chapter on Traefik. https://mailu.io/1.6/reverse.html#traefik-as-reverse-proxy

Also, the demo server is running behind Traefik. Please see the Mailu/infra repository with the working config as an example.

[ghost]

No I havent.... I believe i need to maybe pay someone to get this installation finished, i have a working mailu system, just not behind traefik yet... any takers??

On Thu, Feb 14, 2019 at 4:28 PM Sporesirius notifications@github.com wrote:

@outbackdingo https://github.com/outbackdingo are you fixed your issue with traefik and mailu?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Mailu/Mailu/issues/430#issuecomment-463555992, or mute the thread https://github.com/notifications/unsubscribe-auth/ABXFTujcgdUA1URxnKmY3udI3M_EczvKks5vNSwxgaJpZM4TBkMc .

[kaiyou]

Any news on this? anyone with feedback on using caddy/traefik?

[kaiyou]

I will close this, feel free to reopen if you wish.

[N-ickJones]

I had issues with mailu 1.7 and using TLS_FLAVOR=letsencrypt and TLS_FLAVOR=mail-letsencrypt...

I'm running nginx as my reverse proxy.

I looked inside the 'front' with docker-compose exec front bash and viewed their letsencrypt.py.

Essentially what the letsencrypt is running is the following:

certbot -n --agree-tos -d os.environ["HOSTNAMES"] -m os.environ["POSTMASTER"]@os.environ["DOMAIN"] certonly --standalone --cert-name mailu --preferred-challenges http --http-01-port 8008 --keep-until-expiring --rsa-key-size 4096 --config-dir /certs/letsencrypt --post-hook /config.py

I used this to generate my certificates as follows:

certbot -n --agree-tos -d mydomain.com -m admin@mydomain.com certonly --standalone --cert-name mailu --preferred-challenges http --http-01-port 8008 --keep-until-expiring --rsa-key-size 4096

Then added them to the /mailu/certs as specified in their documentation... cp /etc/letsencrypt/live/mailu/privkey.pem /mailu/certs/key.pem cp /etc/letsencrypt/live/mailu/fullchain.pem /mailu/certs/cert.pem

Now TLS_Flavor=mail is working

I haven't figured out what's causing inability for certbot to generate my certificates inside the docker container.

[kswtch]

Is there a reason why softlinks in /mailu/certs/ don't work?

ln -s /etc/letsencrypt/live/foobar.com/fullchain.pem /mailu/certs/cert.pem This does not work.

cp /etc/letsencrypt/live/foobar.com/fullchain.pem /mailu/certs/cert.pem This does work.

[eev0779]

Also not working this path in docker-compose.yml

front:
    ...
    volumes:
    - "/etc/letsencrypt/live/somesite:/certs:ro"

With overridden TLS_CERT_FILENAME and TLS_KEYPAIR_FILENAME. I will copy too. Sad.

[JeppeX]

@kswtch I think the reason why softlinks don't work is due to docker isolation. @eev0779 I did a bind mount and it worked:

      - type: bind
        source: /etc/letsencrypt/live/[REDACTED]/privkey.pem
        target: /certs/key.pem
        read_only: true
      - type: bind
        source: /etc/letsencrypt/live/[REDACTED]/fullchain.pem
        target: /certs/cert.pem
        read_only: true

[EdGetMeLink]

hi all, i'm running mailu 1.9 and since way to long now the letsencrypt certs do not refrech automaticaly. It worked in the begining, but stopped working at some point in time. Now i have to take down the container and bring it up again to have fresh new certs from letsencrypt. could some one give me a hint where to start?