search

stalwartlabs / mail-server

Secure & Modern All-in-One Mail Server (IMAP, JMAP, POP3, SMTP)
https://stalw.art
4.8k stars 194 forks source link

[bug]: Running Stalwart behind Traefik and setting the Traefik network as a trusted network results in Bad Gateway #526

Closed SmollClover closed 3 months ago

SmollClover commented 3 months ago

What happened?

So I've been using Stalwart for a bit now behind the Traefik reverse proxy without any issue, though one problem has been bugging me ever since setting Stalwart up.

In the documentation it says that I should set the reverse proxy network as a trusted network, which I have tried multiple times from setting the IP of the traefik container itself to using the subnetmask of the network traefik and stalwart communicate on, in my case 172.18.0.0/16.
But whenever I do set that and restart the container to make sure the new config is loaded, I receive a bad gateway error on every port of the Stalwart container through traefik.

Here's the portainer stack config I use for the Stalwart server.

services:
  mailserver:
    image: stalwartlabs/mail-server:latest
    container_name: mailserver
    restart: unless-stopped
    hostname: mail.CENSORED
    networks:
      - proxy
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - data:/opt/stalwart-mail
      - certs:/data/certs:ro
    labels:
      - traefik.enable=true

      - traefik.http.routers.mailserver.rule=Host(`mail.CENSORED`) || Host(`autodiscover.CENSORED`) || Host(`autoconfig.CENSORED`) || Host(`mta-sts.CENSORED`)
      - traefik.http.routers.mailserver.entrypoints=https
      - traefik.http.routers.mailserver.service=mailserver
      - traefik.http.services.mailserver.loadbalancer.server.port=443

      - traefik.tcp.routers.smtp.rule=HostSNI(`*`)
      - traefik.tcp.routers.smtp.entrypoints=smtp
      - traefik.tcp.routers.smtp.service=smtp
      - traefik.tcp.services.smtp.loadbalancer.server.port=25

      - traefik.tcp.routers.smtps.rule=HostSNI(`*`)
      - traefik.tcp.routers.smtps.entrypoints=smtps
      - traefik.tcp.routers.smtps.tls.passthrough=true
      - traefik.tcp.routers.smtps.service=smtps
      - traefik.tcp.services.smtps.loadbalancer.server.port=465

      - traefik.tcp.routers.imaps.rule=HostSNI(`*`)
      - traefik.tcp.routers.imaps.entrypoints=imaps
      - traefik.tcp.routers.imaps.tls.passthrough=true
      - traefik.tcp.routers.imaps.service=imaps
      - traefik.tcp.services.imaps.loadbalancer.server.port=993

volumes:
  data:
  certs:
    name: traefik_certs
    external: true

networks:
  proxy:
    name: traefik_proxy
    external: true

I'm also doing TLS passthrough because Stalwart didn't like it when Traefik was handling the TLS connection. Therefore I linked the certs created by traefik into the Stalwart container so it can use them for the connection and handle all of the TLS stuff.

Also, I disabled https on port 443 of Stalwart since that is handled through Traefik itself.

How can we reproduce the problem?

Version

v0.8.x

What database are you using?

RocksDB

What blob storage are you using?

RocksDB

Where is your directory located?

Internal

What operating system are you using?

Docker

Relevant log output

2024-06-09T15:37:44.374143Z TRACE common::listener::listen: Failed to accept proxied TCP connection context="io" event="error" instance="https" protocol=Http reason=invalid proxy header
2024-06-09T15:37:44.849083Z TRACE common::listener::listen: Failed to accept proxied TCP connection context="io" event="error" instance="https" protocol=Http reason=invalid proxy header

Code of Conduct

mdecimus commented 3 months ago

Hi,

I'm closing this as it is a configuration issue. The error you're seeing is because Traefik is not sending the proxy protocol headers as the error message explains. You need to make sure that Traefik is sending the proxy headers to Stalwart.

SmollClover commented 3 months ago

Well, I tried everything, including the exact configuration used on a completely fresh server detailed in the Docs, with no luck. I even tried setting every IP in Traefik as a trusted one for the Proxy Protocol, but still the same result.

I'll stick with my less-than-ideal solution then since I, for the life of me, can't figure out where the problem lies.

mdecimus commented 3 months ago

You can check this discussion for help. I've never used Traefik but other people have it working according to the comments.

SmollClover commented 3 months ago

You can check this discussion for help. I've never used Traefik but other people have it working according to the comments.

Yes, thank you so much! That discussion and the gist inside it was a real help.
After setting the proxy network override in just the SMTP and IMAP listeners and configuring traefik to trust the mailserver IP it now works with the proxy protocol set properly.
Oh, and since I had a few problems, I set the exact IP of my traefik container and not the subnet mask since that didn't seem to work.

mdecimus commented 3 months ago

Yes, thank you so much! That discussion and the gist inside it was a real help.

Could you share the configuration that worked for you so I add it to the documentation? Thanks!

SmollClover commented 3 months ago

Sure, here are the two different compose files, one for Traefik and one for Stalwart, with their relevant configs too.

In the Stalwart config I only included important stuff such as the listeners actually used.
The most important part here is to only set the proxy protocol override on the IMAPTLS and SUBMISSIONS listeners since when set on the HTTP or HTTPS listener it will result in the error I used to have.

Also of note is that I had to include the Subnetmask and the IP of the traefik container in the override, just using one of them didn't seem to work for me. And of course in the traefik config the IP of the Stalwart container should be trusted for it to work.

That is my setup, sorry if it's a bit much and a bit messy, though I am happy to explain parts of it when needed!
Oh, and traefik-certs-dumper is just there to dump the certificates created by traefik which it stores in the acme.json to actual certificate files so that Stalwart can use them.

Traefik Compose ```compose services: traefik: image: traefik:v3.0 container_name: traefik restart: always networks: proxy: ipv4_address: 172.19.0.2 ports: - 80:80 - 443:443 - 25:25 - 465:465 - 993:993 volumes: - /etc/localtime:/etc/localtime:ro - /etc/traefik:/etc/traefik - acme:/etc/certs - /var/run/docker.sock:/var/run/docker.sock:ro traefik-certs-dumper: image: ghcr.io/kereis/traefik-certs-dumper:latest container_name: traefik-certs-dumper restart: unless-stopped depends_on: - traefik volumes: - /etc/localtime:/etc/localtime:ro - acme:/traefik:ro - certs:/output volumes: acme: certs: networks: proxy: ```
Traefik Config ```yml global: checkNewVersion: true sendAnonymousUsage: false certificatesResolvers: letsencrypt: acme: keyType: EC256 dnsChallenge: provider: cloudflare email: REDACTED storage: /etc/certs/acme.json entryPoints: http: address: :80 http3: {} http: redirections: entryPoint: to: https scheme: https https: address: :443 http3: {} http: tls: certResolver: letsencrypt smtp: address: :25 proxyProtocol: trustedIPs: - 172.19.0.2 - 172.19.0.5 smtps: address: :465 proxyProtocol: trustedIPs: - 172.19.0.2 - 172.19.0.5 imaps: address: :993 proxyProtocol: trustedIPs: - 172.19.0.2 - 172.19.0.5 tls: options: default: minVersion: VersionTLS12 providers: docker: exposedByDefault: false ```
Stalwart Compose ```compose services: mailserver: image: stalwartlabs/mail-server:latest container_name: mailserver restart: unless-stopped hostname: mail.example.com networks: proxy: ipv4_address: 172.19.0.5 volumes: - /etc/localtime:/etc/localtime:ro - data:/opt/stalwart-mail - certs:/data/certs:ro labels: - traefik.enable=true - traefik.http.routers.mailserver.rule=Host(`mail.example.com`) || Host(`autodiscover.example.com`) || Host(`autoconfig.example.com`) || Host(`mta-sts.example.com`) - traefik.http.routers.mailserver.entrypoints=https - traefik.http.routers.mailserver.service=mailserver - traefik.http.services.mailserver.loadbalancer.server.port=8080 - traefik.tcp.routers.smtp.rule=HostSNI(`*`) - traefik.tcp.routers.smtp.entrypoints=smtp - traefik.tcp.routers.smtp.service=smtp - traefik.tcp.services.smtp.loadbalancer.server.port=25 - traefik.tcp.services.smtp.loadbalancer.proxyProtocol.version=2 - traefik.tcp.routers.jmap.rule=HostSNI(`*`) - traefik.tcp.routers.jmap.tls.passthrough=true - traefik.tcp.routers.jmap.entrypoints=https - traefik.tcp.routers.jmap.service=jmap - traefik.tcp.services.jmap.loadbalancer.server.port=443 - traefik.tcp.services.jmap.loadbalancer.proxyProtocol.version=2 - traefik.tcp.routers.smtps.rule=HostSNI(`*`) - traefik.tcp.routers.smtps.tls.passthrough=true - traefik.tcp.routers.smtps.entrypoints=smtps - traefik.tcp.routers.smtps.service=smtps - traefik.tcp.services.smtps.loadbalancer.server.port=465 - traefik.tcp.services.smtps.loadbalancer.proxyProtocol.version=2 - traefik.tcp.routers.imaps.rule=HostSNI(`*`) - traefik.tcp.routers.imaps.tls.passthrough=true - traefik.tcp.routers.imaps.entrypoints=imaps - traefik.tcp.routers.imaps.service=imaps - traefik.tcp.services.imaps.loadbalancer.server.port=993 - traefik.tcp.services.imaps.loadbalancer.proxyProtocol.version=2 volumes: data: certs: name: traefik_certs external: true networks: proxy: name: traefik_proxy external: true ```
Stalwart Config ```toml certificate.default.cert = "%{file:/data/certs/mail.example.com/cert.pem}%" certificate.default.default = true certificate.default.private-key = "%{file:/data/certs/mail.example.com/key.pem}%" lookup.default.hostname = "mail.example.com" server.http.hsts = true server.http.permissive-cors = false server.http.url = "protocol + '://' + key_get('default', 'hostname') + ':' + local_port" server.http.use-x-forwarded = true server.listener.http.bind = "[::]:8080" server.listener.http.protocol = "http" server.listener.https.bind = "[::]:443" server.listener.https.protocol = "http" server.listener.https.tls.implicit = true server.listener.imaptls.bind = "[::]:993" server.listener.imaptls.protocol = "imap" server.listener.imaptls.proxy.override = true server.listener.imaptls.proxy.trusted-networks.0 = "172.19.0.2" server.listener.imaptls.proxy.trusted-networks.1 = "172.19.0.0/16" server.listener.imaptls.tls.implicit = true server.listener.smtp.bind = "[::]:25" server.listener.smtp.protocol = "smtp" server.listener.smtp.proxy.override = true server.listener.smtp.proxy.trusted-networks.0 = "172.19.0.2" server.listener.smtp.proxy.trusted-networks.1 = "172.19.0.0/16" server.listener.submissions.bind = "[::]:465" server.listener.submissions.protocol = "smtp" server.listener.submissions.proxy.override = true server.listener.submissions.proxy.trusted-networks.0 = "172.19.0.2" server.listener.submissions.proxy.trusted-networks.1 = "172.19.0.0/16" server.listener.submissions.tls.implicit = true ```
mdecimus commented 3 months ago

Thanks for the config @SmollClover !

It is important that you also enable the proxy protcol on the SMTP port, otherwise the wrong IP address is going to be used for SPF validation which will cause messages to be sent to the spam folder.

SmollClover commented 3 months ago

It is important that you also enable the proxy protcol on the SMTP port, otherwise the wrong IP address is going to be used for SPF validation which will cause messages to be sent to the spam folder.

Oh, you're right. I don't know how I forgot that. Thanks!