search

docker-mailserver / docker-mailserver

Production-ready fullstack but simple mail server (SMTP, IMAP, LDAP, Antispam, Antivirus, etc.) running inside a container.
https://docker-mailserver.github.io/docker-mailserver/latest/
MIT License
14.84k stars 1.84k forks source link

feature request: Configure "PROXY Protocol" for Dovecot + Postfix #3866

Open cfis opened 10 months ago

cfis commented 10 months ago

Context

As discussed on the kubernetes page, PROXY protocol support is quite important for Kubernetes.

See the docker-mailserver-helm ports section.

Description

Is there any interest in migrating this code to DMS? Would it be useful with Docker too?

What the chart does is create new ports in the 10,000 range that support the proxy protocol for both Postfix and Dovecot.

docker-mailserver-helm related config:

Could this be added to DMS? Either always on or controlled by an ENV like ENABLE_PROXY_PROTOCOL_SUPPORT?

Alternatives

Leave PROXY protocol support to just kubernetes.

Applicable Users

All users

Are you going to implement it?

Yes, because I know the probability of someone else doing it is low and I can learn from it.

What are you going to contribute?

See already implemented code in the Helm chart.

polarathene commented 10 months ago

I don't have bandwidth available to provide much feedback on this right now (EDIT: Ok.. semi-tackled it). But glancing over at the config you referenced there should be easier ways to support that.

Looking at our PROXY Protocol docs page, where does it suggest anything about these extra ports like you've configured? (10587 / 10465, no relevant matches found in our docs)

Issues wise, there's only your recent helm chart update request mentioning them too: https://github.com/docker-mailserver/docker-mailserver/issues/3825 (and the associated helm project README port section)

You document many of the ports we support with 10xxx variants, but not port 25, which you don't appear to have covered in that config file? That treats any inbound connection to that port with the issue you're trying to prevent from losing the remote IP? If someone adjusted PERMIT_DOCKER beyond our default none to a configuration relaxed enough to trust the incorrect internal IP of the proxy container, you'd have an open relay? ⚠️

All I have to go by is this snippet in our docs:

image

These are internal services, but why is that an issue? I'd need to reproduce this to verify but it sounds odd to me.

Neither of those have any relevance to port 993, so that documented concern is bizarre. Whomever contributed it was probably mistaken. Best I can get with git blame is a commit from @wernerfred but was tied to the Github Wiki migration PR where I see no context 🤷‍♂️ (EDIT: Found this comment which suggests it may have been part of the original Github Wiki)

References:

While that HAProxy blog post does put port 25 on the reverse proxy, and adjusts Postfix to use a different port internally, this isn't applicable for containers where each container has a separate IP with it's own set of ports. Perhaps some misunderstandings have carried over from that? I see no indication for such a requirement.

The docs for Dovecot settings is a bit different though. They seem more strict implying that the Proxy Protocol header must be received and that it must be from a client in the configured trusted networks for haproxy connections.

We do have issues reporting internal services that don't route through the reverse proxy are incompatible with Postfix as well:

Another public port (optional) is 4190 for ManageSieve (Dovecot has implicit service config at /etc/dovecot/conf.d/20-managesieve.conf). That should probably be proxied appropriately too?

I came across this mailcow commit that applied the same config ports for Postfix / Dovecot, but it lacks a PR reference for any other context.

If the ports are not used internally by anything however, then it should be fine to just toggle proxy protocol support on the existing service ports. Nothing should really change then in usage, provided all traffic external of that container is going through the reverse proxy service to reach DMS.

One last thing, you've defined a set of trusted networks for proxy are those accurate? (If I understand correctly, it's a configurable setting?)

For Docker at least, the networks it supports are from a pool of large subnets:

image

Thus if you had two containers in the same network, rather than a 192.168.0.0/16 it's a smaller 192.168.0.0/20 subnet.

This sort of trust btw is a bit risky when IPv6 is enabled, at least for Docker where the daemon has userland-proxy: true and an IPv4 only container an IPv6 connection is proxied into the trusted private subnet. Hopefully not an issue with a reverse proxy as the entrypoint instead where that sort of trust shouldn't be the case? (although I'm not quite sure what to make of Traefik with it's Proxy Protocol setting trustedIPs, I assume that's for proxies in front of Traefik... and we're actually interested in the services config instead).

Oh and your doc links should also prefer to use the version (v13.3 instead of latest / edge, doc version tags are only v<MAJOR>.<MINOR> no patch).

polarathene commented 10 months ago

Regarding actual support being implemented. If the concerns above can be addressed first for if the additional ports are actually needed / beneficial, that'd be appreciated.

Implementing the support shouldn't be too troublesome, but I'd like some test coverage which is where some friction will likely be introduced for the contributor. Presently we don't run tests with compose.yaml, which adds more friction there too 😓

If other maintainers aren't fussed about test coverage, but see it as a worthwhile contribution beyond guidance offered in docs being updated, I guess I'd be ok with a reproduction I can use locally verify the contribution proposed is necessary.

cfis commented 10 months ago

The helm chart was already using port 10993,

Port 25 - yes - it should be added.

For the text you pasted, yes, I'm not sure if that is true or not - thanks for looking into that.

But here is the real use case and why this is important.

I did test this a fair bit on my cluster:

The trusted networks are easily configurable (just override in a custom values.yaml). And in fact, I do override them because I need to trust the IPs of the pods inside my cluster). The default values were already part of the Helm chart and are the three IPV4 non-routable ranges. Meaning they likely are (must be?) computers on your network and therefore should be trustworthy. So I thought it was a reasonable default and kept it.

I think it is worth adding PROXY support to DMS if there are cases, like in Kubernetes, where you need it to figure out the client ip address (seems like it - https://github.com/moby/moby/issues/15086). If not, then it can stay in the Helm chart.

If it stay in the Helm chart, would love to know if there are better ways to configure it, especially the postfix setup. I used user-patches.sh because I didn't see another way of doing it based on the docs. But of course I could easily have missed a better alternative. And for dovecot, it was unclear to me if I should redefine the base ports or not. The chart was already doing that, so I kept it.

polarathene commented 10 months ago

The helm chart was already using port 10993,

Yes, I don't maintain the helm chart repo as I have no real k8s experience, likewise with the docs. I have very little familiarity with ProxyProtocol too. I have however pointed out very clearly that the justification in our docs page is very odd/wrong regarding port 10993.

The other link you provided is for a non-container setup (only one service can bind a specific port per IP, thus separate ports is necessary). They put the regular ports to connect to only via the reverse proxy configured (HAProxy) and only provide the alternative ports in Postfix and Dovecot. You're proposing to have both available from these services on the DMS container though.

The LinuxBabe article makes the real ports available via the "frontend" network, which then go through the ProxyProtocol "backend" ports. This doesn't help justify your use-case for other local services connecting to the real ports on DMS directly though, since that's not what is configured there either.

Port 25 - yes - it should be added.

  • My assumption is that port 25 is most likely blocked anyway, but it should be added.
  • As for an open relay, well, you can have that problem whether or not we add a port 10025 (you could of course not proxy traffic and send it to port 25 - that is out of the control of the Helm chart).

Port 25 has two roles, inbound and outbound connections.

If you don't proxy inbound port 25 for the same reasons you're proxying the others, restrictions are bypassed. You permit receiving mail without DKIM/DMARC/SPF checks for example which is quite bad.

The issue with an open relay that I was raising was with trusted networks (such as Postfix mynetworks setting). If the remote IP information is lost from the proxy service being involved (which is what we're addressing here with ProxyProtocol), and you've allowed Postfix to trust the same subnet (eg: PERMIT_DOCKER=connected-networks) then you'll allow anyone to relay mail through your DMS instance by sending on port 25 without authentication. Even though the intention was only from clients on your trusted private network.

NOTE: Don't forget the mentioned concern with ManageSieve Dovecot service also needing to be configured for proxy.

But here is the real use case and why this is important.

  • A Kubernetes cluster can have other applications running that expect to send/receive email. In my case, I have NextCloud, Gitlab, Bitwarden, etc all running along with DMS.
  • They all want to send and receive emails, and thus need access to "normal" ports 465/587 and 993 without Proxy support enabled (because that breaks them).

You can have the same kind of deployment with Docker Compose, but how you connect to the DMS container can change by configuring DNS and networks.

Take Roundcube as a separate container in your cluster. How would you deploy this container when all users connecting through it in their browsers delegates the auth process through the Roundcube container to the DMS container?

This is another known scenario where the single IP source of the Roundcube container is problematic, should a user fail to login enough times that Fail2Ban triggers and bans that IP, all users are now banned via that service. Roundcube does support ProxyProtocol, but this would only work when going through the compatible port (either directly to DMS or through the reverse proxy with ProxyProtocol support on both inbound (from RoundCube) + outbound (to DMS) traffic).

When that sort of problem described with Roundcube is not a concern, the only reason you'd need the non-proxied ports is for convenience of a direct connection between the two services without the proxy involved?

I did test this a fair bit on my cluster:

  • If your load balancer/ingress turns on PROXY support for incoming connections, then PROXY support must also be enabled on the receiving dovecot/postfix ports or the communication fails.
  • Alternatively, if you turn on PROXY support in Postfix or Dovecot but do not PROXY incoming connections then they will also fail.
  • Thus you can't just have just one port, say 993 for IMAPs. You need two ports, one with PROXY support and one without.

I stated in my previous comment that the ProxyProtocol feature in both Postfix and Dovecot enforce that requirement AFAIK, so I'm aware 👍

The trusted networks are easily configurable (just override in a custom values.yaml). And in fact, I do override them because I need to trust the IPs of the pods inside my cluster). The default values were already part of the Helm chart and are the three IPV4 non-routable ranges. Meaning they likely are (must be?) computers on your network and therefore should be trustworthy. So I thought it was a reasonable default and kept it.

It's a very wide trust. DMS defaults to not trusting these because of security risks that introduces (especially with some networking gotchas/bugs I know of with Docker). I'd kind of suggest the same approach if you care about security, then this is something the user should opt-in for trust. You'll find that reverse proxies like Caddy and Traefik explicitly need opt-in for trusted networks, they don't just assume the private ranges are safe to trust.

I can have two VMs with their own private internal networks, but the two VMs can reach each other via a shared network. While they normally couldn't access the private internal networks of the other VM, such as those managed by Docker, Docker can configure the network via a kernel tunable which unintentionally allows the other VM to route traffic through the shared network to the private network.

This is enabled by default IIRC, but not applicable to connections from public IPs, still it's a vulnerability concern for some deployments like at a company where an employee with a system on the network could compromise. I verified it in private cloud networks too, so all it'd take is one server compromised by a malicious user to access the containers of another that were assumed private.

With Docker at least, my point was that the subnet a container belongs in is notably narrower, not sure how that applies to k8s networking. I might trust my PC at home, but I don't trust the devices of others on the same network.

I think it is worth adding PROXY support to DMS if there are cases, like in Kubernetes, where you need it to figure out the client ip address (seems like it - moby/moby#15086). If not, then it can stay in the Helm chart.

See my comment in the issue you referenced: https://github.com/moby/moby/issues/15086#issuecomment-1535767303

If the reverse proxy is a container as well, it wouldn't matter since the real client IP is already lost in the scenario many users would experience from IPv6 / userland-proxy. Docker specific AFAIK. May have been some other cases where the networking rules introduced other issues there and something like ProxyProtocol may have helped sure.

I'm not saying it isn't worth adding the feature you're requesting. I'm just questioning the approach, especially with duplicating ports vs sending all ingress through the proxy instead of direct to DMS?

If it stay in the Helm chart, would love to know if there are better ways to configure it, especially the postfix setup. I used user-patches.sh because I didn't see another way of doing it based on the docs. But of course I could easily have missed a better alternative. And for dovecot, it was unclear to me if I should redefine the base ports or not. The chart was already doing that, so I kept it.

I don't have the time to put much thought into this beyond the feedback given here.

If you were just toggling proxy support instead of adding new ports, you'd have it handled with a few commands:

For your current approach with postfix, I believe there is a postconf command to retrieve just the config settings of an existing service like submission/unix, which may be formatted as it is in the file. Then you can just modify the submission part to the port (10587) and append it to master.cf file. After just use postconf -P to add/change the main.cf overrides (-o).

Dovecot config, you can find the line service imap-login and append the snippets you have to include after that line. I think sed can do that.

cfis commented 10 months ago

Agreed on adding ports 25 and 4190 (ManageSieve).

Retrieving (yes, not receiving) / sending email in the cluster...Roundcube is a good example and is in fact another application I have running in my cluster. Roundcube talks directly to the docker-mailserver service (and thus pod) on the relevant ports (465/587 or 993). There is no use of the proxy protocol because this is internal cluster traffic.

The proxy protocol only applies to traffic incoming to the cluster. In this case, a user would be using port 443 from their local browser to talk to the roundcube service (no proxy there) and roundcube in turn talks to docker-mailserver to retrieve/send email (no proxy there either).

Proxy support comes into play if you are using an external client, such as Thunderbird, and it talks to docker mailserver. It will send traffic on ports 465/587/993 to the cluster, and those ports are proxied and directed to ports 10465/10587/10993 which have been configured to understand the proxy protocol.

the only reason you'd need the non-proxied ports is for convenience of a direct connection between the two services without the proxy involved?

Not only is that a big convenience, it is essential since most apps know nothing about the proxy protocol. Gitlab? NextCloud? Photo sharing apps (that want to send notifications)? Etc?

What I want to know is why you can't have all services that need IMAP connect through 993 on the reverse proxy?

Because that would mean traffic in the cluster would have to go outside the cluster, do a hair-pin turn, and then come back in the cluster via the proxy. I'm not even sure it would work once you have more than one host (but I'm far from a K8 networking expert).

Doesn't k8s support deploying containers across multiple hosts for scaling reasons? How's the networking handled then?

Every K8 pod on any host can directly to any other pod on any other host. That is all internal cluster traffic. That is the main goal of K8 networking. So that means any app that wants to retrieve/send email just talks directly to the docker mailserver service no matter where in the cluster it might be deployed.

Having two sets of ports enables both exterior cluster clients and interior clients. Having just one set of ports breaks one of those two use cases.

polarathene commented 10 months ago

Roundcube talks directly to the docker-mailserver service (and thus pod) on the relevant ports (465/587 or 993). There is no use of the proxy protocol because this is internal cluster traffic.

Ok, but if someone were to deploy Roundcube + DMS, how does this avoid the issue with the IP of all remote users authenticating through Roundcube being the same? Isn't ProxyProtocol required?

The proxy protocol only applies to traffic incoming to the cluster. In this case, a user would be using port 443 from their local browser to talk to the roundcube service (no proxy there) and roundcube in turn talks to docker-mailserver to retrieve/send email (no proxy there either).

My understanding was:

I'm only aware of the issue here being when Fail2Ban is enabled within DMS, since Roundcube delegates the user auth to the users DMS account. However with the source IP always appearing the same from Roundcube regardless of user authenticating, this is a compatibility issue. Doesn't the ProxyProtocol fix that?

For other services it's not a concern, since they should all be submitting mail through a single account for that service specifically, not tied to any individual user of the service.

Proxy support comes into play if you are using an external client, such as Thunderbird, and it talks to docker mailserver. It will send traffic on ports 465/587/993 to the cluster, and those ports are proxied and directed to ports 10465/10587/10993 which have been configured to understand the proxy protocol.

Yes I understand this.

It should be just as viable to only use 465/587/993 though. Postfix/Dovecot only expecting ProxyProtocol and any connection should be through the reverse proxy that speaks ProxyProtocol. Why is that not viable?

the only reason you'd need the non-proxied ports is for convenience of a direct connection between the two services without the proxy involved?

Not only is that a big convenience, it is essential since most apps know nothing about the proxy protocol. Gitlab? NextCloud? Photo sharing apps (that want to send notifications)? Etc?

I'm sorry, why do they need to know ProxyProtocol to connect through the reverse proxy?

Take the NextCloud SMTP config docs for example.

For any external client, like via ThunderBird for example this is already configured the same way to connect to DMS via DNS resolving the IP and connecting to that port. Why is this different in your internal services? It should all resolve to the reverse proxy container IP + port.

Doing so will then route seamlessly to DMS with ProxyProtocol for everyone, and only the reverse proxy and DMS Postfix/Dovecot services need to care about that. In other circumstances like with Roundcube, you can do a little more configuration to ensure ProxyProtocol is used when connecting to the reverse proxy (Traefik makes this optional at least, so no conflicting compatibility concerns there).

To me these are separate concerns:

The latter is relevant for an alternative relay transport for outbound 465 connections which likewise requires a setting change that'd make it incompatible with outbound STARTTLS (port 25 / 587). So long as the service exists in master.cf already, it's trivial with our postfix-master.cf. The only missing feature there is to duplicate an existing service and change the service port/socket.

What I want to know is why you can't have all services that need IMAP connect through 993 on the reverse proxy?

Because that would mean traffic in the cluster would have to go outside the cluster, do a hair-pin turn, and then come back in the cluster via the proxy. I'm not even sure it would work once you have more than one host (but I'm far from a K8 networking expert).

I don't have the k8s expertise to chime in the networking situation there.

With Docker AFAIK the traffic within the same subnet will resolve directly to the reverse proxy regardless of internal vs external DNS. It has hairpin rules in iptables established related to this IIRC.

But in the sense of local DNS resolution, I don't see why this would be any different when we're just talking about resolving to a different container IP? Perhaps you're relying on hostnames already configured for you implicitly in /etc/hosts or k8s has a similar internal DNS like Docker does for a similar feature.

In Docker I can just add a host directly:

$ docker run --rm -it --add-host 'reverse-proxy:10.20.30.40' --hostname some-container alpine cat /etc/hosts

127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
10.20.30.40     reverse-proxy
172.17.0.3      some-container

How you go about assigning IPs is up to you, I have a specific network for a group of containers to belong in that can access each other, if needed I can choose that subnet and explicitly assign which IP belongs to a container. That's all just the same to me like ENV config is.

Surely k8s has something similar to that. So really, why can't you configure to send a different hostname that maps to the reverse-proxy which recognizes something like dms-internal.example.com just like it would mail.example.com, directing traffic to the same location?

# Quick example by spinning up a container:
# This hostname added has no relevance anywhere else, DMS is configured as mail.origin.test, IP is assigned dynamically
$ docker run --rm -it --add-host dms-internal.example.test:172.18.0.4 --network my-dms-network alpine:edge

# Add swaks to send a test mail:
$ apk add swaks --repository=https://dl-cdn.alpinelinux.org/alpine/edge/testing

# Send that test mail:
$ swaks --server dms-internal.example.test:587 --from alpine@wherever.test --to john.doe@origin.test --auth-user jane.doe@origin.test --auth-pass secret

# Success!

Simple as that. Should be no different if that host/server configured pointed to a reverse proxy container, so long as it has an entry configured to route for that?

Having two sets of ports enables both exterior cluster clients and interior clients. Having just one set of ports breaks one of those two use cases.

I'm not convinced.

All I can see is a convenience in routing traffic directly, which may complicate troubleshooting when that difference introduces some subtle bug.... and then we the maintainers get burdened by issue reports.

We've already established config mistakes in the helm chart and our proxy protocol docs page, and how these have been relied upon rather than better understanding the why (or that reasoning being invalid), which doesn't bring much confidence that maintainers will be able to troubleshoot issues related to the feature (just see the past issue history on the subject).

I'm open towards a PR(s) that implements the needed support, but I don't see the need to complicate our maintenance with a feature that duplicates ports redundantly for convenience.

Reference from alternative projects:

Do you have some other references where this need for duplicate ports is supported elsewhere (with context of k8s deployments / containers)? Do you have examples for anything else that isn't mail related?

I'm sorry about the friction, just trying to avoid the XY problem. DMS has taken contributions in the past for features that weren't given much review consideration, and we ended up with messes like the current relay host feature and LDAP support. I just want to get a clear understanding of what is needed and the value that actually offers to the bulk of our users.

cfis commented 10 months ago

Ok, but if someone were to deploy Roundcube + DMS, how does this avoid the issue with the IP of all remote users authenticating through Roundcube being the same? Isn't ProxyProtocol required?

Traffic to roundcube goes is via HTTP(S) via a kubernetes ingress. That uses a different mechanism, called externalTrafficPolicy, to preserve the source ip address. See https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip.

It should be just as viable to only use 465/587/993 though. Postfix/Dovecot only expecting ProxyProtocol and any connection should be through the reverse proxy that speaks ProxyProtocol. Why is that not viable?

Because clients inside the cluster are not using the Proxy Protocol. And their traffic does not go through the Proxy because they are communicating with each other entirely inside the cluster.

Server is relevant to DNS. Have this point to the reverse proxy.

Which requires traffic to leave the cluster to then come back into the cluster.

Kubernetes networking is different than Docker. Kubernetes has a layer of indirection (services), its own DNS server in which it tracks services, and then pods. Pods get IP addresses from the same subnet, you can't assign specific ip addresses to specific pods. The whole point of this setup is a flat address space to make it easy for pods to talk to each other through services, no matter what host they are running on. Maybe this helps - https://kubernetes.io/docs/concepts/services-networking/

In my view, forcing traffic outside of the cluster is working against the way Kubernetes is designed. I assume you could probably make it work but, at least to me, seems like a clearly a worse solution than having two sets of mail ports.

cfis commented 10 months ago

Another thought - I have been mostly assuming you are deploying docker-mailserver to a bare metal K8 cluster that is running on a home server or EC2 instance. But if you are running in a managed K8 cluster like EKS, then traffic into the cluster is coming in via an external load balancer controlled by the hosting provider (ie, AWS, Azure, GCP).

Now, this is not my area of expertise, so I may be totally wrong on this. But my reading of this:

https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout

Is that if you have an app (roundcube) that makes a network call outside the cluster that goes to a load balancer that routes back to the same node (say where docker-mailserver happens to be) then it will timeout if you want to preserve the client ip address. So you would have to disable that, which then of course makes using the Proxy protocol pointless.

Which is my main worry - sending traffic outside the cluster to come back in seems likely to introduce unexpected networking issues that will differ depending on your specific Kubernetes setup and provider.

georglauterbach commented 10 months ago

Having two sets of ports enables both exterior cluster clients and interior clients. Having just one set of ports breaks one of those two use cases.

I'm not convinced.

All I can see is a convenience in routing traffic directly, which may complicate troubleshooting when that difference introduces some subtle bug.... and then we the maintainers get burdened by issue reports.

Interior clients are actually something that is very common, and something that happens in my cluster as well. The reasons are mostly concerned with effectiveness:

  1. Why send traffic outside the cluster if there is no reason to do so at all?
  2. Moreover, you have complete control over this traffic by keeping it internal.
  3. Finally, the default routing configuration would have to be adjusted to route the traffic outside the cluster only for it to come back?

From my PoV, what @cfis pointed out concerning K8s network is correct.

It should be just as viable to only use 465/587/993 though. Postfix/Dovecot only expecting ProxyProtocol and any connection should be through the reverse proxy that speaks ProxyProtocol. Why is that not viable?

Example: I have GitLab running, which sends e-mails for CI jobs. GitLab uses my public mail server address mail.example.com and submissions to authenticate as a dedicated user and send e-mails to my main account. The cluster DNS (in my case, coreDNS) knows DMS' FQDN via hostAliases, and the Container Network Interface (CNI) (in my case, Cilium) and the cluster load balancer (in my case, MetalLB) can route the traffic before it leaves the host. One could equally make it work by using the cluster internal DNS-name (<SERVICE NAME>.<NAMESPACE>.svc.cluster.local, which coreDNS knows how to resolve).

It's the same for RoundCube, which accesses ports 465 and 993 inside the cluster as well.

Now, the question that was posed reads, "Why not run the traffic through the ingress and use the ProxyProtocol?" This is not easily doable (how would one route the traffic for which the destination IP is already known and on the same host to an intermediate that is not on the same host, just so we can handle it as external traffic)? And we need the traffic to be external in order for it to go through the ingress.

What seems to make sense to me now:


┌───────┐
│       │  <── cluster-internal traffic port 465/587 (not proxied) (e.g., for GitLab)
│       │  <── cluster-internal traffic port 993/995 (not proxied) (e.g., for RoundCube)
│       │
│  DMS  │       ┌───────────┐
│       │       │           │  <── cluster-external traffic port 25 (proxied)
│       │  <──  │  Ingress  │  <── cluster-external traffic port 465/587 (proxied)
│       │       │           │  <── cluster-external traffic port 993/995 (proxied)
│       │       └───────────┘
└───────┘
polarathene commented 10 months ago

Traffic to roundcube goes is via HTTP(S) via a kubernetes ingress. That uses a different mechanism, called externalTrafficPolicy, to preserve the source ip address. See https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip.

Ok, but this is docker-mailserver not k8s-mailserver 😅 It is a valid concern when not using k8s at least.

Your description is only conveying that the client IP is preserved to roundcube though, which wasn't my question. It was that roundcube continues to preserve that client IP information when connecting to DMS on behalf of that user, which AFAIK was why they added ProxyProtocol support?

What does DMS log for the client IP when you authenticate through Roundcube? Is DMS getting the client IP or only the IP of the Roundcube container?

It should be just as viable to only use 465/587/993 though. Postfix/Dovecot only expecting ProxyProtocol and any connection should be through the reverse proxy that speaks ProxyProtocol. Why is that not viable?

Because clients inside the cluster are not using the Proxy Protocol. And their traffic does not go through the Proxy because they are communicating with each other entirely inside the cluster.

Are you saying it's not possible for k8s to route through the proxy container? Because I just showed that you can do it with just Docker, which was between two containers in the same network. It should be no different with a reverse proxy container in the middle, I see no reason for that to differ in k8s.

I lack the k8s experience to say otherwise, but I'm skeptical of such a limitation. If you believe it really isn't possible, I can ping some maintainers with k8s experience for their input.

Server is relevant to DNS. Have this point to the reverse proxy.

Which requires traffic to leave the cluster to then come back into the cluster.

No? If it doesn't do that with your expectation of container to container, why do you think that is different when all that is changing is the IP it connects to is a different containers IP within the same subnet?

Kubernetes networking is different than Docker. Kubernetes has a layer of indirection (services), its own DNS server in which it tracks services, and then pods.

Docker Compose has services connected to networks Docker creates and manages with it's own internal DNS service for resolving links between containers/services. Not familiar with pods (I know Docker Swarm deals with replicas of a container for scaling, but I've never bothered with Swarm).

Pods get IP addresses from the same subnet, you can't assign specific ip addresses to specific pods. The whole point of this setup is a flat address space to make it easy for pods to talk to each other through services, no matter what host they are running on.

Ok.. so in Docker Compose, or just plain Docker, containers are assigned an IP within the subnet they belong to. Assigning an IP to a container explicitly is opt-in.

Pods from what I've read represents a group of IPs that are load balanced for a DNS record the cluster manages, unless you're using a headless service or something along those lines 🤷‍♂️

I had a read over a variety of the docs networking pages, and some other sites, but it seems k8s adds friction for what I proposed.

https://kubernetes.io/docs/concepts/services-networking/

In my view, forcing traffic outside of the cluster is working against the way Kubernetes is designed. I assume you could probably make it work but, at least to me, seems like a clearly a worse solution than having two sets of mail ports.

Traffic shouldn't have to leave the cluster, this isn't what I meant with Docker Compose, just routing to the container that handles reverse proxy and letting that resolve it. Apparently this is quite different distinction/separation from the cluster with ingress role in k8s 🤔

I'd suggest a 2nd instance of the proxy for internal usage if that were the case, but that doesn't help with k8s having less flexibility in the internal routing.

I did come across this on StackOverflow where one of the answers to get a network alias for a service is:

apiVersion: v1
kind: Service
metadata:
  name: "my-alias"
spec:
  type: ExternalName
  externalName: "other-service.my-namespace.svc.cluster.local"

Which lets you access other-service via my-alias by defining another service. So presumably if you had a reverse proxy container like Traefik in k8s as a service, you'd be able to assign it an alias like that for DMS, and have Traefik route that to DMS via it's service name.

Another suggestion is bundling the containers into a single pod where they can reach each other via localhost. Which I understand isn't likely any better for you, I'm used to having a separate container with caddy or nginx in front of a web application as it handles a different set of responsibilities, but in k8s that sounds like it needs to be stripped of some features in favor of delegating that to ingress, including the site address matching to just a port instead.

polarathene commented 10 months ago

The reasons are mostly concerned with effectiveness:

  1. Why send traffic outside the cluster if there is no reason to do so at all?
  2. Moreover, you have complete control over this traffic by keeping it internal.
  3. Finally, the default routing configuration would have to be adjusted to route the traffic outside the cluster only for it to come back?

I am not familiar enough with k8s, see the example I gave with a simple alpine container routing to a different container to send mail via swaks. That separate container could be a reverse proxy and is all in the same subnet, all internal traffic.

I was only suggesting a common flow for all traffic (internal or external) via the reverse proxy service. That's apparently not simple to achieve in k8s.

Not that we would, but if DMS bundled Traefik... then enabling that would proxy all the ports with ProxyProtocol, and it wouldn't matter what you connect (ingress or internal traffic) as that'd be compatible for both scenarios. You'd just need to configure Traefik to trust the ingress IP/subnet for ProxyProtocol. I have heard of side-car containers with k8s, but I suppose it can't be used in that manner 🤷‍♂️

Regardless, I still see it as two separate features to implement that compliment each other for those that need the workaround (k8s, similar to it's need for hostname workaround).

georglauterbach commented 10 months ago

After all of this, I think my current setup that just uses a load-balancer to open a port directly is the most straight-forward 😂

polarathene commented 10 months ago

@georglauterbach since you have k8s and CoreDNS experience, can you have a quick skim over this article and let me know if that's still valid?

Just trying to grok core blocker with routing with k8s that requires the separate duplicated ports for internal use.

I am not familiar with Traefik and Kubernetes well enough, but curious if their Gateway API support allows for similar internal routing or proxying?

For @cfis to handle duplicate ports as an alternate workaround, there is a little bit more change than below, but I have a simpler Postfix user-patches.sh for you than the one in your helm chart config.

That'll be part of a revised docs page PR 👍

Reproduction with Docker Compose

services:
  reverse-proxy:
    image: docker.io/traefik:2.10
    networks:
      default:
        ipv4_address: 172.16.42.10
    command:
      - "--providers.docker"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=dms-test-network"
      # Postfix:
      - "--entryPoints.smtp.address=:25"
      - "--entryPoints.smtp-submission.address=:587"
      - "--entryPoints.smtp-submissions.address=:465"
      # Dovecot:
      - "--entryPoints.imap.address=:143"
      - "--entryPoints.imaps.address=:993"
      - "--entryPoints.sieve.address=:4190"
    # CAUTION: Production usage should configure socket access better (see Traefik docs)
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - "25:25"
      - "587:587"
      - "465:465"
      - "143:143"
      - "993:993"

  dms:
    image: mailserver/docker-mailserver:edge
    container_name: dms-mta
    hostname: mail.example.test
    environment:
      SSL_TYPE: manual
      SSL_CERT_PATH: /tmp/tls/cert.pem
      SSL_KEY_PATH: /tmp/tls/key.pem
    # NOTE: `configs` used for convenience of reproduction,
    # usually these would be split into separate files and used as volume mounts
    configs:
      - source: dms-accounts
        target: /tmp/docker-mailserver/postfix-accounts.cf
      - source: overrides-dovecot
        target: /tmp/docker-mailserver/dovecot.cf
      - source: overrides-postfix-master
        target: /tmp/docker-mailserver/postfix-master.cf
      - source: tls-cert
        target: /tmp/tls/cert.pem
      - source: tls-key
        target: /tmp/tls/key.pem
    labels:
      - "traefik.enable=true"

      # Plain-text connection to DMS, which allows for STARTTLS negotiation:
      - "traefik.tcp.routers.smtp.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.smtp.tls=false"
      - "traefik.tcp.routers.smtp.entrypoints=smtp"
      - "traefik.tcp.routers.smtp.service=smtp"
      - "traefik.tcp.services.smtp.loadbalancer.server.port=25"
      - "traefik.tcp.services.smtp.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.smtp-submission.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.smtp-submission.tls=false"
      - "traefik.tcp.routers.smtp-submission.entrypoints=smtp-submission"
      - "traefik.tcp.routers.smtp-submission.service=smtp-submission"
      - "traefik.tcp.services.smtp-submission.loadbalancer.server.port=587"
      - "traefik.tcp.services.smtp-submission.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.imap.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.imap.tls=false"
      - "traefik.tcp.routers.imap.entrypoints=imap"
      - "traefik.tcp.routers.imap.service=imap"
      - "traefik.tcp.services.imap.loadbalancer.server.port=143"
      - "traefik.tcp.services.imap.loadbalancer.proxyProtocol.version=2"

      # Sieve is also STARTTLS according to RFC 5804
      # NOTE: Dovecot doesn't appear to config explicit/implicit TLS via setting,
      # both would normally be accepted but this isn't possible here with Traefik
      # requiring the TCP router to either have TLS disabled or do TLS passthrough.
      - "traefik.tcp.routers.sieve.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.sieve.tls=false"
      - "traefik.tcp.routers.sieve.entrypoints=sieve"
      - "traefik.tcp.routers.sieve.service=sieve"
      - "traefik.tcp.services.sieve.loadbalancer.server.port=4190"
      - "traefik.tcp.services.sieve.loadbalancer.proxyProtocol.version=2"

      # These implicit TLS ports additionally configure TLS passthrough
      - "traefik.tcp.routers.smtp-submissions.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.smtp-submissions.tls.passthrough=true"
      - "traefik.tcp.routers.smtp-submissions.entrypoints=smtp-submissions"
      - "traefik.tcp.routers.smtp-submissions.service=smtp-submissions"
      - "traefik.tcp.services.smtp-submissions.loadbalancer.server.port=465"
      - "traefik.tcp.services.smtp-submissions.loadbalancer.proxyProtocol.version=2"

      - "traefik.tcp.routers.imaps.rule=HostSNI(`*`)"
      - "traefik.tcp.routers.imaps.tls.passthrough=true"
      - "traefik.tcp.routers.imaps.entrypoints=imaps"
      - "traefik.tcp.routers.imaps.service=imaps"
      - "traefik.tcp.services.imaps.loadbalancer.server.port=993"
      - "traefik.tcp.services.imaps.loadbalancer.proxyProtocol.version=2"

# An explicit subnet was only configured to assign an explicit IP to the reverse-proxy container
# Otherwise the Dovecot haproxy_trusted_networks would need to specify a very wide CIDR range
# (172.16.0.0/12 => 172.16.0.0 to 172.31.255.255)
networks:
  default:
    name: dms-test-network
    ipam:
      config:
        - subnet: "172.16.42.0/24"

configs:
  # DMS requires an account to complete setup
  # NOTE: All accounts are configured with the same `secret` password.
  dms-accounts:
    # NOTE: `$` needed to be repeated to escape it, which opts out of the `compose.yaml` variable interpolation feature.
    content: |
      john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
      jane.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.

  # The ECDSA test suite cert (self-signed):
  tls-cert:
    content: |
      -----BEGIN CERTIFICATE-----
      MIIBvDCCAWGgAwIBAgIRAOpRrmDNAnhLvxuk42f/s10wCgYIKoZIzj0EAwIwIDEe
      MBwGA1UEAxMVU21hbGxzdGVwIHNlbGYtc2lnbmVkMB4XDTIxMDEwMTAwMDAwMFoX
      DTMxMDEwMTAwMDAwMFowIDEeMBwGA1UEAxMVU21hbGxzdGVwIHNlbGYtc2lnbmVk
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsfexEnUXqHKaOTHv8GGy9AyIxgWy
      EvKZ4KyBeExylOlSj+nBe7AVg5AGMNLAa2ZjMRRUKIAdiW6kLN1ZF1+mPqN8MHow
      DgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAd
      BgNVHQ4EFgQUFJ+66xRSnywTsunxd9pb9uIdVL0wKgYDVR0RBCMwIYIMZXhhbXBs
      ZS50ZXN0ghFtYWlsLmV4YW1wbGUudGVzdDAKBggqhkjOPQQDAgNJADBGAiEA+HI9
      kH7bnnpPbYD7+txCQ+Lcj2rsGMWv4eoD/WZ4ogECIQD3hliBF/V0WxTID5Piu7jp
      kEfA97Fggtm0Gvz6ZvpIXA==
      -----END CERTIFICATE-----

  tls-key:
    content: |
      -----BEGIN EC PRIVATE KEY-----
      MHcCAQEEIO4nGpEVFeDjKsUKBBumdJxg0tOx/BEasG6G/denFif1oAoGCCqGSM49
      AwEHoUQDQgAEsfexEnUXqHKaOTHv8GGy9AyIxgWyEvKZ4KyBeExylOlSj+nBe7AV
      g5AGMNLAa2ZjMRRUKIAdiW6kLN1ZF1+mPg==
      -----END EC PRIVATE KEY-----

  overrides-dovecot:
    content: |
      haproxy_trusted_networks=172.16.42.10
      # CIDR range is optional, it's better to be more specific about the reverse-proxy IP when possible
      #haproxy_trusted_networks=172.16.42.0/24

      # Override Dovecot services to use ProxyProtocol:
      service imap-login {
        inet_listener imaps {
          haproxy = yes
        }

        inet_listener imap {
          haproxy = yes
        }
      }

      service managesieve-login {
        inet_listener sieve {
          haproxy = yes
        }
      }

  overrides-postfix-master:
    content: |
      smtp/inet/postscreen_upstream_proxy_protocol=haproxy
      submission/inet/smtpd_upstream_proxy_protocol=haproxy
      submissions/inet/smtpd_upstream_proxy_protocol=haproxy

Reproduction commands with above config:

# Bring up the two containers configured for using PROXY protocol:
$ docker compose up --force-recreate

# Add a third container to that network, with an extra DNS entry to /etc/hosts
# Technically the DNS name is optional as swaks can take an IP directly to --server arg
$ docker run --rm -it \
  --add-host dms-internal.example.test:172.16.42.10 \
  --network dms-test-network \
  alpine:edge

# Add swaks with TLS support:
$ apk add swaks --repository=https://dl-cdn.alpinelinux.org/alpine/edge/testing
$ apk add perl-net-ssleay

# Send a test mail indirectly to DMS with TLS successfully through the reverse-proxy:
$ swaks --server dms-internal.example.test --port 587 -tls \
  --from hello@alpine.test --to jane.doe@example.test \
  --auth-user john.doe@example.test --auth-password secret

# DMS logs (172.16.42.3 is the Alpine container):
postfix/submission/smtpd[796]: connect from df2cc066c982.dms-test-network[172.16.42.3]
postfix/submission/smtpd[796]: Anonymous TLS connection established from df2cc066c982.dms-test-network[172.16.42.3]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256
postfix/submission/smtpd[796]: AD5C3189D4: client=df2cc066c982.dms-test-network[172.16.42.3], sasl_method=LOGIN, sasl_username=john.doe@example.test
postfix/qmgr[726]: AD5C3189D4: from=<hello@alpine.test>, size=251, nrcpt=1 (queue active)
dovecot: lmtp(jane.doe@example.test)<924><ofLbLcvJwWWcAwAAUi6ngw>: sieve: msgid=<20240206055523.000012@df2cc066c982>: stored mail into mailbox 'INBOX'
postfix/lmtp[923]: AD5C3189D4: to=<jane.doe@example.test>, relay=mail.example.test[/var/run/dovecot/lmtp], delay=0.07, delays=0.05/0/0.01/0.01, dsn=2.0.0, status=sent (250 2.0.0 <jane.doe@example.test> ofLbLcvJwWWcAwAAUi6ngw Saved)

Thunderbird client works fine too, although since it's on the same system (Windows via WSL2), the IP used is via the Docker network gateway IP (due to iptables rules for host connections via localhost on published ports), it should be correct if the connection was external.

georglauterbach commented 10 months ago

The coreDNS configuration that you linked seems to be up to date. With this, I can also see how all ports can be PROXY-Protocol-enabled.

polarathene commented 9 months ago

With this, I can also see how all ports can be PROXY-Protocol-enabled.

Awesome, @cfis does that work for you as well? Or is there an inconvenience with the CoreDNS rewrite approach?

cfis commented 9 months ago

CoreDNS - you likely need to modify it anyway if you are using a certificate. For example, in my case:

rewrite name mail.savagexi.com docker-mailserver.mail.svc.cluster.local

Where mail.savagexi.com matches the name of the TLS certificate I am using (in my case *.savagexi.com). And docker-mailserver.mail.svc.cluster.local is the full DNS hostname inside of Kubernetes.

Now lets say you create a container inside the cluster that has a proxy and point mail.savagexi.com at it. That would work fine, but then how does the proxy talk to docker mailserver? It can't use mail.savagexi.com because you already pointed that at the proxy. And you can't use docker-mailserver.mail.svc.cluster.local because TLS verification will fail. And you can't point it at an IP address because that will change everytime docker-mailserver is restarted. You could use a second server name, like mail2.savagexi.com, if you have a wildcard certificate. But if you don't, now you need a second certificate.

So yes, you can make it work. And I think it is less issue-prone than bouncing traffic outside the cluster. But it still seems like a lot of work to avoid opening extra standard mail ports that aren't even exposed outside the cluster.

cfis commented 9 months ago

FYI thanks for that docker compose file, it is quite illuminating to see a full example, especially the config section.

polarathene commented 9 months ago

That would work fine, but then how does the proxy talk to docker mailserver? It can't use mail.savagexi.com because you already pointed that at the proxy. And you can't use docker-mailserver.mail.svc.cluster.local because TLS verification will fail.

I'm not familiar with the differences with k8s here, you'll need to explain that to me.

But it still seems like a lot of work to avoid opening extra standard mail ports that aren't even exposed outside the cluster.

The extra work is to preserve the client IP.

If you don't care about that from the internal container traffic (which may have been proxied already from the reverse-proxy like with Roundcube), then add the duplicate port config I guess?

I was just curious about why it wasn't possible to resolve without duplicating ports in k8s by routing all traffic to mail.example.com through the reverse-proxy.

As @georglauterbach shared, and my own reproduction with compose in the next comment shows, CoreDNS rewrite works as intended? 🤷‍♂️

https://coredns.io/plugins/rewrite/ rewrite performs internal message rewriting. Rewrites are invisible to the client.

polarathene commented 9 months ago

Reproduction

Quite a bit more config, but below we extend the prior reproduction with the following:

Configuring for extra proxy ports (especially with the referenced helm config) doesn't seem much simpler/terser than the CoreDNS with a Corefile, at least in my opinion 😅

Observation: It doesn't seem that you can use HostSNI to route to separate containers by FQDN if that were needed, as anything but * then requires Traefik to handle the TLS cert it seems. (EDIT: This was due to swaks / openssl not sending a server name for SNI matching by default, thus empty SNI ''_)

Config Additions

compose.dns.yaml (or merge with earlier compose.yaml example)

services:
  dns:
    image: coredns/coredns:1.11.1
    container_name: mail-dns
    networks:
      default:
        ipv4_address: 172.16.42.9
    configs:
      - source: dns-config
        target: /Corefile
      - source: dns-zone
        target: /zones/internal.test.zone
    # This is just a workaround with the CoreDNS image,
    # which introduced an undocumented breaking change with WORK_DIR (Dockerfile)
    working_dir: /

# This CoreDNS example config can be copy/pasted as a separate compose config,
# If doing so instead of merging, join the same network this way:
networks:
  default:
    name: dms-test-network
    external: true

configs:
  dns-config:
    content: |
      . {
        # Rewrite requests for mail.example.test to rp.internal.test
        #rewrite name mail.example.test rp.internal.test

        # If wanting to also resolve `example.test`, or any of it's subdomains use regex match type:
        # NOTE: `answer auto` for ANSWER rewriting is implicit for `name exact` (`exact` is default type for `name`)
        rewrite name regex ^(.*\.)?example\.test.$$ rp.internal.test answer auto

        # DNS lookup for dms.internal.test (if rewritten from above, it'd also perform this lookup).
        # This represents the internal DNS for the private network with the DMS container
        # it is no different to matching the k8s cluster equivalent DNS name.
        file /zones/internal.test.zone internal.test

        # Alternatively, forward to Docker embedded DNS, matching any hostname configured:
        # NOTE: If using file and match is successful there, it is resolved and forward rule is skipped.
        forward internal.test 127.0.0.11:53

        # Forward other requests not yet answered, which is
        # needed for the alpine container to install packages.
        forward . 1.1.1.1
        log
      }

  # UPDATE: This isn't required, turns out you can forward to embedded DNS Docker uses for hostnames
  #
  # Normally this zone is already managed by k8s for you with the .svc.cluster.local addresses,
  # To demonstrate equivalent via compose, the alpine container uses CoreDNS to route through the reverse-proxy
  dns-zone:
    # NOTE: $$ required in content to escape $ from ENV interpolation feature
    content: |
      $$ORIGIN internal.test.
      $$TTL 60

      @       SOA    internal.test. admin.internal.test. 2024102030 4h 1h 14d 10m
      @       NS     ns1.internal.test.

      ; A records
      @       A      172.16.42.10
      rp      A      172.16.42.10
      ns1     A      172.16.42.9

compose.override.yaml (or update the earlier compose.yaml example)

services:
  # Embedded DNS from Docker can resolve to a container in the network via hostname
  # Should be similar to k8s svc.cluster.local assigned addresses
  reverse-proxy:
    hostname: rp.internal.test
  dms:
    # Unset the original mail.example.test hostname
    hostname: !reset
    # k8s can't set DMS FQDN via hostname, it must be configured via ENV instead:
    environment:
      OVERRIDE_HOSTNAME: mail.example.test

  # This service is constrained to only run when the "testing" profile is specified,
  # it just starts a container to run the pre-configured command to prove the CoreDNS rewrite feature works.
  # Usage: docker compose --profile testing run swaks
  swaks:
    # Use CoreDNS for resolving mail.example.test internally to the reverse-proxy container
    dns: 172.16.42.9
    # This hostname will resolve in the network when DMS does a lookup (via embedded DNS) which prevents:
    # 450 4.1.8 <hello@alpine.test>: Sender address rejected: Domain not found
    # It will also be used by swaks as the default client EHLO, which must be an FQDN for a basic SPF check to pass.
    hostname: alpine.test
    profiles:
      - testing
    # Automate steps to get the swaks package ready for testing, inline Dockerfile with HereDoc:
    # Alternatively just use DMS as the image for swaks + curl
    build:
      dockerfile_inline: |
        FROM alpine:edge
        RUN <<EOF
          apk add swaks --repository=https://dl-cdn.alpinelinux.org/alpine/edge/testing
          apk add perl-net-ssleay curl doggo
        EOF
    # `>` folds the multi-line yaml content to single line command
    command: >
      swaks --server mail.example.test --port 25 -tls
        --from hello@alpine.test
        --to jane.doe@example.test
        --tls-ca-path /tmp/ca-cert.pem
    # Optionally add credentials for ports 587/465:
    # --auth-user john.doe@example.test --auth-password secret
    # NOTE: Also include `--tls-sni mail.example.test` if using an explicit HostSNI rule.
    configs:
      - source: tls-ca-cert
        target: /tmp/ca-cert.pem
    # Not necessary, but makes it easier to identify this container by IP in logs when it's static
    networks:
      default:
        ipv4_address: 172.16.42.42

# Override the prior example cert files:
# - Use an ECDSA cert that's been signed by a self-signed CA for TLS cert verification.
# - This cert is only valid for mail.example.test
configs:
  # The swaks container will need to reference this file contents for successful verficiation.
  # Normally not required as a public CA cert like LetsEncrypt is included in the image.
  tls-ca-cert:
    content: |
      -----BEGIN CERTIFICATE-----
      MIIBejCCASGgAwIBAgIQRRIFsycc6tiFqOqcXCHZvDAKBggqhkjOPQQDAjAcMRow
      GAYDVQQDExFTbWFsbHN0ZXAgUm9vdCBDQTAeFw0yMTAxMDEwMDAwMDBaFw0zMTAx
      MDEwMDAwMDBaMBwxGjAYBgNVBAMTEVNtYWxsc3RlcCBSb290IENBMFkwEwYHKoZI
      zj0CAQYIKoZIzj0DAQcDQgAEz2IxYBk9cnhgWR4nE93P2RE2KDKv+ijkDm6rS62i
      SQDcbIkJmPr38o3tUFPbz21Pzp0aYZfFgHJeJjRKu8uBjKNFMEMwDgYDVR0PAQH/
      BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQEwHQYDVR0OBBYEFN6Qs7lNwbPudwCI
      i2nsccQw+fZ/MAoGCCqGSM49BAMCA0cAMEQCID87kOfKgnCOPy5yKrknRqzp4krb
      VgK8orKZ5I0QetVzAiByJWS2HKqmwxThZjW/oduQ6klZ+UToY96owLubIQhZhw==
      -----END CERTIFICATE-----

  tls-cert:
    content: |
      -----BEGIN CERTIFICATE-----
      MIIBxTCCAWqgAwIBAgIQHg296UPzq0FEkJdEzVjrSjAKBggqhkjOPQQDAjAcMRow
      GAYDVQQDExFTbWFsbHN0ZXAgUm9vdCBDQTAeFw0yMTAxMDEwMDAwMDBaFw0zMTAx
      MDEwMDAwMDBaMBkxFzAVBgNVBAMTDlNtYWxsc3RlcCBMZWFmMFkwEwYHKoZIzj0C
      AQYIKoZIzj0DAQcDQgAE9FsAamrKHbgVgIHQgnK+rzo8XqebZCEWGSfzdQvr4P5H
      amye19qADhsJdkX+i/15CfcIIhqTICF0XniRU0WecaOBkDCBjTAOBgNVHQ8BAf8E
      BAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBTY
      vlZSJ+eQsCFbX3nY+NSFV/ArvDAfBgNVHSMEGDAWgBTekLO5TcGz7ncAiItp7HHE
      MPn2fzAcBgNVHREEFTATghFtYWlsLmV4YW1wbGUudGVzdDAKBggqhkjOPQQDAgNJ
      ADBGAiEArQh78IJBLg7NK/eV/e5z2ZONdHzvKU3V2jME8LaxaxMCIQDX8ZXbvhi4
      23e5VwfmuVo9ADTT9esYZ5u6v4hicunJmQ==
      -----END CERTIFICATE-----

  tls-key:
    content: |
      -----BEGIN EC PRIVATE KEY-----
      MHcCAQEEIMl9lvsiSWUqCQ+cvXdiMApfVhQ/XH8zuiNhFD3JbGv9oAoGCCqGSM49
      AwEHoUQDQgAE9FsAamrKHbgVgIHQgnK+rzo8XqebZCEWGSfzdQvr4P5Hamye19qA
      DhsJdkX+i/15CfcIIhqTICF0XniRU0WecQ==
      -----END EC PRIVATE KEY-----

CLI commands

# compose.yaml + compose.override.yaml are used (Reverse Proxy + DMS):
docker compose up -d --force-recreate
# Bring up CoreDNS:
docker compose -f compose.dns.yaml up -d

# Demonstration with an internal container connecting to DMS indirectly with CoreDNS
# Certificate is verified successfully
docker compose --profile testing run swaks

The log output from swaks will show:

 -> STARTTLS
<-  220 2.0.0 Ready to start TLS
=== TLS started with cipher TLSv1.3:TLS_AES_256_GCM_SHA384:256
=== TLS client certificate not requested and not sent
=== TLS no client certificate set
=== TLS peer[0]   subject=[/CN=Smallstep Leaf]
===               commonName=[Smallstep Leaf], subjectAltName=[DNS:mail.example.test] notAfter=[2031-01-01T00:00:00Z]
=== TLS peer certificate passed CA verification, passed host verification (using host mail.example.test to verify)
# Querying CoreDNS should return an answer with the original DNS query and the expected reverse-proxy container IP
$ docker compose --profile testing run swaks doggo mail.example.test
NAME                    TYPE    CLASS   TTL     ADDRESS         NAMESERVER
mail.example.test.      A       IN      600s    172.16.42.10    127.0.0.11:53
polarathene commented 9 months ago

In case it's of any additional value, a webserver variant with Caddy and curl to show the original host query is preserved through the rewrite to the intended service.

services:
  # These must be appended to the main config,
  # command is special and gets replaced instead of appended for overrides.
  reverse-proxy:
    command:
      - "--entrypoints.web.address=:80"
      - "--entryPoints.websecure.address=:443"

  # This relies on configs defined previously for TLS cert files
  web:
    image: caddy:2.7
    labels:
      - traefik.enable=true

      # The HTTP service type lacks ProxyProtocol support, use TCP as Caddy is configured expecting ProxyProtocol connection.
      # Could alternatively have Traefik handle the redirect to 443:
      # https://doc.traefik.io/traefik/middlewares/http/redirectscheme/
      - traefik.tcp.routers.web.rule=HostSNI(`*`)
      - traefik.tcp.routers.web.entrypoints=web
      - traefik.tcp.routers.web.service=web
      - traefik.tcp.services.web.loadbalancer.server.port=80
      - traefik.tcp.services.web.loadbalancer.proxyProtocol.version=2

      # NOTE: Specifying an actual SNI will enforce using a TLS cert managed by Traefik:
      # SNI matching relies on TLS cert, thus Traefik would intervene even with passthrough.
      - traefik.tcp.routers.websecure.rule=HostSNI(`*`)
      - traefik.tcp.routers.websecure.entrypoints=websecure
      - traefik.tcp.routers.websecure.service=websecure
      - traefik.tcp.routers.websecure.tls.passthrough=true
      - traefik.tcp.services.websecure.loadbalancer.server.port=443
      - traefik.tcp.services.websecure.loadbalancer.proxyProtocol.version=2
    configs:
      - source: caddyfile
        target: /etc/caddy/Caddyfile
      - source: tls-cert
        target: /dms/certs/cert.pem
      - source: tls-key
        target: /dms/certs/key.pem

configs:
  caddyfile:
    content: |
      # Global options:
      {
        local_certs

        # Enable ProxyProtocol for incoming Traefik connections
        # https://caddyserver.com/docs/caddyfile/options#listener-wrappers
        servers {
          listener_wrappers {
            proxy_protocol {
              allow 172.16.42.10/32
            }
            # Restore default wrappers:
            # http_redirect must be added here for redirect support,
            # the setting `auto_https disable_redirects` cannot override it.
            http_redirect
            tls
          }
        }
      }

      mail.example.test {
        tls /dms/certs/cert.pem /dms/certs/key.pem
        respond "Hello {client_ip}!"
      }

      # These two have an implicit `tls internal` from the `local_certs` global option.
      # Caddy will provision with self-signed certs.
      hello.example.test {
        respond "No problem with rewrites! {client_ip}"
      }

      rp.internal.test {
        respond "The reverse proxy hostname, Client IP: {client_ip}"
      }
# Cert verifies successfully:
$ docker compose --profile testing run swaks curl -v --cacert /tmp/ca-cert.pem https://mail.example.test -w '\n'
# Alternative FQDN still matches in Caddy via Traefik and Caddy self-signed cert is identified (but cannot verify without CA file)
$ docker compose --profile testing run swaks curl -v --insecure https://hello.example.test -w '\n'
# Same as previous but with different FQDN and shows HTTP redirect to HTTPS working
$ docker compose --profile testing run swaks curl -L --insecure http://rp.internal.test -w '\n'
wernerfred commented 9 months ago

Neither of those have any relevance to port 993, so that documented concern is bizarre. Whomever contributed it was probably mistaken. Best I can get with git blame is a commit from @wernerfred but was tied to the Github Wiki migration PR where I see no context 🤷‍♂️ (EDIT: Found this comment which suggests it may have been part of the original Github Wiki)

Just my 2 cents on this: i do not recall it exactly (think that was somewhere around 2021) but in that time i build a test system to work on one of the issues that were opened asking about how to use proxy protocol. Iirc using port 993 brought up internal conflicts as the system adds/expects proxy information for the real ip and some internal usage of that port was broken then as it only accepts local origins. Using 10993 was working though.

As you already wrote: whomever contributed it was probably mistaken. But unless testing it - to make sure i will not break something we do not have on the radar right now - i wouldn't remove it.

edit: forgive me for not reading all the k8s related comments

georglauterbach commented 9 months ago

But it still seems like a lot of work to avoid opening extra standard mail ports that aren't even exposed outside the cluster.

The extra work is to preserve the client IP.

If you don't care about that from the internal container traffic (which may have been proxied already from the reverse-proxy like with Roundcube), then add the duplicate port config I guess?

I guess this is what this fundamental debate is about: I think on many smaller clusters, preserving the origin IP makes not a lot of sense for cluster-internal traffic as the origin IP is already a local IP anyway. I have whitelisted this traffic already.

cfis commented 9 months ago

Origin IP - I am not seeing the issue.

If you open two sets of ports - one with proxy support and one without - then the original client address is maintained for both internal and external clients. You don't need a proxy to track internal client IPs, they are talking directly to docker-mailserver so it already knows their IP addresses. Client IP is important for external connections, which gets preserved by the external proxy (if configured) that is not inside the cluster.

The only reason to go through a Proxy for internal clients, as far as I can tell, is to avoid having to open two sets of ports (so 993 without proxy support and 10993 with proxy support). But that requires installing a proxy inside the cluster and configuring it - which is way, way more effort than simply opening two ports per protocol (IMAPS, Submission, etc).

polarathene commented 9 months ago

TL;DR:

What about your external connections that route to an internal container, and then connect to DMS where the client IP should have been preserved? Or internal container connections with multiple hops where the client IP of the original container should have been preserved?

You don't need a proxy to track internal client IPs, they are talking directly to docker-mailserver so it already knows their IP addresses.

Roundcube was brought up before where you don't want the roundcube container IP, you want the IP of the user authenticating through it through their browser.

For an internal equivalent, a separate container that provides an API for sending mail (eg via HTTP REST) that multiple internal containers may call instead of having direct SMTP capabilities. One of those containers may be client facing allowing users to provide their own credentials, which introduces the same issue if the API container is always the source IP to DMS; all services relying on it then get locked out by Fail2Ban (alternatively by trusting the API container IP due to local private network, you allow external users to abuse the trust to auth repeatedly without consequence).

Perhaps there are other examples, the point was those concerns are better alleviated by preserving the client IP properly and avoiding any misconfiguration issues by keeping it seamless throughout infrastructure, no need to worry about differences in FQDN or ports.

At least IMO that consistency of all traffic via single ports and either all connections to through the same route (either direct, or via a proxy when present) is preferable. Clearly we disagree, mostly because of differences with how k8s does things, and the only reason exceptions are made for it unlike others like Podman is because of maintainer interest.

Comparison

This is based on the current state, without any DMS feature support implemented.

Route local traffic through proxy

Minimal DNS service configuration (or addition if you already have one):

# Whatever the equivalent is in k8s config:
# https://kubernetes.io/docs/tasks/administer-cluster/coredns/#what-s-next
# https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/
services:
  dns:
    image: coredns/coredns:1.11.1
    working_dir: /
    configs:
      - source: dns-config
        target: /Corefile

configs:
  dns-config:
    content: |
      . {
        rewrite name mail.example.test rp.internal.test

        # Slightly different for k8s, use kubernetes rule instead to forward to the cluster DNS
        forward internal.test 127.0.0.11:53
        forward . 1.1.1.1
      }

Simpler config overrides (especially Postfix):

configs:
  overrides-dovecot:
    content: |
      haproxy_trusted_networks=172.16.42.10

      service imap-login {
        inet_listener imap {
          haproxy = yes
        }
        inet_listener imaps {
          haproxy = yes
        }
      }

      service pop3-login {
        inet_listener pop3 {
          haproxy = yes
        }
        inet_listener pop3s {
          haproxy = yes
        }
      }

  overrides-postfix-master:
    content: |
      smtp/inet/postscreen_upstream_proxy_protocol=haproxy
      submission/inet/smtpd_upstream_proxy_protocol=haproxy
      submissions/inet/smtpd_upstream_proxy_protocol=haproxy

Separate proxy ports configured

Your current helm chart approach (ignoring the postfix SMTP port omission):

configMaps:
  dovecot.cf:
    create: true
    path: dovecot.cf
    data: |
      {{- if .Values.proxyProtocol.enabled }}
        haproxy_trusted_networks = {{ .Values.proxyProtocol.trustedNetworks }}

        {{- if and (.Values.deployment.env.ENABLE_IMAP) (not .Values.deployment.env.SMTP_ONLY) }}
        service imap-login {
            inet_listener imap {
                port = 143
            }

            inet_listener imaps {
                port = 993
                ssl = yes
            }

            inet_listener imap_proxy {
                haproxy = yes
                port = 10143
                ssl = no
            }

            inet_listener imaps_proxy {
                haproxy = yes
                port = 10993
                ssl = yes
            }
        }    
        {{- end -}}

        {{- if and (.Values.deployment.env.ENABLE_POP) (not .Values.deployment.env.SMTP_ONLY) }}
        service pop3-login {
            inet_listener pop3 {
                port = 110
            }

            inet_listener pop3s {
                port = 995
                ssl = yes
            }

            inet_listener pop3_proxy {
                haproxy = yes
                port = 10110
                ssl = no
            }

            inet_listener pop3s_proxy {
                haproxy = yes
                port = 10995
                ssl = yes
            }                        
        }
        {{- end -}}
      {{- end -}}

  user-patches.sh:
    create: true
    path: user-patches.sh
    data: |
      #!/bin/bash

      {{- if .Values.proxyProtocol.enabled }}
      # Make sure to keep this file in sync with https://github.com/docker-mailserver/docker-mailserver/blob/master/target/postfix/master.cf!
      cat <<EOS >> /etc/postfix/master.cf

      # Submission with proxy
      10587     inet  n       -       n       -       -       smtpd
        -o syslog_name=postfix/submission
        -o smtpd_tls_security_level=encrypt
        -o smtpd_sasl_auth_enable=yes
        -o smtpd_sasl_type=dovecot
        -o smtpd_reject_unlisted_recipient=no
        -o smtpd_sasl_authenticated_header=yes
        -o smtpd_client_restrictions=permit_sasl_authenticated,reject
        -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
        -o smtpd_sender_restrictions=\$mua_sender_restrictions
        -o smtpd_discard_ehlo_keywords=
        -o milter_macro_daemon_name=ORIGINATING
        -o cleanup_service_name=sender-cleanup
        -o smtpd_upstream_proxy_protocol=haproxy  

      # Submissions with proxy
      10465     inet  n       -       n       -       -       smtpd
        -o syslog_name=postfix/submissions
        -o smtpd_tls_wrappermode=yes
        -o smtpd_sasl_auth_enable=yes
        -o smtpd_sasl_type=dovecot
        -o smtpd_reject_unlisted_recipient=no
        -o smtpd_sasl_authenticated_header=yes
        -o smtpd_client_restrictions=permit_sasl_authenticated,reject
        -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
        -o smtpd_sender_restrictions=\$mua_sender_restrictions
        -o smtpd_discard_ehlo_keywords=
        -o milter_macro_daemon_name=ORIGINATING
        -o cleanup_service_name=sender-cleanup
        -o smtpd_upstream_proxy_protocol=haproxy
      EOS
      {{- end }}

In fairness, your Postfix configuration can be simpler like this:

#!/bin/bash

# Duplicate the config for the submission(s) service ports (587/465) with adjustments for the proxy ports (10587/10465) and syslog_name setting:
postconf -Mf submission/inet | sed -e s/^submission/10587/ -e 's/submission/submission-proxyprotocol/' >> /etc/postfix/master.cf
postconf -Mf submissions/inet | sed -e s/^submissions/10465/ -e 's/submissions/submissions-proxyprotocol/' >> /etc/postfix/master.cf
# Enable ProxyProtocol support for these new service variants:
postconf -P 10587/inet/smtpd_upstream_proxy_protocol=haproxy
postconf -P 10465/inet/smtpd_upstream_proxy_protocol=haproxy

# Create a variant for port 25 too (NOTE: Port 10025 is already assigned in DMS to Amavis):
postconf -Mf smtp/inet | sed -e s/^smtp/12525/ >> /etc/postfix/master.cf
# Enable ProxyProtocol support (different setting as port 25 is handled via postscreen), optionally configure a syslog name to distinguish in logs:
postconf -P 12525/inet/postscreen_upstream_proxy_protocol=haproxy 12525/inet/syslog_name=smtp-proxyprotocol

Conclusion

I will update the relevant docs page with config advice shared here, it'll be part of the v14 release.

If someone wants to implement this feature for v15, you have the improved approach above.

polarathene commented 9 months ago

@wernerfred

As you already wrote: whomever contributed it was probably mistaken. But unless testing it - to make sure i will not break something we do not have on the radar right now - i wouldn't remove it.

edit: forgive me for not reading all the k8s related comments

No need to read anything else, it's a long thread 😅

If you were the original contributor of the current PROXY protocol docs, then you probably encountered the above. As you mentioned there would otherwise be a conflict with 993 as enabling proxy support rejects any connection that doesn't provide the proxy headers.

The docs page will still clearly cover both needs 👍 with proxy support being simplified in a future DMS release.

georglauterbach commented 9 months ago

What about your external connections that route to an internal container, and then connect to DMS where the client IP should have been preserved? Or internal container connections with multiple hops where the client IP of the original container should have been preserved?

You don't need a proxy to track internal client IPs, they are talking directly to docker-mailserver so it already knows their IP addresses.

Roundcube was brought up before where you don't want the roundcube container IP, you want the IP of the user authenticating through it through their browser.

Does Roundcube actually do that? I had no idea... I see no real use-case for routing to another container first, which then connects to DMS. Even if there was, the traffic is still internal to the cluster. Only in zero-trust architectures would this matter from a security perspective, right?

polarathene commented 9 months ago

Does Roundcube actually do that? I had no idea...

AFAIK, I linked to a thread in our discussions page where a user was having trouble getting it setup. (EDIT: I don't see the link anywhere above, so here it is)

There is a plugin for roundcube to support a way via IMAP that Dovecot supports, while the official roundcube support is only partial for PROXY protocol apparently, they can only use it with STARTTLS from what I saw.

I see no real use-case for routing to another container first, which then connects to DMS. Even if there was, the traffic is still internal to the cluster

I gave one didn't I?

A container that provides an HTTP API to send mail through SMTP submission to DMS. If you have another container that's public facing, or routes the HTTP API, you'd have a similar scenario wouldn't you with user authentication if you were building something like SendGrid? (which has an HTTP API for sending mail)

That'd need to support preserving the client IP itself, similar to how the Caddy example config demonstrates, but since I don't think Traefik helps in manipulating the PROXY protocol, your API service would need to handle proxying the TCP connection itself (so that client IP info is prepended via PROXY header prior to the SMTP connection itself), either to Traefik or directly to DMS.

For rust with Lettre, that seems like you'd need to do your own custom transport I think 🤷‍♂️

Doesn't seem like it'll be a common requirement, and honestly if it's for something like Fail2Ban, then I guess rather than rely solely on DMS, it'd be better to monitor logs from that API service instead.

Only in zero-trust architectures would this matter from a security perspective, right?

I guess? I only have the Roundcube example as a real-world one that I'm aware of where this type of issue could happen unexpectedly, and that perhaps there are other services that could encounter it too. Routing through a proxy won't magically fix that, but it is better to be restrictive on the trusted proxies (where PROXY protocol is established from), since you allow anything on the subnet capable of connecting to forge this connection info otherwise.

Given how we have several networking surprises with Docker (and I recall some of these at least applied to k8s in the past, possibly resolved since) and not just IPv6 related, I'm not one to advise trusting the entire subnet so easily just because it's in the private range so assume it's safe 🤷‍♂️

In fact mailcow docs have a big warning about an IPv6 change to be aware of with recent Docker v25 and backports.

With k8s at least, I know that they use .local incorrectly and that it's a known mistake by the maintainers due to overlap with mDNS, though I can't recall any issues that causes within k8s. There was also an issue related to one that still exists with Docker based on an implicitly configured kernel tunable for routing traffic within a trusted private network into subnets that normally shouldn't be reachable.

These can come/go over time, better to minimize trust.

cfis commented 9 months ago

Based on this discussion, there are 3 types of clients:

Onto Roundcube. From my testing, by default, docker-mailserver sees the source ip as roundcube's ip. For imap, and only imap, you can install a plugin that uses an imap specific feature to send the client ip (which Roundcube knows via HTTP headers). Examples include https://gitlab.com/takerukoushirou/roundcube-dovecot_client_ip and https://gitlab.com/takerukoushirou/roundcube-dovecot_client_ip. Or you can use Roudcube's limited proxy support as mentioned above.

However, for ports 465 or 587 (and 110), from what I saw, there is no proxy support. So Roundcube requires having non-proxy ports available - so now you are back to having two sets of ports.

So unless I am missing something, that means the only safe setup, at least for Roudcube (or NexctCloud email or SnappyMail or RainLoop) would be requiring authentication both for sending and retrieving emails. If the user can authenticate, then the user is presumably trusted, and the source ip is irrelevant.

That isn't any different than other web email clients. For example, if you send yourself an email from Google mail the source ip is Google's server, not my browser's ip address.

polarathene commented 9 months ago

If the user can authenticate, then the user is presumably trusted, and the source ip is irrelevant.

IIRC the issue was related to Fail2Ban within DMS monitoring logs, such as those with IPs that were failing auth multiple times. This is to prevent malicious users trying to access accounts that are not theirs, or DDoS related AFAIK.

Without any other known concerns, I'm fine with that being a documented gotcha and users should instead trust those internal IPs and have the other containers auth logs monitored instead.

That isn't any different than other web email clients. For example, if you send yourself an email from Google mail the source ip is Google's server, not my browser's ip address.

It never was about receiving mail with a source IP different from DMS.

My other example was authenticated submission, related to the same concern with a single container IP representing multiple clients authenticating in the logs through that container service. Except this would have been for ports 587/465 instead.

When other mail servers connect to DMS, it's important that their IP is correct, but that's only a known issue with external connections through a proxy which the PROXY protocol support resolves, so we're good there.

However, for ports 465 or 587 (and 110), from what I saw, there is no proxy support. So Roundcube requires having non-proxy ports available - so now you are back to having two sets of ports.

This doesn't change. You can route through the reverse-proxy or use a direct connection (by duplicating the ports), your choice.

cfis commented 9 months ago

Fail2Ban - yes - I think you are right. I have no better ideas.

This doesn't change. You can route through the reverse-proxy or use a direct connection (by duplicating the ports), your choice.

So I realize I don't follow this part. Let's stick with the Roudcube example. To preserve the ip, Roudcube would need to turn on its proxy protocol support (at least for IMAP) when talking to the proxy which would also need to be expecting the Proxy protocol. But then the proxy isn't actually doing anything useful, roundcube could just talk directly to docker-mailserver. If instead Roudcube is not using the proxy protocol and goes through the proxy, then the client ip is going end up being Roundcube's ip address.

Anyway, after all this long discussion, have we come to any conclusions?

polarathene commented 9 months ago

So I realize I don't follow this part.

I was stating that nothing changed in what you said to require the extra ports be configured.

The outcome is the same regardless of DMS having PROXY protocol only ports (where Roundcube would route through the reverse-proxy) or duplicate ports with PROXY protocol enabled on them (so Roundcube could talk directly to DMS instead).

have we come to any conclusions?

..Yes? See my earlier comment: https://github.com/docker-mailserver/docker-mailserver/issues/3866#issuecomment-1933167693

There's a brief TL;DR and at the end of the comment a longer "Conclusion" section with more info regarding that.

I will document what was discussed here for v14, and encourage a contribution for duplicate port approach for v15+

georglauterbach commented 9 months ago

I was finally able to get into Proxy Protocol myself. After enormous other hassles this week (unrelated to DMS itself), I started fiddling around with Proxy Protocol. I set up Traefik according to our docs and used custom ports (10025 (I do not use Amavis, all well), 10465, etc.) for Proxy Protocol traffic. What can I say: it works flawlessly! Now I can finally get rid of my second IPv4 address that I required earlier due to the need for matching A and PTR record and save a few bucks.

I run a completely custom master.cf anyway, this is what it looks like now 
#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master" or
# on-line: http://www.postfix.org/master.5.html).
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service      type  private unpriv  chroot  wakeup  maxproc command + args
#                    (yes)   (yes)   (no)    (never) (100)
# ==========================================================================

# We list everything except smtp, submission and submissions first.

smtpd          pass  -       -       n       -       -       smtpd
tlsproxy       unix  -       -       n       -       0       tlsproxy
dnsblog        unix  -       -       n       -       0       dnsblog
cleanup        unix  n       -       n       -       0       cleanup
qmgr           unix  n       -       n       300     1       qmgr
tlsmgr         unix  -       -       n       1000?   1       tlsmgr
rewrite        unix  -       -       n       -       -       trivial-rewrite
bounce         unix  -       -       n       -       0       bounce
defer          unix  -       -       n       -       0       bounce
trace          unix  -       -       n       -       0       bounce
verify         unix  -       -       n       -       1       verify
flush          unix  n       -       n       1000?   0       flush
proxymap       unix  -       -       n       -       -       proxymap
proxywrite     unix  -       -       n       -       1       proxymap
smtp           unix  -       -       n       -       -       smtp
relay          unix  -       -       n       -       -       smtp
showq          unix  n       -       n       -       -       showq
error          unix  -       -       n       -       -       error
retry          unix  -       -       n       -       -       error
discard        unix  -       -       n       -       -       discard
local          unix  -       n       n       -       -       local
virtual        unix  -       n       n       -       -       virtual
lmtp           unix  -       -       n       -       -       lmtp
anvil          unix  -       -       n       -       1       anvil
scache         unix  -       -       n       -       1       scache
postlog  unix-dgram  n       -       n       -       1       postlogd

pickup         fifo  n       -       n       60      1       pickup
  -o content_filter=
  -o receive_override_options=no_header_body_checks

sender-cleanup unix  n       -       n       -       0       cleanup
  -o syslog_name=postfix/sender-cleanup
  -o header_checks=pcre:/etc/postfix/maps/sender_header_filter.pcre

# Now we list smtp, submission and submissions without Proxy Protocol.

smtp           inet  n       -       n       -       1       postscreen
  -o syslog_name=postfix/smtp/noproxy

submissions    inet  n       -       n       -       -       smtpd
  -o syslog_name=postfix/submissions/noproxy
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_authenticated_header=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o smtpd_sender_restrictions=reject_authenticated_sender_login_mismatch,$smtpd_sender_restrictions_default
  -o smtpd_discard_ehlo_keywords=
  -o milter_macro_daemon_name=ORIGINATING
  -o cleanup_service_name=sender-cleanup

submission     inet  n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission/noproxy
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_authenticated_header=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o smtpd_sender_restrictions=reject_authenticated_sender_login_mismatch,$smtpd_sender_restrictions_default
  -o smtpd_discard_ehlo_keywords=
  -o milter_macro_daemon_name=ORIGINATING
  -o cleanup_service_name=sender-cleanup

# Now we list smtp, submission and submissions with Proxy Protocol.

10025          inet  n       -       n       -       1       postscreen
  -o syslog_name=postfix/smtp/proxy
  -o postscreen_upstream_proxy_protocol=haproxy

10465          inet  n       -       n       -       -       smtpd
  -o syslog_name=postfix/submissions/proxy
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_authenticated_header=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o smtpd_sender_restrictions=reject_authenticated_sender_login_mismatch,$smtpd_sender_restrictions_default
  -o smtpd_discard_ehlo_keywords=
  -o milter_macro_daemon_name=ORIGINATING
  -o cleanup_service_name=sender-cleanup
  -o smtpd_upstream_proxy_protocol=haproxy

10587          inet  n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission/proxy
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_authenticated_header=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o smtpd_sender_restrictions=reject_authenticated_sender_login_mismatch,$smtpd_sender_restrictions_default
  -o smtpd_discard_ehlo_keywords=
  -o milter_macro_daemon_name=ORIGINATING
  -o cleanup_service_name=sender-cleanup
  -o smtpd_upstream_proxy_protocol=haproxy