Retaining source addresses after forwarding SMTP connections to mox running behind NAT

[jstasiak]

Hey, first of all thank you for this project.

I have a use case where

1. I'd like to run an email server behind NAT (the reasons for that aren't interesting, let's call the machine it'd run on Machine A). It's all IPv4.
2. I'd like the server to be publicly accessible via forwarding connections from a machine with a public IP address (Machine B).
3. The email server running on Machine A needs to use Machine B to send email to the outside world (the public IP address of Machine A's connection wouldn't be accepted by third-party email servers).
4. I'd like Machine B to forward the source address and port information to the email server running on Machine A so that my logs are useful, I can make anti-spam decisions based on that etc.

IPv6 isn't the answer in my case, sadly.

Points 1 and 2 are easy, just set up a reverse TCP proxy. Point 3 is achievable since the SOCKS proxy support has been added to mox[1].

What's left is point 4. I've been wondering – would something like the PROXY protocol[2] support be considered a fitting addition to mox? It's lightweight, protocol-independent and has a reasonably broad support (nginx, which I use, speaks it).

As a bonus, if this actually happened, when running in Docker mox wouldn't require the host-mode networking[3] to see the IP addresses anymore.

[1] https://github.com/mjl-/mox/issues/36 [2] https://github.com/haproxy/haproxy/blob/3a0b44b122067676ad13d8348a17e6c5bd279901/doc/proxy-protocol.txt [3] https://github.com/mjl-/mox/issues/24

[mjl-]

Hi @jstasiak, this seems like a reasonable approach indeed.

We could implement a ProxyListener that implements the net.Listener interface. It would wrap a another listener (created with net.Listen, for tcp connections). Its Accept() method would return a type that implements net.Conn that returns the original client address for RemoteAddr(). That would make it transparent to most of the mox code. I see there is one occurrence that looks for a connection being a net.TCPConn for setting keepalives. That could need a small change.

We could add a field TrustedProxyIPs to a config.Listener, with a list of subnets (0.0.0.0/0 or 192.168.1.1/32, etc). Any connection coming in from a TrustedProxyIP will get the PROXY protocol treatment, others wouldn't. All code in mox where a new listener is created would have to check if TrustedProxyIPs is set and wrap the listener in a ProxyListener.

The PROXY protocol seems simple enough to implement in mox (not adding a new dependency). Probably best to implement both v1 and v2. Must be careful not to Read() data beyond the proxy header. For v2 that's easy, for v1 it's probably best to inefficiently do a bunch of small reads until end of line (the spec has example code that peeks at available data without consuming, but let's not go there with Go).

Would this really help not needing host-mode networking with docker? Does docker implement the PROXY protocol? I think I've looked at support for a PROXY-like protocol in the past, but couldn't find docker implementing it.

Are you interested in working on this? If so I can give hints for development.

[jstasiak]

This is almost exactly how I'd envision the implementation and the behavior so that's good to hear. My only concern is that maybe the protocol is not so simple to implement to warrant skipping a dependency here – judging by https://github.com/pires/go-proxyproto code – but that's just my intuition so far.

What I forgot to mention in my Docker remark is that in that non-host-mode scenario where proxy protocol was used there would have to be a proxy-protocol-speaking reverse proxy in front of it – I don't imagine Docker itself could handle it. So not quite out of the box solution there.

Yes, I'm not sure how soon exactly but I'd like to take a stab at it – any and all tips are appreciated.

[mjl-]

Yes, I'm not sure how soon exactly but I'd like to take a stab at it – any and all tips are appreciated.

Great!

I looked at the listening code again just now. I think it's mostly the code that calls mox.Listen() needs changes. That's the smtp, imap and http servers. See https://github.com/search?q=repo%3Amjl-%2Fmox%20mox.Listen&type=code. mox.Listen abstracts away getting a listener file descriptor either by calling net.Listen or taking a listener fd that was passed by its parent privileged process.

Another relevant topic is the TLS listeners. They are created by tls.NewListener(plainListener, ...). They occur at the same call sites as the mox.Listen() calls from above that need changing.

So far, initializing a listener works like this: mox.Listen() to get a plain listener. Then wrap the listener in tls.NewListener() if TLS should be enabled. I expect the new order to be: mox.Listen() for plain listener, optionally wrap that in a PROXY listener, then optionally wrap that listener in the TLS listener. Perhaps it makes sense to either modify mox.Listen or to make a new listen-helper function that handles the wrapping into a PROXY listener.

mox.Listen is at https://github.com/mjl-/mox/blob/aead73883601ecb259e7a27d834bcbaf8e4af07a/mox-/lifecycle.go#L80.

There are a few direct calls to net.Listen() in mox, but those are for unix domain sockets or tests, so I don't expect they need to be changed.

This check for net.TCPConn may need changing: https://github.com/mjl-/mox/blob/aead73883601ecb259e7a27d834bcbaf8e4af07a/imapserver/server.go#L675 And the few places in smtpserver that expect net.TCPAddr may need attention: https://github.com/search?q=repo%3Amjl-%2Fmox+net.TCP&type=code

As for using a library: If it really fits it's fine to use it, but often there is some issue (e.g. go-proxyproto seems to inject a bufio.Reader in every connection, not great). Keep in mind we also need testing code that dials, so we need a PROXY "client" as well.

We can also discuss by email/irc/matrix/slack.