mjl-
commented
1 week ago Hi @delano, thanks for the writeup, that's good progress and looks like you're going to get mox running on fly.io at least for incoming email.
At the time of issue #14, mox wasn't able to deliver outgoing messages through other services, but that is an option now. Either with SMTP (unauthenticated relay, or authenticated submission), or by making connections using socks. So you would still need another service/machine to send email from, but at least it wouldn't be the place where the incoming emails are stored.
Is fly.io routing/forwarding connections to port 443 on the TCP-level? Or is it doing HTTPS-proxying? If it's possible to get it to just forward tcp connections, mox should be able to use ACME (Let's Encrypt or others) to do all TLS.
The public IPs are static right? That is, once you found them by starting the app. If so, and traffic can be delivered as TCP (not reverse proxied with HTTP(s)), couldn't you set those public IPs directly in the public listener's "IPs" field again instead of in "NATIPs"? Keep in mind that for junk filtering based on IP address (for messages from first-time senders, i.e. most spam messages), it is important that mox sees the original remote IP address, not an internal load balancer IP address.
If I can help with anything, let me know.
delano
I've been playing around with the idea of running a mail server on fly.io for a while and had some time this weekend to look into it. Similar to your sentiment on #14, it's been a good reason to learn more about fly.io at the same time. The same issues with sending mail that were mentioned in #14 still apply, but I was curious to see if there were any updates or new learnings since then, so I decided to give it a shot and document my findings here.
There's something really cool about a mail server that can spin down to 0 machines and quickly re-hydrate it self when a message comes in. It's an interesting twist on how we might get to a place where we (as individuals) run more of our own services again.
Steps
Anyway, here's what I did:
mox(this was just to simplify but the Dockerfile as-is should work fine too).docker build -t mox .mox quickstartcommand to generate the initial configuration. This was a notable shortcut that avoided dealing with fly volumes but has its obvious drawbacks: (1) the IP detection and other heuristics obviously don't match the fly.io environment and (2) the generated passwords, keys, and certs end up in the deployed image which is not ideal for security reasons (although I have a solution for that which I'll mention at the end).fly.toml.fly launch.yto copying the existing config.yto adding public 1pv4 and ipv6 addresses.mox.confinternalIPs as 127.0.0.1publicIPs to0.0.0.0and::0.publicNATIPs to the fly ipv4 and ipv6 addresses (since they are only available after lanuch -- same as #14).fly deployto re-reploy the app with the updated config.This got things running but I ran into a few issues:
tl;dr
Future improvements
fly certsto "prime" the TLS before the web services start. It wouldn't replace it since you want SMTP to be able to negotiate with its own certs but it would be nice to have a more robust TLS story.fly launchargs--file-literal,--file-local, and--file-secretto inject config and data into the container at build time. This[deploy]setting calledrelease_command. e.g. for runningrelease_command = "mox quickstart".Files
fly.tomlDetails
```toml app = 'mox' primary_region = 'ams' [build] dockerfile = 'Dockerfile' [deploy] wait_timeout = '3m0s' [http_service] internal_port = 80 force_https = false auto_stop_machines = true auto_start_machines = true min_machines_running = 0 processes = ['app'] [[services]] protocol = 'tcp' internal_port = 25 [[services.ports]] port = 25 [[services.ports]] port = 465 handlers = ['tls'] [[services.ports]] port = 587 [[services.ports]] port = 80 handlers = ['http'] [[services.ports]] port = 443 handlers = ['http', 'tls'] [[services.ports]] port = 993 handlers = ['tls'] [[services.ports]] port = 143 [[services.ports]] port = 8010 [[restart]] policy = 'never' retries = 1 processes = ['app'] [[vm]] memory = '1gb' cpu_kind = 'shared' cpus = 1 ```
DockerfileDetails
```Dockerfile ARG GO_VERSION=1 FROM golang:${GO_VERSION}-bookworm as builder # Create mox user and homedir (or pick another name or homedir): RUN useradd -m -d /home/mox mox WORKDIR /home/mox # Download and install mox: RUN apt-get update && \ apt-get install -y curl ca-certificates gnupg2 tzdata vim \ iputils-ping dnsutils iproute2 nmap netcat-openbsd # Find the latest release: https://beta.gobuilds.org/github.com/mjl-/mox@latest/linux-amd64-latest/ RUN curl -v -O -L https://beta.gobuilds.org/github.com/mjl-/mox@v0.0.11/linux-amd64-go1.22.5/0pAbuhqp5CMBgh73mTHluWLML6-I/mox-v0.0.11-go1.22.5.gz RUN gunzip mox-v0.0.11-go1.22.5.gz RUN mv mox-v0.0.11-go1.22.5 mox RUN chmod +x mox RUN mv mox /bin/ # When setting up a new system, don't run the mox quickstart in this # Dockerfile. Instead we build and then run the container with a volume # for the config and data, overriding the CMD. The config files generated # are then made for linux (so they include the mox.service) and then # when the image runs the first time all those files are inplace. COPY mox_data /home/mox RUN chown -R 1000:0 /home/mox/config RUN chown -R 1000:0 /home/mox/data # SMTP for incoming message delivery. EXPOSE 25/tcp # SMTP/submission with TLS. EXPOSE 465/tcp # SMTP/submission without initial TLS. EXPOSE 587/tcp # HTTP for internal account and admin pages. EXPOSE 80/tcp # HTTPS for ACME (Let's Encrypt), MTA-STS and autoconfig. EXPOSE 443/tcp # IMAP with TLS. EXPOSE 993/tcp # IMAP without initial TLS. EXPOSE 143/tcp # Prometheus metrics. EXPOSE 8010/tcp #CMD pwd && ls -lha CMD ["/bin/mox", "serve"] ```
mox.confDetails
``` # NOTE: This config file is in 'sconf' format. Indent with tabs. Comments must be # on their own line, they don't end a line. Do not escape or quote strings. # Details: https://pkg.go.dev/github.com/mjl-/sconf. # https://www.xmox.nl/config/ # Directory where all data is stored, e.g. queue, accounts and messages, ACME TLS # certs/keys. If this is a relative path, it is relative to the directory of # mox.conf. DataDir: ../data # Default log level, one of: error, info, debug, trace, traceauth, tracedata. # Trace logs SMTP and IMAP protocol transcripts, with traceauth also messages with # passwords, and tracedata on top of that also the full data exchanges (full # messages), which can be a large amount of data. LogLevel: debug # User to switch to after binding to all sockets as root. Default: mox. If the # value is not a known user, it is parsed as integer and used as uid and gid. # (optional) User: mox # Full hostname of system, e.g. mail.
Hostname: mail.example.com
# If enabled, a single DNS TXT lookup of _updates.xmox.nl is done every 24h to
# check for a new release. Each time a new release is found, a changelog is
# fetched from https://updates.xmox.nl/changelog and delivered to the postmaster
# mailbox. (optional)
#
# RECOMMENDED: please enable to stay up to date
#
#CheckUpdates: true
# Automatic TLS configuration with ACME, e.g. through Let's Encrypt. The key is a
# name referenced in TLS configs, e.g. letsencrypt. (optional)
ACME:
letsencrypt:
# For letsencrypt, use https://acme-v02.api.letsencrypt.org/directory.
DirectoryURL: https://acme-v02.api.letsencrypt.org/directory
# Email address to register at ACME provider. The provider can email you when
# certificates are about to expire. If you configure an address for which email is
# delivered by this server, keep in mind that TLS misconfigurations could result
# in such notification emails not arriving.
ContactEmail: d@mail.example.com
# If set, used for suggested CAA DNS records, for restricting TLS certificate
# issuance to a Certificate Authority. If empty and DirectyURL is for Let's
# Encrypt, this value is set automatically to letsencrypt.org. (optional)
IssuerDomainName: letsencrypt.org
# File containing hash of admin password, for authentication in the web admin
# pages (if enabled). (optional)
AdminPasswordFile: adminpasswd
# Listeners are groups of IP addresses and services enabled on those IP addresses,
# such as SMTP/IMAP or internal endpoints for administration or Prometheus
# metrics. All listeners with SMTP/IMAP services enabled will serve all configured
# domains. If the listener is named 'public', it will get a few helpful additional
# configuration checks, for acme automatic tls certificates and monitoring of ips
# in dnsbls if those are configured.
Listeners:
internal:
# Use 0.0.0.0 to listen on all IPv4 and/or :: to listen on all IPv6 addresses, but
# it is better to explicitly specify the IPs you want to use for email, as mox
# will make sure outgoing connections will only be made from one of those IPs. If
# both outgoing IPv4 and IPv6 connectivity is possible, and only one family has
# explicitly configured addresses, both address families are still used for
# outgoing connections. Use the "direct" transport to limit address families for
# outgoing connections.
IPs:
- 127.0.0.1
# If empty, the config global Hostname is used. (optional)
Hostname: mail.example.com
# Account web interface, for email users wanting to change their accounts, e.g.
# set new password, set new delivery rulesets. Default path is /. (optional)
AccountHTTP:
Enabled: true
# Admin web interface, for managing domains, accounts, etc. Default path is
# /admin/. Preferably only enable on non-public IPs. Hint: use 'ssh -L
# 8080:localhost:80 you@yourmachine' and open http://localhost:8080/admin/, or set
# up a tunnel (e.g. WireGuard) and add its IP to the mox 'internal' listener.
# (optional)
AdminHTTP:
Enabled: true
# Webmail client, for reading email. Default path is /webmail/. (optional)
WebmailHTTP:
Enabled: true
# Like WebAPIHTTP, but with plain HTTP, without TLS. (optional)
WebAPIHTTP:
Enabled: true
# All configured WebHandlers will serve on an enabled listener. (optional)
WebserverHTTP:
Enabled: false
# Serve prometheus metrics, for monitoring. You should not enable this on a public
# IP. (optional)
MetricsHTTP:
Enabled: false
public:
# Use 0.0.0.0 to listen on all IPv4 and/or :: to listen on all IPv6 addresses, but
# it is better to explicitly specify the IPs you want to use for email, as mox
# will make sure outgoing connections will only be made from one of those IPs. If
# both outgoing IPv4 and IPv6 connectivity is possible, and only one family has
# explicitly configured addresses, both address families are still used for
# outgoing connections. Use the "direct" transport to limit address families for
# outgoing connections.
IPs:
- 0.0.0.0
- ::
# If set, the mail server is configured behind a NAT and field IPs are internal
# instead of the public IPs, while NATIPs lists the public IPs. Used during
# IP-related DNS self-checks, such as for iprev, mx, spf, autoconfig,
# autodiscover, and for autotls. (optional)
NATIPs:
- 1.12.123.234
- 0a01:01ff:2::5a:e11b:0
# For SMTP/IMAP STARTTLS, direct TLS and HTTPS connections. (optional)
TLS:
# Name of provider from top-level configuration to use for ACME, e.g. letsencrypt.
# (optional)
ACME: letsencrypt
# Private keys used for ACME certificates. Specified explicitly so DANE TLSA DNS
# records can be generated, even before the certificates are requested. DANE is a
# mechanism to authenticate remote TLS certificates based on a public key or
# certificate specified in DNS, protected with DNSSEC. DANE is opportunistic and
# attempted when delivering SMTP with STARTTLS. The private key files must be in
# PEM format. PKCS8 is recommended, but PKCS1 and EC private keys are recognized
# as well. Only RSA 2048 bit and ECDSA P-256 keys are currently used. The first of
# each is used when requesting new certificates through ACME. (optional)
HostPrivateKeyFiles:
- hostkeys/mail.example.com.20240706T223117.rsa2048.privatekey.pkcs8.pem
- hostkeys/mail.example.com.20240706T223117.ecdsap256.privatekey.pkcs8.pem
# (optional)
SMTP:
Enabled: true
# Addresses of DNS block lists for incoming messages. Block lists are only
# consulted for connections/messages without enough reputation to make an
# accept/reject decision. This prevents sending IPs of all communications to the
# block list provider. If any of the listed DNSBLs contains a requested IP
# address, the message is rejected as spam. The DNSBLs are checked for healthiness
# before use, at most once per 4 hours. IPs we can send from are periodically
# checked for being in the configured DNSBLs. See MonitorDNSBLs in domains.conf to
# only monitor IPs we send from, without using those DNSBLs for incoming messages.
# Example DNSBLs: sbl.spamhaus.org, bl.spamcop.net. See
# https://www.spamhaus.org/sbl/ and https://www.spamcop.net/ for more information
# and terms of use. (optional)
#DNSBLs:
#- sbl.spamhaus.org
#- bl.spamcop.net
# SMTP over TLS for submitting email, by email applications. Requires a TLS
# config. (optional)
Submissions:
Enabled: true
# IMAP over TLS for reading email, by email applications. Requires a TLS config.
# (optional)
IMAPS:
Enabled: true
# Serve autoconfiguration/autodiscovery to simplify configuring email
# applications, will use port 443. Requires a TLS config. (optional)
AutoconfigHTTPS:
Enabled: true
# Serve MTA-STS policies describing SMTP TLS requirements. Requires a TLS config.
# (optional)
MTASTSHTTPS:
Enabled: true
# All configured WebHandlers will serve on an enabled listener. Either ACME must
# be configured, or for each WebHandler domain a TLS certificate must be
# configured. (optional)
WebserverHTTPS:
Enabled: true
# Destination for emails delivered to postmaster addresses: a plain 'postmaster'
# without domain, 'postmaster@' (also for each listener with SMTP
# enabled), and as fallback for each domain without explicitly configured
# postmaster destination.
Postmaster:
Account: d
# E.g. Postmaster or Inbox.
Mailbox: Postmaster
# Destination for per-host TLS reports (TLSRPT). TLS reports can be per recipient
# domain (for MTA-STS), or per MX host (for DANE). The per-domain TLS reporting
# configuration is in domains.conf. This is the TLS reporting configuration for
# this host. If absent, no host-based TLSRPT address is configured, and no host
# TLSRPT DNS record is suggested. (optional)
HostTLSRPT:
# Account to deliver TLS reports to. Typically same account as for postmaster.
Account: d
# Mailbox to deliver TLS reports to. Recommended value: TLSRPT.
Mailbox: TLSRPT
# Localpart at hostname to accept TLS reports at. Recommended value: tls-reports.
Localpart: tls-reports
```
.gitignoreDetails
This kept all of the most sensitive files out of the git repo, and also kept things like `mox` binary from being committed. ``` mox **/mox adminpasswd accounts/ acme/ queue/ *.key *.crt *.db ```