Automatic TLS with LetsEncrypt

emersion commented 7 years ago

Just like Caddy.

emersion commented 5 years ago

https://github.com/mholt/certmagic

foxcpp commented 5 years ago

I think we can get it, but with some important caveats:

Users must be able to change the CertMagic storage directory to avoid clashes with other certificate configurations (caddy, manual management).
We need HTTP endpoint module to respond to challenges (this also leads to a slightly different design, see #67 issue for details)
Don't agree for EULA automatically, require users to explicitly add "agreed" flag.

tls auto {
  filestorage ...
  agreed
}

emersion commented 5 years ago

Don't agree for EULA automatically, require users to explicitly add "agreed" flag.

It would be interesting to know what Caddy's stance about this is.

foxcpp commented 5 years ago

It would be interesting to know what Caddy's stance about this is.

It does it automatically.

emersion commented 5 years ago

Yeah, I meant: have people discussed whether it's okay/legal/desirable/etc?

foxcpp commented 5 years ago

Let's Encrypt Subscriber Agreement

Caddy may prompt to agree to Let's Encrypt Subscriber Agreement. This is configurable with ACME_AGREE environment variable. Set it to true to agree. ACME_AGREE=true.

Or I'm wrong...

foxcpp commented 4 years ago

So far the fundamental problem with automatic TLS is that ACME is built mostly for HTTP(S) servers and that is visible as 2 out of 3 challenges using HTTP(S) ports. maddy is not a HTTP(S) server and making it one (or in reverse, somebody proposed to make maddy a plugin for caddy) is going to push project complexity beyond what I believe is acceptable. In other words, the only challenge I see feasible to be implemented is a DNS-based one. This will also require per-provider plugins for a magnitude of proprietary interfaces.

I welcome work in this direction, somebody needs to wire certmagic library into maddy's TLS certificate loading code. That is how I can see it being used:

smtp tcp://0.0.0.0:25 {
  tls auto mx.example.org
  ...
}

This needs some refactoring in TLS certificate loading to only "load" it if any endpoints are initialized. This is important as maddyctl reads server configuration and that has a side effect of triggering TLS cert. path.

One might also consider implementing On-Demand TLS, but I have concerns about performance of DNS challenge and how that might affect user experience.

emersion commented 4 years ago

In other words, the only challenge I see feasible to be implemented is a DNS-based one

The DNS challenge is pretty lame, due to the lack of standardized DNS provider API.

Another alternative would be to grab certificates from a local certificate daemon, but this doesn't exist, yet.

foxcpp commented 4 years ago

The DNS challenge is pretty lame, due to the lack of standardized DNS provider API.

It kind of exists, RFC 2136. Unfortunately it is not widely supported :(

foxcpp commented 4 years ago

It might be possible to also support HTTP but it will not be magically working in configurations where there is also an HTTP server running. If it is running - there will be a need to configure maddy to use a separate port to serve challenge responses and have running HTTP server proxy requests. That might involve proxying based on Host header field if certbot/etc is also used to issue certificates for other domains (all bets are off if MX is using the same domain). But I think in this setup it might be also possible to just reuse the same certificate management solution but hard-link or copy certificates/keys to make them accessible by maddy. In fact, I do this for @hexanet.dev mail. And we expect people to have HTTP server running, especially if we ever get JMAP in (massive amount of work required, yeah...).

I see an JMAP-enabled maddy setup with webmail from a far future as follows:

There is a HTTPS server running for mail.example.org
Certificates are acquired using ACME (HTTP challenge) by certbot or functionality built into server itself (e.g. Caddy)
JMAP API endpoint is reverse proxied to maddy
Certificates are copied into maddy config directory so it can use them for TLS on IMAP/SMTP. [*] [**]
The same HTTPS server is serving JMAP-based webmail frontend

[*] If reverse proxy supports it and we have XCLIENT extension for SMTP in maddy, it might be possible to have proxy do TLS termination and leave certificate management completely out of maddy. [**] Copying of certs/keys can be replaced by proxying ACME challenge endpoint and having maddy get certificates for itself but I do not feel like it justifies the additional complexity.

And of course alternative to this is DNS-based challenge for a separate domain (e.g. mx.example.org) and maddy being completely independent of HTTP business (if there is JMAP involved - it just works as a usual webapp backend, accepting proxied requests). IMO this is the most clean solution even if it involves having dozens of per-provider components. Fortunately Caddy developers already wrote a large collection of such components.

foxcpp commented 4 years ago

Experimental implementation using CertMagic library and dns-01 challenge is in acme branch. Getting into dev tree is blocked by https://github.com/caddyserver/certmagic/issues/19.

mholt commented 4 years ago

Just a heads-up... I've completely rewritten CertMagic's underlying ACME library from scratch -- it's really quite good though, based on years of experience, and definitely the best one written in Go -- an improvement over what we were using before. It might change a feeeew little things with the exported CertMagic API but nothing major. One benefit is the logging is now (meaning, the commit I'm going to push later this week) configurable using zap.

PS. I do think a mail server would make a fine Caddy 2 module (could be an app module just like the HTTP server or TLS app which manages certificates). Once you get nestled into the JSON config it's pretty comfy: https://caddyserver.com/docs/extending-caddy

foxcpp commented 3 years ago

First case - ACME, http-01 challenge

tls.loader.acme local_tls {
    hostname $(hostname)
    challenge http-01
}

tls &local_tls # set default TLS config for all endpoints

# Configure reverse-proxy on port 80 to forward ACME challenges here
http tcp://127.0.0.1:8080 {
    serve acme_http_01 &local_tls # or just 'serve &local_tls'
}

Second case - ACME, dns-01 challenge

tls.loader.acme local_tls {
    hostname $(hostname)
    challenge dns-01
    # provider configuration is done via env-vars
}

tls &local_tls # set default TLS config for all endpoints

Third case - Manual certs with OCSP stapling

# Handled automatically if we revamp tls.loader.file to use Certmagic
tls file /etc/maddy/cert.pem /etc/maddy/cert.key

emersion commented 3 years ago

I'd advise against supporting the http-01 challenge, and instead for supporting the tls-alpn-01 challenge. Both are equivalent from a high-level point of view, but the latter doesn't rely on HTTP.

foxcpp commented 3 years ago

tls-alpn-01 can be definitely supported but it has an unfortunate limitation of requiring exclusive control over 443 port which maddy might not have.

foxcpp commented 3 years ago

Therefore I also propose supporting http-01 - if there is an web server running already and using both 80 and 443 ports then it can just proxy ACME endpoint to maddy so it can issue certificates for itself.

emersion commented 3 years ago

Hm. That is unfortunate.

FWIW, forwarding tls-alpn-01 to backend servers is something I'm planning to implement in tlstunnel. Since SMTP pretty much requires STARTTLS, so requires the mail server to handle TLS certificates, I'm wondering if it would make sense to have a very thin SMTP parser in tlstunnel that waits for the first SMTP command, upgrades to TLS if it's STARTTLS, then forwards to the email server.

foxcpp commented 3 years ago

Handling TLS termination in a SMTP-aware way is definitely making it possible to keep all TLS management in tlstunnel but it would require more than you described:

ESMTP capabilities exposed by backend server should be forwarded back to the client in response to EHLO
Client might proceed to issue MAIL FROM - in this case tlstunnel would have to act as a plain text proxy
Mail server needs knowledge of the client IP and whether TLS was negotiated (see 2), while former can be implemented using HAProxy's PROXY protocol (#369), I am not sure what to do for the latter.

emersion commented 3 years ago

Yep, these are good points.

Mail server needs knowledge of the client IP and whether TLS was negotiated (see 2), while former can be implemented using HAProxy's PROXY protocol (#369), I am not sure what to do for the latter.

The PROXY protocol allows indicating whether TLS is in use by the frontend client via PP2_TYPE_SSL.

foxcpp commented 3 years ago

The PROXY protocol allows indicating whether TLS is in use by the frontend client via PP2_TYPE_SSL.

Oh, that's nice. Well, other than being cumbersome there is no problem.

To be honest, I really like (possibly opinionated) approach of replacing all certificate management solutions with tlstunnel and would gladly work on SMTP-aware mode support in tlstunnel if you believe this is worth it.

emersion commented 3 years ago

Yeah, it's quite an opinionated approach, hopefully not completely incompatible with other approaches so in the end users can choose. There are downsides to the tlstunnel approach too, for instance domain names need to be configured in both tlstunnel and the backend server (see this issue for brainstorming for fixing this).

foxcpp commented 3 years ago

At least for SMTP this is easily solvable: SMTP server identifies itself in the initial greeting.

foxcpp commented 3 years ago

I am working on implementing tls.loader.acme with support for dns-01 challenge using the most recent CertMagic library version. Since CertMagic moved from using go-acme/lego to acmez, support for dns providers will be more limited (18 providers) and does not include RFC 2136. :(

foxcpp commented 3 years ago

tls.loader.acme implementation is available on the dev branch for testing. See maddy-tls(5) for details.

foxcpp / maddy