acme/letsencrypt: Obtaining `extraDomainNames` certs fails if nginx default server is configured

[nh2]

I believe that the NixOS ACME module is somehow incompatible with defining an nginx default_server block.

My server uses this ACME config:

{
  # LetsEncrypt: Add `Subject Alt Name`s.
  security.acme.certs."nh2.me".extraDomainNames = [
    "muc.nh2.me"
    "uploads.xmpp.nh2.me"
  ];
}

This works by itself and renews its certificate. But the following improvement I tried to make to my nginx config breaks it:

In nginx, by default, the first server {} block configured will reply to requests for all domains, even those that you have not configured it to answer to. This is described in: https://serverfault.com/questions/420351/best-way-to-prevent-default-server

This is often not desired, because it results in seemingly random pages to be loaded. Instead, most people want to see HTTP 404 if a user accesses the server on a DNS domain that is not supposed to serve HTTP. For example, if I have the Matrix Synapse/Element chat server and UI enabled, going to muc.nh2.me in the browser will bring me to its UI, even though these two have nothing to do with each other!

Thus, I tried to usethe following default_server config to return HTTP code 404 on HTTP/HTTPs.

Unfortunately, that led to HTTP 404 being returned for the extraDomainNames validation requests!

{
  services.nginx = {
    enable = true;
    commonHttpConfig =
      let
        snakeoilCert = pkgs.runCommand "nginx-snakeoil-cert" {
          buildInputs = [ pkgs.openssl ];
        } ''
          mkdir "$out"
          openssl req -newkey rsa:4096 -x509 -sha256 -days 36500 \
            -subj '/CN=Snakeoil CA' -nodes \
            -out "$out/cert.pem" -keyout "$out/cert.key"
        '';
      in
        ''
          # Catch-all dummy default_servers so that we only serve domains
          # we actively declare in `server_name`s; see
          #   https://serverfault.com/questions/420351/best-way-to-prevent-default-server
          server {
            listen      80 default_server;
            listen [::]:80 default_server;
            server_name _;
            return 404;
          }
          # Note: Enabling this catch-all SSL dummy default_server breaks HTTPS
          # clients without SNI, see
          #   https://trac.nginx.org/nginx/ticket/195#comment:11
          # That's OK for us, as we require SNI anyway since we run multiple
          # domains on the same IP.
          # That's also why we use `proxy_ssl_server_name on;` for all `proxy_pass`
          # settings.
          server {
            listen      443 ssl default_server;
            listen [::]:443 ssl default_server;
            server_name _;
            # Make clients that connect to this fail with an SSL error
            # by disabling all ciphers.
            ssl_ciphers aNULL;
            # nginx forces us to declare an SSL certificate, so we just
            # give a self-signed snakeoil certificate (as above we will
            # make the connection fail anyway due to empty allowed ciphers).
            ssl_certificate ${snakeoilCert}/cert.pem;
            ssl_certificate_key ${snakeoilCert}/cert.key;
            return 404;
          }
        '';

The LetsEncrypt error I got during NixOS system activation:

Jul 10 12:02:06 nh2me acme-nh2.me-start[6723]: [muc.nh2.me] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: MY_IP: Invalid response from http://muc.nh2.me/.well-known/acme-challenge/...: 404
Jul 10 12:02:06 nh2me acme-nh2.me-start[6723]: [uploads.xmpp.nh2.me] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: MY_IP: Invalid response from http://uploads.xmpp.nh2.me/.well-known/acme-challenge/...: 404

Expected behavior

Renewing extraDomainNames should work even with an nginx default server configured.

Notify maintainers

CC @m1cr0man wo is knowledgeable in ACME

[nh2]

I believe that the fix should be something like this:

The definition of a extraDomainNames should result in the rendering of an acmeLocation block into nginx.conf as defined here:

https://github.com/NixOS/nixpkgs/blob/71d7a4c037dc4f3e98d5c4a81b941933cf5bf675/nixos/modules/services/web-servers/nginx/default.nix#L277

[nh2]

I use this NixOS config as a workaround instead, restricting the return 404 (that was before on the server level) to be inside location /:

{
  services.nginx = {
    enable = true;
    virtualHosts = {

      # Configure dummy default servers that return `404 Not Found`.
      # Without this, an arbitrary vhost will be picked for answering requests
      # that do not match any configured vhost (namely, the first one rendered into `nginx.conf`)!
      # See:
      #     https://serverfault.com/questions/420351/best-way-to-prevent-default-server
      # We must allow the `/.well-known/acme-challenge` to not break LetsEncrypt ACME `extraDomainNames`, see issue:
      #     acme/letsencrypt: Obtaining extraDomainNames certs fails if nginx default server is configured
      # (https://github.com/NixOS/nixpkgs/issues/180980).
      # The root is the default of `security.acme.certs.<name>.webroot`, see:
      # * https://github.com/NixOS/nixpkgs/blob/09b76341b3eb1e3f72ebfc29c7fe04c20bdb92de/nixos/modules/security/acme/default.nix#L468
      # * https://github.com/NixOS/nixpkgs/blob/71d7a4c037dc4f3e98d5c4a81b941933cf5bf675/nixos/modules/services/web-servers/nginx/default.nix#L280
      "defaultDummy404" = {
        default = true;
        serverName = "_";
        locations."/".extraConfig = "return 404;";
        locations."/.well-known/acme-challenge".root = "/var/lib/acme/acme-challenge";
      };
      "defaultDummy404ssl" =
        let
          # A self-signed certificate, just to render "Not Found" messages.
          snakeoilCert = pkgs.runCommand "nginx-snakeoil-cert" { buildInputs = [ pkgs.openssl ]; } ''
            mkdir "$out"
            openssl req -newkey rsa:4096 -x509 -sha256 -days 36500 -subj '/CN=Snakeoil CA' -nodes -out "$out/cert.pem" -keyout "$out/cert.key"
          '';
        in
          # Note: Enabling this catch-all SSL dummy default_server breaks HTTPS clients without SNI, see
          #   https://trac.nginx.org/nginx/ticket/195#comment:11
          # That's OK for us, as we require SNI anyway since we run multiple domains on the same IP.
          # That's also why we use `proxy_ssl_server_name on;` for all `proxy_pass` settings.
          {
            default = true;
            serverName = "_";
            locations."/".extraConfig = "return 404;";
            locations."/.well-known/acme-challenge".root = "/var/lib/acme/acme-challenge";
            # Dummy SSL config
            onlySSL = true;
            sslCertificate = "${snakeoilCert}/cert.pem";
            sslCertificateKey = "${snakeoilCert}/cert.key";
          };

    };
  };
}

[m1cr0man]

Hello. I've gotten some time to look into this. I'm really struggling to understand the problem + proposed solution, but I think the key thing here is that in your situation there is no vhost configured for one or all of the extraDomainNames that you expect to validate via HTTP-01? This feels like expected behaviour, or should I say that not specifying explicit vhosts for HTTP-01 solving has unintentionally worked.

Your workaround looks like the best solution regardless.