[Question] Delegating acme-challenge to secondary server

[felixbuenemann]

I'm trying to use lua-resty-auto-ssl 0.12.1 with two openresty instances in different regions that are using dns-based failover with health checks.

I'm having trouble getting the backup server to register certificates (likely for renewal), while it is not the active server, because obviously acme-challenges are hitting the wrong server.

My idea was to change the nginx config, so that the auto_ssl:challenge_server() is in a separate server block and then use an upstream block with the local acme server as primary and remote acme server as backup and enable retries on 404.

However, for some reason this doesn't work, even though the failover on 404 from the local challenge server appears to be working.

Here's a reduced config, with all the non-essential setup omitted:

http {
    upstream acme_servers {
        server 127.0.0.1:8998;
        server backup.example.org:80 backup;
    }
    # default web server
    server {
        listen 80;
        location /.well-known/acme-challenge/ {
            # This works:
            #content_by_lua_block {
            #    auto_ssl:challenge_server()
            #}
            # This does not:
            proxy_next_upstream http_404;
            proxy_pass http://acme_servers;
            break;
        }
    }
    # auto-ssl acme server
    server {
        listen 127.0.0.1:8998;
        location /.well-known/acme-challenge/ {
            content_by_lua_block {
                auto_ssl:challenge_server()
            }
        }
    }
}

In theory this should work fine:

When adding/renewing a cert on the backup server, the acme-challenge request from the Let's Encrypt validations server should hit the primary server, which would get a 404 from its local acme server and then retry against the backup server in the upstream config.

However, in practice this doesn't work and both the primary and secondary acme server return a 404.

Any idea what's going wrong or how this could be solved better?

If possible I'd like to avoid the overhead of adding a redis-cluster with failover just to share the certs.

Another idea would be to use dns-01 challenge and update records in route 53, but it seems that's not currently possible with lua-resty-auto-ssl.

[sebastianmacarescu]

I kinda have a similar issue. I'm running multiple docker containers behind AWS NLB and the certificate generation always fails with acme challenge not found. I think the challenge server uses local storage instead of redis.

[aviatrix]

I think the challenge server uses local storage instead of redis.

That's correct, that is the default behavior. To have multiple instances on different hosts/containers access the same ssl certificates you need to use Redis or mounted shared folder and mount it to /etc/ssl

[felixbuenemann]

Btw. it is also trivial to implement acme challenge in pure nginx using a regex location match and the account key fingerprint:

location ~ ^/\.well-known/acme-challenge/([-_a-zA-Z0-9]+)$ {
    default_type text/plain;
    return 200 "$1.6fXAG9VyG0IahirPEU2ZerUtItW2DHzDzD9wZaEKpqd";
}

Where 6fXA… gets replaced by the actual fingerprint. Of course this only works, if the same account is used for all certs.