Issue with switching from LE staging to LE prod: 403 urn:acme:error:unauthorized: No registration exists matching provided key"

[ahmetb]

I was able to successfully follow the examples/gce/README.md on Google Container Engine and have my container serving with Fake LE Root X1 CA. I saw that the Kubernetes secret was created fine too.

Then I decided to switch from LE staging to LE prod. Here's what I did and at the end you'll see the continuous error from kube-lego:

0. kubectl delete secret echoserver-tls
1. kubectl edit configmap/kube-lego -n kube-lego
2. modify the url to https://acme-v01.api.letsencrypt.org/directory, save, exit
3. kubectl delete pods --all -n kube-lego so that new config map values get picked up
4. observe new kube-lego container started.
5. tail logs on kube-lego container, see repeated error:

time="2017-04-21T18:49:52Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=ng.alp.im
time="2017-04-21T18:49:52Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego
time="2017-04-21T18:49:52Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-04-21T18:49:52Z" level=info msg="process certificates requests for ingresses" context=kubelego
time="2017-04-21T18:49:52Z" level=info msg="creating new secret" context=secret name=echoserver-tls namespace=default
time="2017-04-21T18:49:52Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:49:52Z" level=info msg="requesting certificate for ng.alp.im" context="ingress_tls" name=echoserver namespace=default

time="2017-04-21T18:51:05Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=ng.alp.im
time="2017-04-21T18:51:05Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego
time="2017-04-21T18:51:05Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-04-21T18:51:05Z" level=info msg="process certificates requests for ingresses" context=kubelego
time="2017-04-21T18:51:05Z" level=info msg="creating new secret" context=secret name=echoserver-tls namespace=default
time="2017-04-21T18:51:05Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:51:05Z" level=info msg="requesting certificate for ng.alp.im" context="ingress_tls" name=echoserver namespace=default

time="2017-04-21T18:52:15Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=ng.alp.im
time="2017-04-21T18:52:15Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego
time="2017-04-21T18:52:15Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-04-21T18:52:15Z" level=info msg="process certificates requests for ingresses" context=kubelego
time="2017-04-21T18:52:15Z" level=info msg="creating new secret" context=secret name=echoserver-tls namespace=default
time="2017-04-21T18:52:15Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:52:15Z" level=info msg="requesting certificate for ng.alp.im" context="ingress_tls" name=echoserver namespace=default

time="2017-04-21T18:53:26Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=ng.alp.im
time="2017-04-21T18:53:26Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego
time="2017-04-21T18:53:26Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-04-21T18:53:26Z" level=info msg="process certificates requests for ingresses" context=kubelego
time="2017-04-21T18:53:26Z" level=info msg="creating new secret" context=secret name=echoserver-tls namespace=default
time="2017-04-21T18:53:26Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:53:26Z" level=info msg="requesting certificate for ng.alp.im" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:54:50Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=ng.alp.im

time="2017-04-21T18:54:50Z" level=error msg="Error while process certificate requests: no domain could be authorized successfully" context=kubelego
time="2017-04-21T18:54:50Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-04-21T18:54:50Z" level=info msg="process certificates requests for ingresses" context=kubelego
time="2017-04-21T18:54:50Z" level=info msg="creating new secret" context=secret name=echoserver-tls namespace=default
time="2017-04-21T18:54:50Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T18:54:50Z" level=info msg="requesting certificate for ng.alp.im" context="ingress_tls" name=echoserver namespace=default

...

Am I doing something wrong?

[ahmetb]

I actually started from scratch, and when I use https://acme-v01.api.letsencrypt.org/directory to configure kube-lego initally, I always get this error:

time="2017-04-21T20:18:32Z" level=info msg="requesting certificate for n.alp.im" context="ingress_tls" name=myngx namespace=default time="2017-04-21T20:19:47Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im

The moment I switch to staging back and restart the kube-lego pod (and change nothing else!), bam!, it succeeds:

time="2017-04-21T20:21:30Z" level=info msg="requesting certificate for n.alp.im" context="ingress_tls" name=myngx namespace=default
time="2017-04-21T20:21:34Z" level=info msg="authorization successful" context=acme domain=n.alp.im
time="2017-04-21T20:21:35Z" level=info msg="successfully got certificate: domains=[n.alp.im] url=https://acme-staging.api.letsencrypt.org/acme/cert/faa98539be6a0f591589261efe3c84022003" context=acme

[ahmetb]

I enabled LEGO_LOG_LEVEL=debug and seeing this:

time="2017-04-21T21:26:04Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T21:26:04Z" level=info msg="requesting certificate for n.alp.im" context="ingress_tls" name=echoserver namespace=default
time="2017-04-21T21:26:05Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:06Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:06Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:07Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:07Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:08Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:09Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:09Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:11Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:11Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:15Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:15Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:20Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:20Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:26Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:26Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:38Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:38Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:26:50Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:26:51Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:27:01Z" level=debug msg="testing reachablity of http://n.alp.im/.well-known/acme-challenge/_selftest" context=acme domain=n.alp.im
time="2017-04-21T21:27:01Z" level=debug msg="error while authorizing: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im
time="2017-04-21T21:27:38Z" level=warning msg="authorization failed after 1m0s: getting authorization failed: 403 urn:acme:error:unauthorized: No registration exists matching provided key" context=acme domain=n.alp.im

[repeats...]

Weird enough, this http://n.alp.im/.well-known/acme-challenge/_selftest thing works just fine on my machine and on a separate container:

curl -iv  http://n.alp.im/.well-known/acme-challenge/_selftest
* Hostname was NOT found in DNS cache
*   Trying 35.186.213.175...
* Connected to n.alp.im (35.186.213.175) port 80 (#0)
> GET /.well-known/acme-challenge/_selftest HTTP/1.1
> User-Agent: curl/7.38.0
> Host: n.alp.im
> Accept: */*
>

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Fri, 21 Apr 2017 21:32:27 GMT
Content-Length: 16
Via: 1.1 google

<
* Connection #0 to host n.alp.im left intact
uSHdnjZytkgQUJZO

I have no idea why kube-lego container sees a 403 from this _selftest. (I am not sure if 403 comes from _selftest either.)

[ankon]

Kube-Lego keeps some LE account data in a secret, and when switching the end points I needed to reset (aka remove) that secret.

[cpu]

No registration exists matching provided key

This seems like a case where your client is using an account key that was registered with the staging environment with the production environment. Accounts are not portable across Let's Encrypt staging & production, they are entirely separate systems.

[ahmetb]

@cpu I am not aware what an account key is. Is it my email address? Is it some secret resource that I should delete? Since I'm a user of this project and not low level LE concepts, I have no idea how to fix this problem. Any pointers are appreciated.

[cpu]

@ahmetb I'm sorry I can't help more directly. I'm coming at the problem from the other end with no experience with the project and only understanding of the low level concepts. I hope a maintainer/user can help you figure out a concrete solution.

[ahmetb]

OK I figured this out.

Turns out when you start a kube-lego container once, it creates a secret kube-lego-account in kube-lego namespace and it hardcodes the LE staging endpoint (and maybe some other stuff). Once you switch to from stage &rarr prod (or vice-versa) you have to delete this secret first and then make the KUBE_LEGO_URL change.

Keeping this issue open as it's not documented anywhere and figuring this out requires a somewhat non-beginner understanding of how things work (which, it shouldn't). Ideally if switching between LE endpoints requires deleting the secret/kube-lego-account, it should be documented. But I think the code can be refactored in a way that it replaces the saved secret based on the endpoint configured.

[ankon]

Indeed.

Note also https://github.com/jetstack/kube-lego/issues/125 - pretty much the same issue.

[ahmetb]

@ankon thanks. Also #43... I was going to open an issue to log the endpoint it uses, apparently it's already there.

[etoews]

Looks like it's now documented at https://github.com/jetstack/kube-lego#run-kube-lego

If you change the LEGO_URL, it is required that you delete the existing secret kube-lego-account and all certificates you want to request from the new URL.