search

jetstack / kube-lego

DEPRECATED: Automatically request certificates for Kubernetes Ingress resources from Let's Encrypt
Apache License 2.0
2.16k stars 267 forks source link

waiting for authorization failed: acme: identifier authorization failed after tearing down and recreating cluster #199

Closed jonaseck2 closed 7 years ago

jonaseck2 commented 7 years ago

I was asked to do a small presentation about kubernetes and and basically created a site containing a presentation how to set up itself using google container engine from scratch. I added kube-lego as per the GCE example (documented in the presentation too, of course) and it worked like a charm. After a couple of days i wanted to test reproducibility, tore down and recreated the cluster, and now requesting certificates fail and i hit the rate limiter whenever i start kube-lego.

Steps to reproduce are available at the host name mentioned in the logs below :)

logs from kube-lego from self test passing to rate limiter hit:

2017-06-02T21:41:57.767748661Z time="2017-06-02T21:41:57Z" level=debug msg="error while authorizing: reachability test failed: wrong status code '502'" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:08.92893298Z time="2017-06-02T21:42:08Z" level=debug msg="testing reachability of http://kubernetes.rocks/.well-known/acme-challenge/_selftest" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:09.631601384Z time="2017-06-02T21:42:09Z" level=debug msg="responding to challenge request" basePath="/.well-known/acme-challenge" context=acme host=kubernetes.rocks token=8QZ11hiqJXk4HrsmVVZlOPvmMuCaJEpEmam1eKLQI-c 
2017-06-02T21:42:11.049206715Z time="2017-06-02T21:42:11Z" level=debug msg="error while authorizing: waiting for authorization failed: acme: identifier authorization failed" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:31.00203884Z time="2017-06-02T21:42:31Z" level=debug msg="testing reachability of http://kubernetes.rocks/.well-known/acme-challenge/_selftest" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:31.480852416Z time="2017-06-02T21:42:31Z" level=debug msg="responding to challenge request" basePath="/.well-known/acme-challenge" context=acme host=kubernetes.rocks token="6n96XpZSVwbzVecJ9OZA-MeUsxkxTY_MDNaM9qXuM54" 
2017-06-02T21:42:33.192175385Z time="2017-06-02T21:42:33Z" level=debug msg="error while authorizing: waiting for authorization failed: acme: identifier authorization failed" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:33.192210207Z time="2017-06-02T21:42:33Z" level=warning msg="authorization failed after 1m0s: waiting for authorization failed: acme: identifier authorization failed" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:33.192238738Z time="2017-06-02T21:42:33Z" level=error msg="Error while processing certificate requests: no domain could be authorized successfully" context=kubelego 
2017-06-02T21:42:33.192243832Z time="2017-06-02T21:42:33Z" level=debug msg="worker: done processing true" context=kubelego 
2017-06-02T21:42:33.192249919Z time="2017-06-02T21:42:33Z" level=debug msg="worker: begin processing true" context=kubelego 
2017-06-02T21:42:33.195780799Z time="2017-06-02T21:42:33Z" level=debug msg=reset context=provider provider=gce 
2017-06-02T21:42:33.20068869Z time="2017-06-02T21:42:33Z" level=debug msg=finalize context=provider provider=gce 
2017-06-02T21:42:33.200700756Z time="2017-06-02T21:42:33Z" level=debug msg="UPDATE ingress/default/kubernetes-presentation-ingress" context=kubelego 
2017-06-02T21:42:33.202819556Z time="2017-06-02T21:42:33Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.124.8.3 provider=gce 
2017-06-02T21:42:33.211175395Z time="2017-06-02T21:42:33Z" level=debug msg=reset context=provider provider=nginx 
2017-06-02T21:42:33.21118727Z time="2017-06-02T21:42:33Z" level=debug msg=finalize context=provider provider=nginx 
2017-06-02T21:42:33.214224797Z time="2017-06-02T21:42:33Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx 
2017-06-02T21:42:33.214237025Z time="2017-06-02T21:42:33Z" level=info msg="process certificate requests for ingresses" context=kubelego 
2017-06-02T21:42:33.215801204Z time="2017-06-02T21:42:33Z" level=info msg="Attempting to create new secret" context=secret name=kubernetes-presentation-tls namespace=default
2017-06-02T21:42:33.215812457Z time="2017-06-02T21:42:33Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=kubernetes-presentation-ingress namespace=default 
2017-06-02T21:42:33.215816891Z time="2017-06-02T21:42:33Z" level=info msg="requesting certificate for kubernetes.rocks" context="ingress_tls" name=kubernetes-presentation-ingress namespace=default 
2017-06-02T21:42:33.625283939Z time="2017-06-02T21:42:33Z" level=debug msg="testing reachability of http://kubernetes.rocks/.well-known/acme-challenge/_selftest" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:34.429604354Z time="2017-06-02T21:42:34Z" level=debug msg="responding to challenge request" basePath="/.well-known/acme-challenge" context=acme host=kubernetes.rocks token="OqpJQZoUdv84NAA15Gng4pHI1BUgzeikuJT_gYxUIyA" 
2017-06-02T21:42:36.598596642Z time="2017-06-02T21:42:36Z" level=debug msg="error while authorizing: waiting for authorization failed: acme: identifier authorization failed" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:36.923244431Z time="2017-06-02T21:42:36Z" level=debug msg="testing reachability of http://kubernetes.rocks/.well-known/acme-challenge/_selftest" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:38.399489775Z time="2017-06-02T21:42:38Z" level=debug msg="responding to challenge request" basePath="/.well-known/acme-challenge" context=acme host=kubernetes.rocks token=XEz9niHNY3r714z8btd3cfi5In-apA5oWvq6p9G2SMM 
2017-06-02T21:42:39.066327449Z time="2017-06-02T21:42:39Z" level=debug msg="error while authorizing: waiting for authorization failed: acme: identifier authorization failed" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:39.697006796Z time="2017-06-02T21:42:39Z" level=debug msg="testing reachability of http://kubernetes.rocks/.well-known/acme-challenge/_selftest" context=acme domain=kubernetes.rocks 
2017-06-02T21:42:39.920717523Z time="2017-06-02T21:42:39Z" level=debug msg="error while authorizing: getting authorization failed: 429 urn:acme:error:rateLimited: Error creating new authz :: Too many invalid authorizations recently." context=acme domain=kubernetes.rocks 
2017-06-02T21:42:41.150655045Z time="2017-06-02T21:42:41Z" level=debug msg="testing reachability of http://kubernetes.rocks/.well-known/acme-challenge/_selftest" context=acme domain=kubernetes.rocks
munnerz commented 7 years ago

There's nothing on the kube-lego end that can be done to stop this. Letsencrypt outline their rate limits here: https://letsencrypt.org/docs/rate-limits/.

Could this perhaps be due to your domain still point at the old ingress IP, and DNS propagation taking a while, thus Letsencrypt is still making requests to another address and causing invalid challenges?

I'm going to close this for now as it doesn't seem to be something that kube-lego can deal with. Feel free to reopen if you disagree!