search

jetstack / kube-lego

DEPRECATED: Automatically request certificates for Kubernetes Ingress resources from Let's Encrypt
Apache License 2.0
2.16k stars 267 forks source link

Do not loop on account creation when RBAC is failing #229

Open apognu opened 7 years ago

apognu commented 7 years ago

Hi,

I just migrated a working Kubernetes + kube-lego (0.1.5) configuration to a new cluster, and got my RBAC rules mixed up. As a result, kube-lego was unable to persist the account data received through ACME to the secret.

Instead of retrying persisting it (waiting for permissions to change), kube-lego looped over the account creation request (until I fixed my RBAC configuration), resulting in my IP being rate-limited from LE. I'm running on GCE, so I just got a new public IP address, but I can see how that can become an issue.

What's the intended behavior here? I understand proper operation needs a valid and persisted LE account, but instead of continuously creating an account on ACME and fail to store it, should kube-lego just try to create the secret from the created account until it actually can?

Regards, Antoine.

munnerz commented 7 years ago

In this scenario I think we should back off exponentially on account creation, and be quite conservative with requests.

Really, we need to define our own RBAC policy within this repo to solve issues like this once and for all.

It'd be great if kube-lego had a validation stage on start up, where it checks to make sure it has the required permissions to start...

abh commented 7 years ago

It should be backing off exponentially on any error (rate limits etc), I think.

timasplin commented 7 years ago

I just ran into this issue. The RBAC was not set up properly which resulted in multiple calls being made which resulted in IP address being blocked. Unlike 'apognu' we are not able to change IP so now have to wait.

Sample of failing requests:

time="2017-09-01T11:43:55Z" level=info msg="Attempting to create new secret" context=secret name=kube-lego-account namespace=default
time="2017-09-01T11:43:55Z" level=error msg="Error while processing certificate requests: 429 urn:acme:error:rateLimited: Error creating new registration :: too many registrations for this IP" context=kubelego
time="2017-09-01T11:43:55Z" level=info msg="Attempting to create new secret" context=secret name=kube-lego-account namespace=default
time="2017-09-01T11:43:56Z" level=error msg="Error while processing certificate requests: 429 urn:acme:error:rateLimited: Error creating new registration :: too many registrations for this IP" context=kubelego
time="2017-09-01T11:43:56Z" level=info msg="Attempting to create new secret" context=secret name=kube-lego-account namespace=default
time="2017-09-01T11:43:57Z" level=error msg="Error while processing certificate requests: 429 urn:acme:error:rateLimited: Error creating new registration :: too many registrations for this IP" context=kubelego
time="2017-09-01T11:43:57Z" level=info msg="Attempting to create new secret" context=secret name=kube-lego-account namespace=default
time="2017-09-01T11:43:58Z" level=error msg="Error while processing certificate requests: 429 urn:acme:error:rateLimited: Error creating new registration :: too many registrations for this IP" context=kubelego
time="2017-09-01T11:43:58Z" level=info msg="Attempting to create new secret" context=secret name=kube-lego-account namespace=default
time="2017-09-01T11:43:58Z" level=error msg="Error while processing certificate requests: 429 urn:acme:error:rateLimited: Error creating new registration :: too many registrations for this IP" context=kubelego

Letsencrypt limits 50 account requests per IP, and then blocks for 3 hours, so hit the limit very quickly.