search

jetstack / kube-lego

DEPRECATED: Automatically request certificates for Kubernetes Ingress resources from Let's Encrypt
Apache License 2.0
2.16k stars 267 forks source link

Stuck in a loop, updates ingress every second #198

Closed ahmetb closed 7 years ago

ahmetb commented 7 years ago

I think this issue is critical, it might be stressing GCE API and eventually could cause all LB requests to be throttled.

First I got a certificate:

time="2017-06-01T18:04:45Z" level=info msg="requesting certificate for echo.ahmet.im" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:04:46Z" level=debug msg="testing reachability of http://echo.ahmet.im/.well-known/acme-challenge/_selftest" context=acme domain=echo.ahmet.im
time="2017-06-01T18:04:46Z" level=debug msg="responding to challenge request" basePath="/.well-known/acme-challenge" context=acme host=echo.ahmet.im token="zhe0hiA4Cr2zGlF8QSQOzTE_vWZPa024HSY_8KnIXZw"
time="2017-06-01T18:04:48Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/P568Kb-oVQL5_InzR_0LATQ2kWJGGdqwrVzSrUJ7QO4/1263610806 Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=echo.ahmet.im
time="2017-06-01T18:04:48Z" level=info msg="authorization successful" context=acme domain=echo.ahmet.im
time="2017-06-01T18:04:48Z" level=info msg="successfully got certificate: domains=[echo.ahmet.im] url=https://acme-v01.api.letsencrypt.org/acme/cert/038cfb487e9da9d58f14034f7a908f60f0a1" context=acme
time="2017-06-01T18:04:48Z" level=debug msg="certificate pem data:\n-----BEGIN CERTIFICATE-----\nMIIE/TCCA+..............................................................................................................................................................................................................................................................................==\n-----END CERTIFICATE-----\n" context=acme
time="2017-06-01T18:04:48Z" level=info msg="Attempting to create new secret" context=secret name=echoserver-tls namespace=default
time="2017-06-01T18:04:48Z" level=info msg="Secret successfully stored" context=secret name=echoserver-tls namespace=default
time="2017-06-01T18:04:48Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:04:48Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:04:48Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:04:48Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:04:48Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:04:48Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:04:48Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:04:48Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:04:48Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:04:48Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:04:48Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:04:48Z" level=debug msg="worker: done processing true" context=kubelego

7 minutes later, suddenly things started to go crazy and my kubectl get ingress -w started updating every second indefinitely:

time="2017-06-01T18:11:04Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:04Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:04Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:04Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:04Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:04Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:04Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:04Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:04Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:04Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:04Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:04Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:04Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:04Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:04Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:04Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:04Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:05Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:05Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:05Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:05Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:05Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:05Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:05Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:05Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:05Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:05Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:05Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:05Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:05Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:05Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:05Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:06Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:06Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:06Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:06Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:06Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:06Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:06Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:06Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:06Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:06Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:06Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:06Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:06Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:07Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:07Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:07Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:07Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:07Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:07Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:07Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:07Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:07Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:08Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:08Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:08Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:08Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:08Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:08Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:08Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:08Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:08Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:08Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:08Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:08Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:08Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:08Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:09Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:09Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:09Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:09Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:09Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:09Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:09Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:09Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:09Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:10Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:10Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:10Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:10Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:10Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:11Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:11Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:11Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:11Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:11Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:11Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:11Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:11Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:11Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:11Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:12Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:12Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:12Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:12Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:12Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:12Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:12Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:12Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:12Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:12Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:12Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:12Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:12Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:13Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:13Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:13Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:13Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:13Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:13Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:13Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:13Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:13Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:14Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:14Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:14Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:14Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:14Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:14Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:14Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:14Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:14Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:14Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:14Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:14Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:15Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:15Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:15Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:15Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:15Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:15Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:15Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:15Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:15Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:15Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:15Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:16Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:16Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:16Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:16Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:16Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:17Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:17Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:17Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:17Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:17Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:17Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:17Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:17Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:17Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:17Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:18Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:18Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:18Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:18Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:18Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:18Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:18Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:18Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:18Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:18Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:18Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:18Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:18Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:19Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:19Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:19Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:19Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:19Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:19Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:19Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:19Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:19Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:20Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:20Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:20Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:20Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:20Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:20Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:20Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:20Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:20Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:20Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:20Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:20Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:21Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:21Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:21Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:21Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:21Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:21Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:21Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:21Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:21Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:21Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:21Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:22Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:22Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:22Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:22Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:22Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:23Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:23Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:23Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:23Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:23Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:23Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:23Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:23Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:23Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:23Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:23Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:24Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:24Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:24Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:24Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:24Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:24Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:24Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:24Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:24Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:25Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:25Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:25Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:25Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:25Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:25Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:25Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:25Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:25Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:25Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:25Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:25Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:26Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:26Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:26Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:26Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:26Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:26Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:26Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:26Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:26Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:26Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:26Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:26Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce

time="2017-06-01T18:11:27Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:27Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:27Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:27Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:27Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:27Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:27Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:27Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:27Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:27Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:27Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:28Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:28Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:28Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:28Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:28Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:29Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:29Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:29Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:29Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:29Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:29Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:29Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:29Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce

time="2017-06-01T18:11:29Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:29Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:30Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:30Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:30Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:30Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:30Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:30Z" level=debug msg="worker: begin processing true" context=kubelego
time="2017-06-01T18:11:30Z" level=debug msg=reset context=provider provider=gce
time="2017-06-01T18:11:30Z" level=debug msg="UPDATE ingress/default/echoserver" context=kubelego
time="2017-06-01T18:11:30Z" level=debug msg=finalize context=provider provider=gce
time="2017-06-01T18:11:30Z" level=debug msg="setting up svc endpoint" context=provider namespace=default pod_ip=10.4.6.7 provider=gce
time="2017-06-01T18:11:31Z" level=debug msg=reset context=provider provider=nginx
time="2017-06-01T18:11:31Z" level=debug msg=finalize context=provider provider=nginx
time="2017-06-01T18:11:31Z" level=info msg="disable provider no TLS hosts found" context=provider provider=nginx
time="2017-06-01T18:11:31Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-06-01T18:11:31Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-08-30 17:05:00 +0000 UTC name=echoserver namespace=default
time="2017-06-01T18:11:31Z" level=info msg="no cert request needed" context="ingress_tls" name=echoserver namespace=default
time="2017-06-01T18:11:31Z" level=debug msg="worker: done processing true" context=kubelego
time="2017-06-01T18:11:31Z" level=debug msg="worker: begin processing true" context=kubelego
[...]

In the events I see this message flooding as well:

default   2017-06-01 11:14:49 -0700 PDT   2017-06-01 10:58:06 -0700 PDT   57        echoserver   Ingress             Normal    Service   loadbalancer-controller   no user specified default backend, using system default
default   2017-06-01 11:14:54 -0700 PDT   2017-06-01 10:58:06 -0700 PDT   58        echoserver   Ingress             Normal    Service   loadbalancer-controller   no user specified default backend, using system default
default   2017-06-01 11:14:58 -0700 PDT   2017-06-01 10:58:06 -0700 PDT   59        echoserver   Ingress             Normal    Service   loadbalancer-controller   no user specified default backend, using system default

My ingress:

Name:           echoserver
Namespace:      default
Address:        35.190.20.86
Default backend:    default-http-backend:80 (10.4.4.7:8080)
TLS:
  echoserver-tls terminates echo.ahmet.im
Rules:
  Host      Path    Backends
  ----      ----    --------
  echo.ahmet.im
            /.well-known/acme-challenge/*   kube-lego-gce:8080 (<none>)
            /*              echoserver:80 (<none>)
Annotations:
  backends:         {"k8s-be-31106--5d6b2c3481ed4706":"HEALTHY","k8s-be-31218--5d6b2c3481ed4706":"HEALTHY","k8s-be-31859--5d6b2c3481ed4706":"HEALTHY"}
  https-target-proxy:       k8s-tps-default-echoserver--5d6b2c3481ed4706
  static-ip:            k8s-fw-default-echoserver--5d6b2c3481ed4706
  url-map:          k8s-um-default-echoserver--5d6b2c3481ed4706
  forwarding-rule:      k8s-fw-default-echoserver--5d6b2c3481ed4706
  https-forwarding-rule:    k8s-fws-default-echoserver--5d6b2c3481ed4706
  ssl-cert:         k8s-ssl-default-echoserver--5d6b2c3481ed4706
  target-proxy:         k8s-tp-default-echoserver--5d6b2c3481ed4706
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason  Message
  --------- --------    -----   ----            -------------   --------    ------  -------
  18m       18m     1   loadbalancer-controller         Normal      ADD default/echoserver
  17m       17m     1   loadbalancer-controller         Normal      CREATE  ip: 35.190.20.86
  17m       4s      63  loadbalancer-controller         Normal      Service no user specified default backend, using system default

Deleting kube-lego pod and having it restart did not solve the issue.

I must also note that my site (https://echo.ahmet.im) is working fine. However flooding continues.

Kenblair1226 commented 7 years ago

137

munnerz commented 7 years ago

@ahmetb thanks for the bug report - I've commented over on #137 (here: https://github.com/jetstack/kube-lego/issues/137#issuecomment-305745273). I'm going to close this as a dupe, but if you disagree then please re-open 😄