search

kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
16.98k stars 8.15k forks source link

Ingress Controller switches to Fake certificate after a few hours #4199

Closed SpicySyntax closed 4 years ago

SpicySyntax commented 5 years ago

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): ERR_CERT_AUTHORITY_INVALID Fake Certificate

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report

NGINX Ingress controller version: 0.24.1

Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-25T02:41:57Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Environment: Azure Kubernetes Services

What happened: I setup my cluster to use Ingress Nginx for https. (using lets encrypt for dynamic IP). It creates and issues the cert properly and I am able to use https successfully in tests and from a deployed website. However after a few hours the controller starts using the default fake cert and then my website deployment no longer works with: image

What you expected to happen: The Ingress controller to continue to use valid certs to serve over https

How to reproduce it (as minimally and precisely as possible): Deploy Ingress Controller into AKS cluster in front of an HTTP service. (see here)

aledbf commented 5 years ago

@SpicySyntax please post the ingress controller pod logs

SpicySyntax commented 5 years ago

@aledbf

Controller 1

10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:34:46 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 408 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 126 60.002 [azure-functions-crack-detection-deployment-service-80] - - - - 44bd16611ec53926df9717866e5a31f5 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:34:46 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 408 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 124 60.002 [azure-functions-crack-detection-deployment-service-80] - - - - e1d39bfb2079bb26da28122064a8ae9e 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:34:46 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 408 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 119 60.001 [azure-functions-crack-detection-deployment-service-80] - - - - a1457b3361fbc80b2f756486e8afddbc 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:34:46 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 408 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 126 59.998 [azure-functions-crack-detection-deployment-service-80] - - - - bf0e0e8fa426199d078cb7955164cac5 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:34:46 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 408 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 124 59.998 [azure-functions-crack-detection-deployment-service-80] - - - - 53067daeb576609b58a510a1dd54fb3d 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:34:46 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 408 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 125 59.998 [azure-functions-crack-detection-deployment-service-80] - - - - 6742ffe052847142f1697d164d039337 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:34:46 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 408 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 119 59.997 [azure-functions-crack-detection-deployment-service-80] - - - - c9192310bc5d3b9ea6a189455a238dcf 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:35:17 +0000] "GET /api/image-records HTTP/2.0" 404 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 64 0.312 [azure-functions-crack-detection-deployment-service-80] 10.244.0.128:80 0 0.312 404 4ed2b3b1256a41cd69a68563e49e666d 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:35:23 +0000] "GET /api/image-records HTTP/2.0" 404 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 65 0.209 [azure-functions-crack-detection-deployment-service-80] 10.244.2.120:80 0 0.204 404 fa1cdcf8614394f62d379bda51cd936f 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:35:36 +0000] "OPTIONS /api/upload-image?segment=true HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 300 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - ab01b8daf04434fdd15294d5f0092d59 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:35:36 +0000] "OPTIONS /api/upload-image?segment=true HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 12 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - 0d9a71856e9b0299c554c06c49a7e852 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:35:58 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 499 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 124 21.780 [azure-functions-crack-detection-deployment-service-80] - - - - ae40fdc1a79b4d80bea13e66ea19becf 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:35:58 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 499 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 169 21.780 [azure-functions-crack-detection-deployment-service-80] - - - - b03657a52f17b423f773e8185aad2294 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:38:18 +0000] "GET /api/image-records HTTP/2.0" 404 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 64 0.196 [azure-functions-crack-detection-deployment-service-80] 10.244.1.135:80 0 0.192 404 fdcfb5349deb8a56dc44620315cb1000 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:38:30 +0000] "OPTIONS /api/upload-image?segment=true HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 12 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - 1351682a5e2d3a16e58d44283de142bc 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:38:30 +0000] "OPTIONS /api/upload-image?segment=true HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 12 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - dbd3c10026bf92666501b26e266a65ed 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:38:58 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 499 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 123 27.652 [azure-functions-crack-detection-deployment-service-80] - - - - 3eb67a54710877895fe3069e57b3a7fb 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:38:58 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 499 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 126 27.651 [azure-functions-crack-detection-deployment-service-80] - - - - 4498112157bbf8c0a91878a99fb19ac9 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:39:00 +0000] "GET /api/image-records HTTP/2.0" 499 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 65 0.262 [azure-functions-crack-detection-deployment-service-80] 10.244.0.128:80 0 0.264 - 41dc2e1fa93f4cab97955ceff4bb2471 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:39:02 +0000] "OPTIONS /api/image-records HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 13 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - 880fdbd6d14c22e4548d503572c25266 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:39:02 +0000] "GET /api/image-records HTTP/2.0" 404 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 65 0.176 [azure-functions-crack-detection-deployment-service-80] 10.244.2.120:80 0 0.176 404 5dd0f4bb27b54b01e573c503ed8e04fd 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:39:12 +0000] "OPTIONS /api/upload-image?segment=true HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 12 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - c257850a6dcc17966c2e7b2f5d147247 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:39:12 +0000] "OPTIONS /api/upload-image?segment=true HTTP/2.0" 204 0 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 12 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - 958dd4a2f3071d2ea8a088b5b38fdfa8 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:40:12 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 408 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 119 60.003 [azure-functions-crack-detection-deployment-service-80] - - - - d7cee417ae6fc9781082193db2e9b829 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:40:12 +0000] "POST /api/upload-image?segment=true HTTP/2.0" 408 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 120 60.002 [azure-functions-crack-detection-deployment-service-80] - - - - c39aa79bac343efc454980014b8af9b2 10.240.0.4 - [10.240.0.4] - - [13/Jun/2019:16:40:21 +0000] "GET /api/image-records HTTP/2.0" 404 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0" 64 0.279 [azure-functions-crack-detection-deployment-service-80] 10.244.1.135:80 0 0.276 404 fb0a7f7daee8463a1ec6a0fd867e2266 I0613 17:30:22.351351 6 leaderelection.go:263] failed to renew lease azure-functions/ingress-controller-leader-nginx: failed to tryAcquireOrRenew context deadline exceeded I0613 17:30:22.351426 6 leaderelection.go:217] attempting to acquire leader lease azure-functions/ingress-controller-leader-nginx... I0613 17:30:23.580871 6 leaderelection.go:227] successfully acquired lease azure-functions/ingress-controller-leader-nginx 10.244.2.1 - [10.244.2.1] - - [13/Jun/2019:17:57:18 +0000] "\x01\x00\x00\x00" 400 158 "-" "-" 0 0.245 [] - - - - f60f8fb4836fcf82d9c0641318fec4ec 10.244.2.1 - [10.244.2.1] - - [13/Jun/2019:18:07:01 +0000] "GET http://110.249.212.46/testget?q=23333&port=80 HTTP/1.1" 400 158 "-" "-" 144 0.000 [] - - - - bb61179b9532aad2291e2a1d7d6e2170 10.244.2.1 - [10.244.2.1] - - [13/Jun/2019:18:31:08 +0000] "GET /api/image-records HTTP/2.0" 401 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36" 289 0.657 [azure-functions-crack-detection-deployment-service-80] 10.244.2.120:80 0 0.656 401 dcb603a788ef8be1d76f877682fda6f3 10.244.2.1 - [10.244.2.1] - - [13/Jun/2019:18:31:14 +0000] "GET / HTTP/2.0" 200 94622 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36" 9 0.003 [azure-functions-crack-detection-deployment-service-80] 10.244.1.135:80 149793 0.000 200 8761921bd6c9264236e5135d951ae3be

Controller 2 (Doesn't seem to be reacting)

10.240.0.6 - [10.240.0.6] - - [13/Jun/2019:16:20:40 +0000] "OPTIONS /api/image-records?id=5d024a0088f3b87b206cdd26 HTTP/2.0" 204 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36" 65 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - 04819a52e3aeea4e21943b08ff521c6f 10.240.0.6 - [10.240.0.6] - - [13/Jun/2019:16:20:40 +0000] "DELETE /api/image-records?id=5d024a0088f3b87b206cdd26 HTTP/2.0" 200 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36" 63 0.199 [azure-functions-crack-detection-deployment-service-80] 10.244.0.128:80 0 0.196 200 554bd9f0981a6fe8e31778c9e194f35d 10.240.0.6 - [10.240.0.6] - - [13/Jun/2019:16:20:41 +0000] "OPTIONS /api/image-records?id=5d024a0088f3b87b206cdd28 HTTP/2.0" 204 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36" 65 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - 4b6cf7989daf5799ebe1046d45e3e7d3 10.240.0.6 - [10.240.0.6] - - [13/Jun/2019:16:20:41 +0000] "DELETE /api/image-records?id=5d024a0088f3b87b206cdd28 HTTP/2.0" 200 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36" 63 0.329 [azure-functions-crack-detection-deployment-service-80] 10.244.2.120:80 0 0.328 200 10aae137f33adea63ca7f3b79e287a7c 10.240.0.6 - [10.240.0.6] - - [13/Jun/2019:16:20:42 +0000] "OPTIONS /api/image-records?id=5d024a0088f3b87b206cdd27 HTTP/2.0" 204 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36" 65 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - 6cc9a2c4c52785e25a1cacba56f8e40b 10.240.0.6 - [10.240.0.6] - - [13/Jun/2019:16:20:43 +0000] "DELETE /api/image-records?id=5d024a0088f3b87b206cdd27 HTTP/2.0" 200 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36" 63 0.455 [azure-functions-crack-detection-deployment-service-80] 10.244.1.135:80 0 0.456 200 13d4b337c5105295e98a6c3cda55c25c 10.240.0.6 - [10.240.0.6] - - [13/Jun/2019:16:20:44 +0000] "OPTIONS /api/image-records?id=5d024a0088f3b87b206cdd29 HTTP/2.0" 204 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36" 65 0.000 [azure-functions-crack-detection-deployment-service-80] - - - - 537370c741ffab3621090ab8dcf014fd 10.240.0.6 - [10.240.0.6] - - [13/Jun/2019:16:20:44 +0000] "DELETE /api/image-records?id=5d024a0088f3b87b206cdd29 HTTP/2.0" 200 0 "https://objectdetectstoragedev.z19.web.core.windows.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.80 Safari/537.36" 63 0.224 [azure-functions-crack-detection-deployment-service-80] 10.244.0.128:80 0 0.224 200 873e6efacc53c17c1c8b8d3b84682664

SpicySyntax commented 5 years ago

@aledbf were the logs not helpful?

aledbf commented 5 years ago

@SpicySyntax not really. I don't see any reload or event related to a change in the secret. Also, your logs seem to be "ok" for the host https://objectdetectstoragedev.z19.web.core.windows.net/

SpicySyntax commented 5 years ago

@aledbf When you navigate to the root of the backend (https://aks-ingress.eastus.cloudapp.azure.com/) Before it loads the default page you can see it uses that fake certificate: image Proceed without it and the default page loads: image

So I know it using the default fake cert, but I am not sure why it stops using the cert created by lets-encrypt.

I am not quite sure how to proceed other than nuking all of my ingress controller and certificate resources and starting from scratch. Any suggestions before I try that?

Thanks

aledbf commented 5 years ago

@SpicySyntax the host aks-ingress.eastus.cloudapp.azure.com != objectdetectstoragedev.z19.web.core.windows.net (in the logs) How are you configuring this hostname?

SpicySyntax commented 5 years ago

Following these instructions: Specifically the sections 'Configure a DNS name' and 'Create an ingress route'. Should my ingress.yaml also have a host for the website that uses it with TLS. (spec.tls.host)?

aledbf commented 5 years ago

Should my ingress.yaml also have a host for the website that uses it with TLS. (spec.tls.host)?

Yes. From the link check the definition of the ingress for the host demo-aks-ingress.eastus.cloudapp.azure.com

lfshr commented 5 years ago

Could this be related to #3834?

SpicySyntax commented 5 years ago

@aledbf something like this? (the addition is the host objectdetectstoragedev.z19.web.core.windows.net):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: obj-det-ingress
  namespace: azure-functions
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, DELETE, GET, POST, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://objectdetectstoragedev.z19.web.core.windows.net/"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
spec:
  tls:
  - hosts:
    - aks-ingress.eastus.cloudapp.azure.com
    - objectdetectstoragedev.z19.web.core.windows.net
    secretName: tls-secret
  rules:
  - host: aks-ingress.eastus.cloudapp.azure.com
    http:
      paths:
      - backend:
          serviceName: crack-detection-deployment-service
          servicePort: 80
        path: /
SpicySyntax commented 5 years ago

@aledbf I started completely from scratch and am experiencing the same issue. This time the logs seem more helpful. Ingress Controller 1:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.24.1
  Build:      git-ce418168f
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
 I0626 14:50:53.278914       6 flags.go:185] Watching for Ingress class: nginx
W0626 14:50:53.279179       6 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.10
W0626 14:50:53.288813       6 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0626 14:50:53.289185       6 main.go:205] Creating API client for https://10.0.0.1:443
I0626 14:50:53.323431       6 main.go:249] Running in Kubernetes cluster version v1.12 (v1.12.8) - git (clean) commit a89f8c11a5f4f132503edbc4918c98518fd504e3 - platform linux/amd64
I0626 14:50:53.330406       6 main.go:102] Validated default/wandering-moth-nginx-ingress-default-backend as the default backend.
I0626 14:50:53.602508       6 main.go:124] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
W0626 14:50:53.620715       6 store.go:613] Unexpected error reading configuration configmap: configmaps "wandering-moth-nginx-ingress-controller" not found
I0626 14:50:53.627543       6 nginx.go:265] Starting NGINX Ingress controller
I0626 14:50:54.828225       6 nginx.go:311] Starting NGINX process
I0626 14:50:54.828427       6 leaderelection.go:217] attempting to acquire leader lease  default/ingress-controller-leader-nginx...
I0626 14:50:54.829118       6 controller.go:170] Configuration changes detected, backend reload required.
I0626 14:50:54.867664       6 status.go:86] new leader elected: wandering-moth-nginx-ingress-controller-778b6966d4-t6qnm
I0626 14:50:54.960034       6 controller.go:188] Backend successfully reloaded.
I0626 14:50:54.960208       6 controller.go:202] Initial sync, sleeping for 1 second.
[26/Jun/2019:14:50:55 +0000]TCP200000.000
W0626 15:01:35.347540       6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
W0626 15:01:35.348146       6 controller.go:1042] Error getting SSL certificate "default/tls-secret": local SSL certificate default/tls-secret was not found. Using default certificate
I0626 15:01:35.348209       6 controller.go:170] Configuration changes detected, backend reload required.
I0626 15:01:35.348039       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"obj-det-ingress", UID:"4a854b2c-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842379", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/obj-det-ingress
I0626 15:01:35.452637       6 controller.go:188] Backend successfully reloaded.
[26/Jun/2019:15:01:35 +0000]TCP200000.000
I0626 15:01:35.801410       6 store.go:419] secret default/tls-secret was added and it is used in ingress annotations. Parsing...
I0626 15:01:35.802314       6 backend_ssl.go:68] Adding Secret "default/tls-secret" to the local store
I0626 15:01:37.919034       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-2vfrt", UID:"4c090dc6-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842408", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/cm-acme-http-solver-2vfrt
10.244.2.1 - [10.244.2.1] - - [26/Jun/2019:15:01:38 +0000] "GET /.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q HTTP/1.1" 404 0 "http://od-aks-ingress.eastus.cloudapp.azure.com/.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q" "Go-http-client/1.1" 339 0.003 [default-fashionable-kitten-crack-detection-80] 10.244.0.119:80 0 0.004 404 a9b9722fbeb9b7ab7589a6af138768fe
I0626 15:01:38.132560       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-6mlbm", UID:"4c1104c2-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842411", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/cm-acme-http-solver-6mlbm
W0626 15:01:38.681673       6 controller.go:797] Service "default/cm-acme-http-solver-8llrh" does not have any active Endpoint.
I0626 15:01:38.681827       6 controller.go:170] Configuration changes detected, backend reload required.
I0626 15:01:38.797067       6 controller.go:188] Backend successfully reloaded.
[26/Jun/2019:15:01:38 +0000]TCP200000.001
[26/Jun/2019:15:01:42 +0000]TCP200000.000
I0626 15:01:48.092454       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-6mlbm", UID:"4c1104c2-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842443", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/cm-acme-http-solver-6mlbm
I0626 15:01:48.150624       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-2vfrt", UID:"4c090dc6-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842447", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/cm-acme-http-solver-2vfrt
I0626 15:01:48.218396       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-4vc5z", UID:"52361f5e-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842451", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/cm-acme-http-solver-4vc5z
I0626 15:01:53.872083       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"obj-det-ingress", UID:"4a854b2c-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842475", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/obj-det-ingress
I0626 15:01:53.872393       6 backend_ssl.go:60] Updating Secret "default/tls-secret" in the local store
10.244.2.1 - [10.244.2.1] - - [26/Jun/2019:15:01:54 +0000] "GET /.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q HTTP/1.1" 200 87 "-" "Go-http-client/1.1" 210 0.002 [default-cm-acme-http-solver-8llrh-8089] 10.244.1.157:8089 87 0.000 200 45d5bde593834ce3426a9abbe8f56e87
10.240.0.4 - [10.240.0.4] - - [26/Jun/2019:15:01:58 +0000] "GET /.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 292 0.001 [default-cm-acme-http-solver-8llrh-8089] 10.244.1.157:8089 87 0.000 200 6ad0024f3c3220d4c8ea63d510c56325
10.240.0.5 - [10.240.0.5] - - [26/Jun/2019:15:01:58 +0000] "GET /.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 292 0.001 [default-cm-acme-http-solver-8llrh-8089] 10.244.1.157:8089 87 0.000 200 36cc0811f28e031f43376d53e9bd0f44
I0626 15:02:00.141308       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-4vc5z", UID:"52361f5e-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842502", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/cm-acme-http-solver-4vc5z
I0626 15:02:00.826835       6 store.go:446] secret default/tls-secret was updated and it is used in ingress annotations. Parsing...
I0626 15:02:00.828466       6 backend_ssl.go:60] Updating Secret "default/tls-secret" in the local store
I0626 15:02:02.015082       6 controller.go:170] Configuration changes detected, backend reload required.
I0626 15:02:02.120268       6 controller.go:188] Backend successfully reloaded.
[26/Jun/2019:15:02:02 +0000]TCP200000.000

Controller 2:

I0626 14:50:52.261839       6 flags.go:185] Watching for Ingress class: nginx
W0626 14:50:52.262060       6 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.10
W0626 14:50:52.265453       6 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0626 14:50:52.265665       6 main.go:205] Creating API client for https://10.0.0.1:443
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.24.1
  Build:      git-ce418168f
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
 I0626 14:50:52.304326       6 main.go:249] Running in Kubernetes cluster version v1.12 (v1.12.8) - git (clean) commit a89f8c11a5f4f132503edbc4918c98518fd504e3 - platform linux/amd64
I0626 14:50:52.309170       6 main.go:102] Validated default/wandering-moth-nginx-ingress-default-backend as the default backend.
I0626 14:50:52.443585       6 main.go:124] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
W0626 14:50:52.470380       6 store.go:613] Unexpected error reading configuration configmap: configmaps "wandering-moth-nginx-ingress-controller" not found
I0626 14:50:52.480370       6 nginx.go:265] Starting NGINX Ingress controller
I0626 14:50:53.680794       6 nginx.go:311] Starting NGINX process
I0626 14:50:53.681036       6 leaderelection.go:217] attempting to acquire leader lease  default/ingress-controller-leader-nginx...
I0626 14:50:53.681723       6 controller.go:170] Configuration changes detected, backend reload required.
I0626 14:50:53.723800       6 leaderelection.go:227] successfully acquired lease default/ingress-controller-leader-nginx
I0626 14:50:53.724190       6 status.go:86] new leader elected: wandering-moth-nginx-ingress-controller-778b6966d4-t6qnm
I0626 14:50:53.828193       6 controller.go:188] Backend successfully reloaded.
I0626 14:50:53.828380       6 controller.go:202] Initial sync, sleeping for 1 second.
[26/Jun/2019:14:50:54 +0000]TCP200000.000
[26/Jun/2019:14:50:57 +0000]TCP200000.000
W0626 15:01:35.347617       6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
W0626 15:01:35.347727       6 controller.go:1042] Error getting SSL certificate "default/tls-secret": local SSL certificate default/tls-secret was not found. Using default certificate
I0626 15:01:35.347751       6 controller.go:170] Configuration changes detected, backend reload required.
I0626 15:01:35.347769       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"obj-det-ingress", UID:"4a854b2c-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842379", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/obj-det-ingress
I0626 15:01:35.453524       6 controller.go:188] Backend successfully reloaded.
[26/Jun/2019:15:01:35 +0000]TCP200000.000
I0626 15:01:35.801463       6 store.go:419] secret default/tls-secret was added and it is used in ingress annotations. Parsing...
I0626 15:01:35.802121       6 backend_ssl.go:68] Adding Secret "default/tls-secret" to the local store
I0626 15:01:37.919071       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-2vfrt", UID:"4c090dc6-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842408", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/cm-acme-http-solver-2vfrt
10.240.0.6 - [10.240.0.6] - - [26/Jun/2019:15:01:37 +0000] "GET /.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q HTTP/1.1" 308 172 "-" "Go-http-client/1.1" 210 0.000 [default-fashionable-kitten-crack-detection-80] - - - - 5357d6a3f4235ecc5647b05f126bbf43
10.240.0.6 - [10.240.0.6] - - [26/Jun/2019:15:01:38 +0000] "GET /.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q HTTP/1.1" 404 0 "http://od-aks-ingress.eastus.cloudapp.azure.com/.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q" "Go-http-client/1.1" 339 0.004 [default-fashionable-kitten-crack-detection-80] 10.244.0.119:80 0 0.004 404 d9b199a536429555bd9ab378bc1d2590
10.240.0.6 - [10.240.0.6] - - [26/Jun/2019:15:01:38 +0000] "GET /.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q HTTP/1.1" 308 172 "-" "Go-http-client/1.1" 210 0.000 [default-fashionable-kitten-crack-detection-80] - - - - 3b0deec82b9febdd04bc0bd132205d68
I0626 15:01:38.132263       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-6mlbm", UID:"4c1104c2-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842411", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/cm-acme-http-solver-6mlbm
W0626 15:01:38.681370       6 controller.go:797] Service "default/cm-acme-http-solver-8llrh" does not have any active Endpoint.
I0626 15:01:38.681451       6 controller.go:170] Configuration changes detected, backend reload required.
I0626 15:01:38.810202       6 controller.go:188] Backend successfully reloaded.
[26/Jun/2019:15:01:38 +0000]TCP200000.000
[26/Jun/2019:15:01:42 +0000]TCP200000.000
I0626 15:01:48.092757       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-6mlbm", UID:"4c1104c2-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842443", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/cm-acme-http-solver-6mlbm
I0626 15:01:48.151100       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-2vfrt", UID:"4c090dc6-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842447", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/cm-acme-http-solver-2vfrt
I0626 15:01:48.219174       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-4vc5z", UID:"52361f5e-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842451", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/cm-acme-http-solver-4vc5z
I0626 15:01:53.751577       6 status.go:295] updating Ingress default/cm-acme-http-solver-4vc5z status from [] to [{ }]
I0626 15:01:53.810489       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-4vc5z", UID:"52361f5e-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842474", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/cm-acme-http-solver-4vc5z
I0626 15:01:53.810921       6 status.go:295] updating Ingress default/obj-det-ingress status from [] to [{ }]
I0626 15:01:53.871949       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"obj-det-ingress", UID:"4a854b2c-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842475", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/obj-det-ingress
I0626 15:01:53.872157       6 backend_ssl.go:60] Updating Secret "default/tls-secret" in the local store
10.240.0.6 - [10.240.0.6] - - [26/Jun/2019:15:01:56 +0000] "GET /.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q HTTP/1.1" 200 87 "-" "Go-http-client/1.1" 210 0.001 [default-cm-acme-http-solver-8llrh-8089] 10.244.1.157:8089 87 0.004 200 6656b7fd7f9802a2ce0e51c6d80831e0
10.240.0.6 - [10.240.0.6] - - [26/Jun/2019:15:01:58 +0000] "GET /.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 292 0.000 [default-cm-acme-http-solver-8llrh-8089] 10.244.1.157:8089 87 0.000 200 55067d99e8ef490235c0e67954df6b1e
10.240.0.5 - [10.240.0.5] - - [26/Jun/2019:15:01:58 +0000] "GET /.well-known/acme-challenge/HJm1Cvqvz-dUEyzzx0g8xoGksFbn5_iDLKRPuVVwl3Q HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 292 0.000 [default-cm-acme-http-solver-8llrh-8089] 10.244.1.157:8089 87 0.000 200 a1d1b68832d0389cacfaa121c8d6e2e9
I0626 15:02:00.140246       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-4vc5z", UID:"52361f5e-9823-11e9-af4a-5e8a19266c86", APIVersion:"extensions/v1beta1", ResourceVersion:"842502", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/cm-acme-http-solver-4vc5z
I0626 15:02:00.824438       6 store.go:446] secret default/tls-secret was updated and it is used in ingress annotations. Parsing...
I0626 15:02:00.825659       6 backend_ssl.go:60] Updating Secret "default/tls-secret" in the local store
I0626 15:02:02.014573       6 controller.go:170] Configuration changes detected, backend reload required.
I0626 15:02:02.126171       6 controller.go:188] Backend successfully reloaded.
[26/Jun/2019:15:02:02 +0000]TCP200000.000
10.240.0.5 - [10.240.0.5] - - [26/Jun/2019:15:02:55 +0000] "GET / HTTP/2.0" 200 94622 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" 272 0.242 [default-fashionable-kitten-crack-detection-80] 10.244.0.119:80 149793 0.032 200 5e9a0d72b20f90c50869fcd06e1ff4ca
10.240.0.5 - [10.240.0.5] - - [26/Jun/2019:15:02:55 +0000] "GET /favicon.ico HTTP/2.0" 404 0 "https://od-aks-ingress.eastus.cloudapp.azure.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" 113 0.026 [default-fashionable-kitten-crack-detection-80] 10.244.1.153:80 0 0.024 404 95810354efc10dc697b70b7f6abd3a05

Cert Manager:

I0626 15:01:58.305554       1 sync.go:307] cert-manager/controller/challenges/acceptChallenge "level"=0 "msg"="accepting challenge with ACME server" "dnsName"="od-aks-ingress.eastus.cloudapp.azure.com" "resource_kind"="Challenge" "resource_name"="tls-secret-2240443849-0" "resource_namespace"="default" "type"="http-01" 
I0626 15:01:58.305581       1 logger.go:63] Calling AcceptChallenge
I0626 15:01:58.371839       1 sync.go:324] cert-manager/controller/challenges/acceptChallenge "level"=0 "msg"="waiting for authorization for domain" "dnsName"="od-aks-ingress.eastus.cloudapp.azure.com" "resource_kind"="Challenge" "resource_name"="tls-secret-2240443849-0" "resource_namespace"="default" "type"="http-01" 
I0626 15:01:58.371852       1 logger.go:78] Calling WaitAuthorization
I0626 15:01:59.775685       1 controller.go:219] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="default/tls-secret-2240443849-0" 
I0626 15:01:59.775840       1 controller.go:213] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="default/tls-secret-2240443849-0" 
I0626 15:01:59.775788       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="default/tls-secret-2240443849" 
I0626 15:01:59.776325       1 sync.go:274] Need to create 0 challenges
I0626 15:01:59.776340       1 logger.go:43] Calling GetOrder
I0626 15:01:59.914355       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="default/tls-secret-2240443849" 
I0626 15:01:59.914395       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="default/tls-secret-2240443849" 
I0626 15:01:59.914472       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="default/tls-secret" 
I0626 15:01:59.914554       1 logger.go:58] Calling FinalizeOrder
I0626 15:01:59.915043       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="tls-secret-2240443849" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="tls-secret" "resource_namespace"="default" 
I0626 15:01:59.915278       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/tls-secret" 
I0626 15:02:00.140200       1 controller.go:178] ingress-shim controller: syncing item 'default/cm-acme-http-solver-4vc5z'
E0626 15:02:00.140394       1 controller.go:202] ingress 'default/cm-acme-http-solver-4vc5z' in work queue no longer exists
I0626 15:02:00.140468       1 controller.go:184] ingress-shim controller: Finished processing work item "default/cm-acme-http-solver-4vc5z"
I0626 15:02:00.171680       1 controller.go:219] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="default/tls-secret-2240443849-0" 
I0626 15:02:00.171867       1 controller.go:213] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="default/tls-secret-2240443849-0" 
I0626 15:02:00.171944       1 controller.go:219] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="default/tls-secret-2240443849-0" 
I0626 15:02:00.643316       1 logger.go:43] Calling GetOrder
I0626 15:02:00.767091       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="default/tls-secret" 
I0626 15:02:00.769225       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="default/tls-secret-2240443849" 
I0626 15:02:00.769249       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="default/tls-secret-2240443849" 
I0626 15:02:00.816618       1 sync.go:288] cert-manager/controller/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="1438h59m58.183421359s" "related_resource_kind"="Secret" "related_resource_name"="tls-secret" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="tls-secret" "resource_namespace"="default" 
I0626 15:02:00.816870       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/tls-secret" 
I0626 15:02:00.824668       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="default/tls-secret" 
I0626 15:02:00.825577       1 conditions.go:143] Found status change for Certificate "tls-secret" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2019-06-26 15:02:00.825572977 +0000 UTC m=+145.821133558
I0626 15:02:00.825815       1 sync.go:288] cert-manager/controller/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="1438h59m58.174205822s" "related_resource_kind"="Secret" "related_resource_name"="tls-secret" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="tls-secret" "resource_namespace"="default" 
I0626 15:02:00.829585       1 controller.go:213] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="default/tls-secret-2240443849-0" 
I0626 15:02:00.830389       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="default/tls-secret-2240443849" 
I0626 15:02:00.831053       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="default/tls-secret-2240443849" 
I0626 15:02:00.871048       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/tls-secret" 
I0626 15:02:00.871263       1 controller.go:178] ingress-shim controller: syncing item 'default/obj-det-ingress'
I0626 15:02:00.871292       1 sync.go:181] Certificate "tls-secret" for ingress "obj-det-ingress" already exists
I0626 15:02:00.871339       1 sync.go:184] Certificate "tls-secret" for ingress "obj-det-ingress" is up to date
I0626 15:02:00.871428       1 controller.go:184] ingress-shim controller: Finished processing work item "default/obj-det-ingress"
I0626 15:02:00.871526       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="default/tls-secret" 
I0626 15:02:00.872126       1 sync.go:288] cert-manager/controller/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="1438h59m58.127902539s" "related_resource_kind"="Secret" "related_resource_name"="tls-secret" "related_resource_namespace"="default" "resource_kind"="Certificate" "resource_name"="tls-secret" "resource_namespace"="default" 
I0626 15:02:00.872448       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="default/tls-secret" 
I0626 15:02:00.879321       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="default/tls-secret-2240443849" 
I0626 15:02:00.917650       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="default/tls-secret-2240443849" 
I0626 15:02:00.918276       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="default/tls-secret-2240443849" 
I0626 15:02:00.918843       1 controller.go:219] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="default/tls-secret-2240443849-0" 
I0626 15:02:00.918880       1 controller.go:213] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="default/tls-secret-2240443849-0" 
E0626 15:02:00.918984       1 controller.go:238] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"tls-secret-2240443849-0\" not found"

Do these logs help? Should I just try with my own self signed certs?

Thanks

SpicySyntax commented 5 years ago

@aledbf Can you see above? this a blocking issue for use currently. Thank you

sajuptpm commented 5 years ago

facing same issue in cert-manager v0.6.7, nginx controller taking only the "default certificate". I can see the message "Adding Secret "default/tls-secret-frontend" to the local store" in nginx controller log, but it not taking added secret.

log from nginx controller

10.20.1.1 - [10.20.1.1] - - [10/Jul/2019:13:59:22 +0000] "GET /.well-known/acme-challenge/92Bdd--XSmhZuMwB_etLmhi_Oh0i0uXxe_DXr8PrS7M HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 273 0.001 [default-cm-acme-http-solver-kqrr6-8089] 10.20.3.20:8089 87 0.001 200 364a549ebd9687ba096c2e101f4acd91 W0710 13:59:23.672840 6 controller.go:724] Error obtaining Endpoints for Service "default/cm-acme-http-solver-wr7pq": no object matching key "default/cm-acme-http-solver-wr7pq" in local store W0710 13:59:23.673283 6 controller.go:1042] Error getting SSL certificate "default/tls-secret-frontend": local SSL certificate default/tls-secret-frontend was not found. Using default certificate <========= W0710 13:59:23.675491 6 controller.go:1042] Error getting SSL certificate "default/tls-secret-backend": local SSL certificate default/tls-secret-backend was not found. Using default certificate<========= I0710 13:59:23.675589 6 controller.go:170] Configuration changes detected, backend reload required. I0710 13:59:23.735436 6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-mjr9v", UID:"dddb4725-a31a-11e9-b8c6-42010aa0002e", APIVersion:"extensions/v1beta1", ResourceVersion:"1838377", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/cm-acme-http-solver-mjr9v I0710 13:59:24.059267 6 controller.go:188] Backend successfully reloaded. [10/Jul/2019:13:59:24 +0000]TCP200000.000 I0710 13:59:24.298806 6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"cm-acme-http-solver-7pbbz", UID:"de02abbc-a31a-11e9-b8c6-42010aa0002e", APIVersion:"extensions/v1beta1", ResourceVersion:"1838390", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/cm-acme-http-solver-7pbbz I0710 13:59:25.249769 6 status.go:295] updating Ingress default/tkingress-resource status from [] to [{35.234.210.167 }] W0710 13:59:25.259778 6 backend_ssl.go:48] Error obtaining X.509 certificate: unexpected error creating SSL Cert: no certificate PEM data found, make sure certificate content starts with 'BEGIN CERTIFICATE' W0710 13:59:25.259827 6 backend_ssl.go:48] Error obtaining X.509 certificate: unexpected error creating SSL Cert: no certificate PEM data found, make sure certificate content starts with 'BEGIN CERTIFICATE' I0710 13:59:25.260208 6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"tkingress-resource", UID:"dc36a945-a31a-11e9-b8c6-42010aa0002e", APIVersion:"extensions/v1beta1", ResourceVersion:"1838400", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/tkingress-resource I0710 13:59:26.601632 6 store.go:446] secret default/tls-secret-backend was updated and it is used in ingress annotations. Parsing... W0710 13:59:26.601979 6 backend_ssl.go:48] Error obtaining X.509 certificate: unexpected error creating SSL Cert: no certificate PEM data found, make sure certificate content starts with 'BEGIN CERTIFICATE' I0710 13:59:26.602448 6 backend_ssl.go:68] Adding Secret "default/tls-secret-backend" to the local store <======= I0710 13:59:26.847881 6 store.go:446] secret default/tls-secret-frontend was updated and it is used in ingress annotations. Parsing... I0710 13:59:26.848652 6 backend_ssl.go:68] Adding Secret "default/tls-secret-frontend" to the local store <======= I0710 13:59:27.006438 6 controller.go:170] Configuration changes detected, backend reload required. I0710 13:59:27.280785 6 controller.go:188] Backend successfully reloaded. [10/Jul/2019:13:59:27 +0000]TCP200000.000

aledbf commented 4 years ago

@SpicySyntax please update to 0.26.0 to see if the issue persists

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 4 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/4199#issuecomment-590638502): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.