Bug Report

Describe the bug

When requesting ACME certificates, cert-manager created Order and Challenges to complete the request. However, the certificate was not issued because there was a self check failure.

❯ kubectl describe challenge -n backend letsencrypt-nginx-7zb5w-1326296253-1793932787
...
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://auth.mydomain.com/.well-known/acme-challenge/EnaFii1D33BgwiMfDL4gn0j1T3KgVK_5NYzeAS3XTl8': Get "http://auth.mydomain.com/.well-known/acme-challenge/EnaFii1D33BgwiMfDL4gn0j1T3KgVK_5NYzeAS3XTl8": EOF
  State:       pending
Events:
  Type    Reason     Age   From                     Message
  ----    ------     ----  ----                     -------
  Normal  Started    17m   cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  17m   cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism

Affected Components

Challenge

Expected Behavior

❯ kubectl get certificates -n backend
NAME                READY   SECRET              AGE
letsencrypt-nginx   True    letsencrypt-nginx   57s

Actual Behavior

From troubleshooting the certificate request all the way to the challenge, the status shows that the challenge has been presented using the HTTP-01 solver successfully and now cert-manager is waiting for the 'self check' to pass. According to the troubleshooting docs, both HTTP01 and DNS01 go through a "self-check" first before cert-manager presents the challenge to the ACME provider. This is done not to overload the ACME provider with failed challenges due to DNS or loadbalancer propagations.

Steps to Reproduce

Follow guide to add Ingress to cluster as found here

Additional context

The issue is with an incompatibility with DO’s loadBalancer and the way k8s works.

The workaround is:

create a dedicated DNS entry pointing to the public IP of your load balancer (checking for EXTERNAL-IP via `kubectl get svc -n ingress-nginx), eg. kube.mydomain.com
In the manifest used to create your load balancer, add an annotation pointing to that newly created DNS entry, like described in the DO doc.

annotations: 
    service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"

Final nginx values file should look similar to the below:

...
  service:
    type: LoadBalancer
    annotations:
      # Enable proxy protocol
      service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
      # Specify whether the DigitalOcean Load Balancer should pass encrypted data to backend droplets
      service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"
      # Specify an annotation pointing to that newly created DNS entry
      service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com" # <--- here

More info here

digitalocean / Kubernetes-Starter-Kit-Developers

Waiting for HTTP-01 challenge propagation: failed to perform self check GET request #205