search

jetstack / kube-lego

DEPRECATED: Automatically request certificates for Kubernetes Ingress resources from Let's Encrypt
Apache License 2.0
2.16k stars 267 forks source link

Can't request a certificate, error: waiting for authorization failed: acme: identifier authorization failed" c #171

Open jescarri opened 7 years ago

jescarri commented 7 years ago

My setup is as follows:


I can reach ingress enpoints, so Ingress is working properly.

This is my Ingress definition:

kind: Ingress apiVersion: extensions/v1beta1 metadata: name: cafe-ingress namespace: test selfLink: /apis/extensions/v1beta1/namespaces/test/ingresses/cafe-ingress uid: 90d49530-298a-11e7-8f6f-ceb05f3ffd15 resourceVersion: '3039728' generation: 18 creationTimestamp: '2017-04-25T07:41:17Z' annotations: kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: 'true' spec: tls:

kube-lego manifest:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  annotations:
  name: kube-lego-kube-lego
  namespace: kube-system
  labels:
    app: kube-lego
    chart: kube-lego-0.1.8
    heritage: Tiller
    release: kube-lego
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-lego
      release: kube-lego
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kube-lego
        release: kube-lego
    spec:
      containers:
        - name: kube-lego
          image: 'jetstack/kube-lego:canary'
          ports:
            - containerPort: 8080
              protocol: TCP
          env:
            - name: LEGO_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: LEGO_POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: LEGO_EMAIL
              value: jesuscarrillo8@gmail.com
            - name: LEGO_PORT
              value: '8080'
            - name: LEGO_URL
              value: 'https://acme-v01.api.letsencrypt.org/directory'
            - name: LEGO_LOG_LEVEL
              value: debug
            - name: LEGO_SUPPORTED_INGRESS_CLASS
              value: nginx
            - name: LEGO_DEFAULT_INGRESS_CLASS
              value: nginx
          resources: {}
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      nodeSelector:
        internal: 'true'
      securityContext: {}
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1

I can reach curl http://foo.aws.identitylabs.mx/.well-known/acme-challenge/_selftest

*   Trying 104.156.96.34...
* Connected to foo.aws.identitylabs.mx (104.156.96.34) port 80 (#0)
> GET /.well-known/acme-challenge/_selftest HTTP/1.1
> Host: foo.aws.identitylabs.mx
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.11.3
< Date: Fri, 28 Apr 2017 07:08:35 GMT
< Content-Type: text/plain
< Content-Length: 16
< Connection: keep-alive
< 
* Connection #0 to host foo.aws.identitylabs.mx left intact

kube-lego errors:

time="2017-04-28T06:36:47Z" level=debug msg=reset context=provider provider=gce
time="2017-04-28T06:36:47Z" level=debug msg=finalize context=provider provider=gce
time="2017-04-28T06:36:47Z" level=debug msg=reset context=provider provider=nginx
time="2017-04-28T06:36:47Z" level=debug msg=finalize context=provider provider=nginx
time="2017-04-28T06:36:47Z" level=info msg="process certificate requests for ingresses" context=kubelego
time="2017-04-28T06:36:47Z" level=info msg="Attempting to create new secret" context=secret name=test-tls namespace=test
time="2017-04-28T06:36:47Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=cafe-ingress namespace=test
time="2017-04-28T06:36:47Z" level=info msg="requesting certificate for foo.aws.identitylabs.mx" context="ingress_tls" name=cafe-ingress namespace=test
time="2017-04-28T06:36:47Z" level=debug msg="testing reachability of http://foo.aws.identitylabs.mx/.well-known/acme-challenge/_selftest" context=acme domain=foo.aws.identitylabs.mx
time="2017-04-28T06:36:53Z" level=debug msg="error while authorizing: waiting for authorization failed: acme: identifier authorization failed" context=acme domain=foo.aws.identitylabs.mx
time="2017-04-28T06:36:54Z" level=debug msg="testing reachability of http://foo.aws.identitylabs.mx/.well-known/acme-challenge/_selftest" context=acme domain=foo.aws.identitylabs.mx
time="2017-04-28T06:36:59Z" level=debug msg="error while authorizing: waiting for authorization failed: acme: identifier authorization failed" context=acme domain=foo.aws.identitylabs.mx
time="2017-04-28T06:37:00Z" level=debug msg="testing reachability of http://foo.aws.identitylabs.mx/.well-known/acme-challenge/_selftest" context=acme domain=foo.aws.identitylabs.mx
time="2017-04-28T06:37:05Z" level=debug msg="error while authorizing: waiting for authorization failed: acme: identifier authorization failed" context=acme domain=foo.aws.identitylabs.mx
time="2017-04-28T06:37:07Z" level=debug msg="testing reachability of http://foo.aws.identitylabs.mx/.well-known/acme-challenge/_selftest" context=acme domain=foo.aws.identitylabs.mx
time="2017-04-28T06:37:13Z" level=debug msg="error while authorizing: waiting for authorization failed: acme: identifier authorization failed" context=acme domain=foo.aws.identitylabs.mx
time="2017-04-28T06:37:15Z" level=debug msg="testing reachability of http://foo.aws.identitylabs.mx/.well-known/acme-challenge/_selftest" context=acme domain=foo.aws.identitylabs.mx
time="2017-04-28T06:37:21Z" level=debug msg="error while authorizing: waiting for authorization failed: acme: identifier authorization failed" context=acme domain=foo.aws.identitylabs.mx
time="2017-04-28T06:37:24Z" level=debug msg="testing reachability of http://foo.aws.identitylabs.mx/.well-known/acme-challenge/_selftest" context=acme domain=foo.aws.identitylabs.mx

Any idea whats wrong?

Is is worth mentioning that I can manually request certificates for the same domain for hosts running on the same network using certbot

Thanks!

gianrubio commented 7 years ago

What happens if you run the curl command outside of your cluster?

I guess letsencrypt cannot reach your kube-lego, make sure your http/https ports are open for external access.

jescarri commented 7 years ago

I ran curl outside the cluster/another network. if I tail the ingress access logs I can see the kube-lego endpoint is being hit by a go-lang client, I.guess it is lets encrypt?

gianrubio commented 7 years ago

Please increase your kube-lego log level to verbose and share your ingress and lego logs.

munnerz commented 7 years ago

My only thought here is that kube-lego has cached an old DNS entry from before DNS was set up for your domain.

Did you have any more success with this @jescarri? Please also try running curl against the selftest endpoint from within the kube-lego container - that may give us a bit more insight!

liclac commented 7 years ago

I'm getting this too, and interestingly it only seems to happen on 0.1.5; 0.1.3 and 0.1.4 are both fine. Did something change between those?

arnisoph commented 7 years ago

I see this error too now in 1.4 and 1.5 and "acme: identifier authorization failed" as err msg is a bit hard to understand, if you don't know the code. At least for me. :)

// ErrAuthorizationFailed indicates that an authorization for an identifier
// did not succeed.
ErrAuthorizationFailed = errors.New("acme: identifier authorization failed")
arnisoph commented 7 years ago

well my problem was caused by DNS misconfiguration...

DNS problem: SERVFAIL looking up CAA for according to certbot.

It would've been great if the ACME client or kube-lego would've told me this in a proper error message beforehand..

tolleiv commented 7 years ago

Also had this error... my steps to analyse the issue:

 # kubectl run --tty -i --restart=Never --namespace=kube-lego --image=alpine:edge sh
 $ apk --update add curl
 $ curl http://MY_HOST.FQDN/.well-known/acme-challenge/_selftest
 xxxxxxxxx
 $ apk --update add bind-tools
 $ dig MY_HOST.FQDN caaa
 output ... this should confirm that the DNS entry resolves to the public IP
 $ exit
 # kubectl delete pod sh --namespace=kube-lego

Both the curl and the dig should succeed, otherwise the kube-lego has no chance to issue a certificate.

mshappe commented 6 years ago

I am seeing this as well. WhenI chase the actual link (eg. http://mydomain/.well-known/acme-challenge/_selftest), it's actually routing to my app, which of course has no idea what's going on. The interception that's set up by kube-lego is being ignored, basically.