search

jetstack / kube-lego

DEPRECATED: Automatically request certificates for Kubernetes Ingress resources from Let's Encrypt
Apache License 2.0
2.16k stars 267 forks source link

404 for HTTP version #135

Open metalshark opened 7 years ago

metalshark commented 7 years ago

Am using the Nginx Ingress Controller and Kubernetes on Azure.

Once a certificate has been created for a domain, accessing the HTTP version results in a 404.

I have to manually edit the ingress kube-lego-nginx and remove the lines looking like:

  - host: domainname.com
    http:
      paths:
      - backend:
          serviceName: kube-lego-nginx
          servicePort: 8080
        path: /.well-known/acme-challenge

for the Nginx ingress controller HTTPS redirect to work.

This happens on all of our sites.

gianrubio commented 7 years ago

@metalshark please share the logs of your application, ingress/kube-lego version, also the deployment files you had used.

metalshark commented 7 years ago

@gianrubio

Yesterday ingress names where the first letter was alphabetically after kube-lego-nginx (in my case ingress names beginning with p, s and t) generated certificates but then returned a 404 for HTTP requests once it switched over to HTTPS (as reported above).

Ingress names where the first letter was below k didn't generate certificates at all and would result in the underlying server generating its own 404 for /.well-known/acme-challenge when looking at the logs of kube-lego (I didn't report this and assumed it was my fault).

I restarted everything this morning and suddenly all apart from one ingress rule alphabetically below k generated certificates AND no longer resulted in a 404 for HTTP requests when it switched over to HTTPS. Ingress rules after k still resulted in a 404 until I turned off acme and had to manually remove them from kube-lego's ingress rule.

Example Site Ingress Rule:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ADOMAINIDONTWISHTODISCLOSE-ing
  annotations:
    kubernetes.io/tls-acme: "true"
spec:
  tls:
  - secretName: ADOMAINIDONTWISHTODISCLOSE-tls
    hosts:
    - ADOMAINIDONTWISHTODISCLOSE
    - www.ADOMAINIDONTWISHTODISCLOSE
  rules:
  - host: ADOMAINIDONTWISHTODISCLOSE
    http:
      paths:
      - path: /
        backend:
          serviceName: ADOMAINIDONTWISHTODISCLOSE-wp
          servicePort: 80
  - host: www.ADOMAINIDONTWISHTODISCLOSE
    http:
      paths:
      - path: /
        backend:
          serviceName: ADOMAINIDONTWISHTODISCLOSE-wp
          servicePort: 80

Ingress Service:

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-svc
  labels:
    app: nginx-ingress
spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
    - name: https
      protocol: TCP
      port: 443
  selector:
    app: nginx-ingress
  type: LoadBalancer

Replication Controller:

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-ingress-rc
  labels:
    app: nginx-ingress
spec:
  replicas: 1
  selector:
    app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
    spec:
      containers:
      - image: nginxdemos/nginx-ingress:latest
        imagePullPolicy: Always
        name: nginx-ingress-controller
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        # Uncomment the lines below to enable extensive logging and/or customization of
        # NGINX configuration with configmaps
        #args:
         #- -v=3
         #- -nginx-configmaps=default/nginx-config

ConfigMap:

apiVersion: v1
metadata:
  name: kube-lego
data:
  # modify this to specify your address
  lego.email: "beechhorn@ADOMAINIDONTWISHTODISCLOSE"
  # configure letencrypt's production api
  lego.url: "https://acme-v01.api.letsencrypt.org/directory"
kind: ConfigMap

Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-lego
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kube-lego
    spec:
      containers:
      - name: kube-lego
        image: jetstack/kube-lego:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: LEGO_EMAIL
          valueFrom:
            configMapKeyRef:
              name: kube-lego
              key: lego.email
        - name: LEGO_URL
          valueFrom:
            configMapKeyRef:
              name: kube-lego
              key: lego.url
        - name: LEGO_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: LEGO_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 1

Logs:

time="2017-04-05T10:40:15Z" level=info msg="cert expires in 89.9 days, no renewal needed" context="ingress_tls" expire_time=2017-07-04 08:49:00 +0000 UTC name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:40:15Z" level=info msg="no cert request needed" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:40:15Z" level=info msg="creating new secret" context=secret name=ADOMAINIDONTWISHTODISCLOSE-tls namespace=default
time="2017-04-05T10:40:15Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:40:15Z" level=info msg="requesting certificate for ADOMAINIDONTWISHTODISCLOSE,www.ADOMAINIDONTWISHTODISCLOSE" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:41:23Z" level=warning msg="authorization failed after 1m0s: reachabily test failed: wrong status code '404'" context=acme domain=www.ADOMAINIDONTWISHTODISCLOSE
time="2017-04-05T10:41:31Z" level=warning msg="authorization failed after 1m0s: reachabily test failed: wrong status code '404'" context=acme domain=ADOMAINIDONTWISHTODISCLOSE
time="2017-04-05T10:41:31Z" level=info msg="creating new secret" context=secret name=ADOMAINIDONTWISHTODISCLOSE-tls namespace=default
time="2017-04-05T10:41:31Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:41:31Z" level=info msg="requesting certificate for ADOMAINIDONTWISHTODISCLOSE,www.ADOMAINIDONTWISHTODISCLOSE" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:42:32Z" level=warning msg="authorization failed after 1m0s: reachabily test failed: Get http://ADOMAINIDONTWISHTODISCLOSE/.well-known/acme-challenge/_selftest: dial tcp: lookup ADOMAINIDONTWISHTODISCLOSE on 10.0.0.10:53: no such host" context=acme domain=ADOMAINIDONTWISHTODISCLOSE
time="2017-04-05T10:42:44Z" level=warning msg="authorization failed after 1m0s: reachabily test failed: Get http://www.ADOMAINIDONTWISHTODISCLOSE/.well-known/acme-challenge/_selftest: dial tcp: lookup www.ADOMAINIDONTWISHTODISCLOSE on 10.0.0.10:53: no such host" context=acme domain=www.ADOMAINIDONTWISHTODISCLOSE
time="2017-04-05T10:42:44Z" level=info msg="cert expires in 89.9 days, no renewal needed" context="ingress_tls" expire_time=2017-07-04 08:49:00 +0000 UTC name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:42:44Z" level=info msg="no cert request needed" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:42:44Z" level=info msg="cert expires in 89.9 days, no renewal needed" context="ingress_tls" expire_time=2017-07-04 08:49:00 +0000 UTC name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:42:44Z" level=info msg="no cert request needed" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:42:44Z" level=info msg="cert expires in 89.9 days, no renewal needed" context="ingress_tls" expire_time=2017-07-04 08:49:00 +0000 UTC name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:42:44Z" level=info msg="no cert request needed" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:42:44Z" level=info msg="creating new secret" context=secret name=ADOMAINIDONTWISHTODISCLOSE-tls namespace=default
time="2017-04-05T10:42:44Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:42:44Z" level=info msg="requesting certificate for ADOMAINIDONTWISHTODISCLOSE,www.ADOMAINIDONTWISHTODISCLOSE" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:43:50Z" level=warning msg="authorization failed after 1m0s: reachabily test failed: wrong status code '404'" context=acme domain=www.ADOMAINIDONTWISHTODISCLOSE
time="2017-04-05T10:43:54Z" level=warning msg="authorization failed after 1m0s: reachabily test failed: wrong status code '404'" context=acme domain=ADOMAINIDONTWISHTODISCLOSE
time="2017-04-05T10:43:54Z" level=info msg="creating new secret" context=secret name=ADOMAINIDONTWISHTODISCLOSE-tls namespace=default
time="2017-04-05T10:43:54Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
time="2017-04-05T10:43:54Z" level=info msg="requesting certificate for ADOMAINIDONTWISHTODISCLOSE,www.ADOMAINIDONTWISHTODISCLOSE" context="ingress_tls" name=ADOMAINIDONTWISHTODISCLOSE-ing namespace=default
gianrubio commented 7 years ago

I haven't used nginx from nginxdemos so I'm not sure if it's possible to use kube-lego with it. I suggest you to check the nginx example to setup certificates.

If it can't help you:

metalshark commented 7 years ago

@gianrubio

was having trouble pulling the image for gcr.io/google_containers/nginx-ingress-controller but have just noticed it doesn't have a "latest" tag so that was probably the issue. Will give it a go (out of hours due to downtime).

Are the namespaces important/required or just best practice?

gianrubio commented 7 years ago

Are the namespaces important/required or just best practice?

For now I recommend you to follow the namespace, when we figure it out the issue you can change the namespace.

metalshark commented 7 years ago

@gianrubio thanks for coming back to me on that point. There will be a window of maintenance this weekend for testing. Am hoping it's as simple as me going off-pisté from the documentation.