search

jetstack / kube-lego

DEPRECATED: Automatically request certificates for Kubernetes Ingress resources from Let's Encrypt
Apache License 2.0
2.16k stars 267 forks source link

reachability test failed: wrong status code 404 #293

Open patrickhempel opened 6 years ago

patrickhempel commented 6 years ago

My kube-lego setup is not able to get a cert for one particular domain. 

time=2017-12-11T18:34:38Z level=debug msg=testing reachability of http://sub.domain.xyz/.well-known/acme-challenge/_selftest context=acme domain=sub.domain.xyz

Fetching the selftest url by hand yields a HTTP 200 OK Status.

curl -v -4 http://sub.domain.xyz/.well-known/acme-challenge/_selftest
*   Trying xxx.xxx.xxx.xxx...
* Connected to sub.domain.xyz (xxx.xxx.xxx.xxx) port 80 (#0)
> GET /.well-known/acme-challenge/_selftest HTTP/1.1
> Host: sub.domain.xyz
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.13.7
< Date: Mon, 11 Dec 2017 19:05:19 GMT
< Content-Type: text/plain
< Content-Length: 16
< Connection: keep-alive
<
* Connection #0 to host sub.domain.xyz left intact

Other domains are working well. I'm using jetstack/kube-lego:0.1.5 with kubernetes 1.8.

Thank you in advance, Patrick

hmacias commented 6 years ago

Same here. I am on k8s 1.8.3 and get:

level=warning msg="authorization failed after 1m0s: reachability test failed: wrong status code '404'"

alterationx10 commented 6 years ago

I had this issue recently. This is definitely specific to me/my network, but not a crazy scenario:

I effectively had a routing issue in my cluster.

I had a DNS record pointing back to my Master node (which has a public IP, and is routed on my internal network differently than the workers). I could access the site/domain just fine there. I could access it from my home network just fine.

On my other kubernetes nodes, however, trying to resolve the public IP caused a bit of an issue, as the internal network "knew better", and tried (unsuccessfully) to re-route.

If you haven't, try your curl command on the node the pod is running in, and see if it still works. If it doesn't, perhaps reroute traffic to that IP via a NAT Gateway.

hmacias commented 6 years ago

Thank for the valuable comments above.

In my case it turned out to be another ingress (from an old kube-lego installation) resource causing conflict with the same domain I was trying to get set up. So when the acme challenge was attempted it was pointing to a service that no longer existed and the challenge could not complete successfully and therefore I was getting time="2017-12-20T22:21:55Z" level=debug msg="testing reachability of http:///.well-known/acme-challenge/_selftest" context=acme domain=smartlabel.ci.rplus.ncommerce.com As soon as I deleted the conflicting ingress pointing to the same domain lego's acme challenge finally went through and the certificate was successfully created.

My advice to troubleshoot, in case you're seeing this error is this:

  1. Set up ingress controller and wait for it to start up correctly (nginx controller in my case)
  2. Create an ingress resource that serves requests at a given path by your desired service
  3. Ensure that your ingress is working and you're able to reach your service using both HTTP and HTTPS with the DOMAIN you set up in your ingress.
  4. Once you're able to access your service (https is not secure yet at this point) using your ingress resource, then proceed to install kube-lego to install certificates.
  5. If you're still seeing the 404 error in bold above, then there's something in your env causing requests from kube-lego to go through a different route other than through your Ingress controller -> Your service path.

Good luck :)

dwjohnston commented 5 years ago

Fantastic. The problem I had was I had two ingresses on the same IP address - solution was to delete both, and then deploy the new one (without the certificate) first, followed by my existing one.

sirzak commented 5 years ago

I had this problem. I use Nginx inc ingress controller , but I think my solution suitable for everyone

Solution - Don't give a name for ingress controller starting with a,b,c its Nginx's feature - Nginx load config files by alphabet and cert-manager named own configs "%namespace%-cm-*", so Nginx load you ingress config before cert-manger

rmetzler commented 4 years ago

I had this problem too. Turned out, GKE created a new Loadbalancer with another IP and my DNS had the wrong IP. Once I found this out, the cert was issued and served.