Error 403 on HTTP-01 challenge

[pdoreau]

Hello.

I'm trying to use kube-lego with nginx controller. I used echoserver as explained in the example. However the HTTP-01 challenge fails :

time="2016-09-21T18:33:14Z" level=warning msg="Error while obtaining certificate: Errors while obtaining cert: map[mydomain.com:acme: Error 403 - urn:acme:error:unauthorized - Invalid response from http://mydomain.com/.well-known/acme-challenge/OHTuEazqigcmn1x-kSXMmzWnHNRJTrLFNfB0kZL7Py8: \"CLIENT VALUES:\r\nclient_address=('10.x.x.x', 47151) (10.x.x.x)\r\ncommand=GET\r\npath=/.well-known/acme-challenge/OHTuEazqigcmn1x-k\"\nError Detail:\n\tValidation for mydomain.com:80\n\tResolved to:\n\t\t104.y.y.y\n\tUsed: 104.y.y.y\n\n]" context=acme 

It looks normal as the echoserver catches the request and respond something not related to the token specified in the url.

How is that possible ? Is there something that I forgot or something missing in the doc ?

[stibi]

Hello, what version of the nginx-ingress-controller you have deployed? The latest one, 0.8.3? I see the same problem as you with 0.8.3. Try 0.8.1, that one works for me. Now I'm trying to find out where is the problem...

[stibi]

One more thing, do you use the --watch-namespace option with nginx-ingress-controller ?

[pdoreau]

Hi @stibi

what version of the nginx-ingress-controller you have deployed? The latest one, 0.8.3?

Yes. I used the 0.8.2 but couldn't make it work. Now 0.8.3 is deployed.

do you use the --watch-namespace option with nginx-ingress-controller

Yes.

[stibi]

0.8.2 is affected too, the problem is with the --watch-namespace…without the option, it works.

I guess the namespace restriction somehow break the events listening in the nginx controller. If you check the nginx.conf, there is for example no ACME challenge location rules (/.well-known/acme-challenge/...). That's why the certificate request is not validated and issued.

I don't understand the whole thing yet, I have to check the nginx ingress controller code, but maybe someone here could explain it in more details.

[simonswine]

Filtering on a namespace is not yet supported for kube-lego. If you only want to run kube-lego in a single namespace you could move kube-lego into that namespace.

[pdoreau]

Thanks @simonswine For each of my staging env (handled with namespaces) I need to deploy a kube-lego instance. Is that correct ? Is filtering on a namespace planned for a future kube-lego release ?

[tsloughter]

I have this same 403 issue but with GKE. I also see it responding with a 200 to the request and returning Resolved to:\n\t\t104.y.y.y\n\tUsed: 104.y.y.y\n\n.

[tsloughter]

Though I think I'm missing something based on https://blog.jetstack.io/blog/kube-lego/ since that says kube-lego sets up a separate ingress that is to serve the response for /.well-known/acme-challenge/ and mine is going to the echoserver. It looks like @pdoreau 's is as well based on the output, \"CLIENT VALUES:\r\nclient_address=(

[aknuds1]

I am having a similar issue, although I use the GKE L7 ingress. Basically, the kube-lego pod is logging reachability failures, as reachability tests are routed to my web server which responds with a 404 error:

time="2016-11-02T16:37:56Z" level=debug msg="testing reachablity of http://www.realtime-music.com/.well-known/acme-challenge/_selftest" context=acme host=www.realtime-music.com
time="2016-11-02T16:37:57Z" level=warning msg="wrong status code '404'" context=acme host=www.realtime-music.com
time="2016-11-02T16:37:57Z" level=warning msg="Error while obtaining certificate: reachabily test failed for this cert"

I've followed the GCE example in setting my environment up.

[pdoreau]

@tsloughter @aknuds1 It works well on my side after adding kube-lego deployment under the namespace of the pods I want to secure.

[aknuds1]

@pdoreau You mean kube-lego should not be in its own namespace?

[pdoreau]

@aknuds1 I'm using nginx-controller with --watch-namespace option. To make everything work, I used the same namespace everywhere : my pods (and services...), the ingress ressource, the nginx-controller and the kube-lego deployment