reachability test failed: wrong status code '502'" context=acme

[ciokan]

I'm trying to run it on GKE with kubernetes version 1.7

I can access my domain via http and I can see the ingress rules pointing to kube lego and my container accordingly:

Endpoint | Service
example.com/.well-known/acme-challenge/* | kube-lego-gce
example.com/* | backend

curl works ok from outside towards my domain which is responding with 200. Whenever I try to do a self test connection for the lego endpoint I get a 502 immediately - which fails inside lego as well.

kube-lego   2017-10-08T18:14:48.954278621Z  time="2017-10-08T18:14:48Z" level=debug msg="testing reachability of http://example.com/.well-known/acme-challenge/_selftest" context=acme domain=example.com

kube-lego   2017-10-08T18:14:41.834031539Z  time="2017-10-08T18:14:41Z" level=debug msg="error while authorizing: reachability test failed: wrong status code '502'" context=acme domain=example.com

Any idea of what can cause this?

[domparry]

Hi there, double check that the health check on the ingress shows the kube lego backend as healthy. More than likely, it isn't pointing to /healthz

[ciokan]

@domparry I think it is:

{
    "apiVersion": "extensions/v1beta1",
    "kind": "Deployment",
    "metadata": {
        "name": "kube-lego",
        "namespace": "kube-lego"
    },
    "spec": {
        "replicas": 1,
        "template": {
            "metadata": {
                "labels": {
                    "app": "kube-lego"
                }
            },
            "spec": {
                "containers": [{
                    "name": "kube-lego",
                    "image": "jetstack/kube-lego:canary",
                    "imagePullPolicy": "Always",
                    "ports": [{
                        "containerPort": 8080
                    }],
                    "env": [{
                            "name": "LEGO_LOG_LEVEL",
                            "value": "debug"
                        },
                        {
                            "name": "LEGO_EMAIL",
                            "valueFrom": {
                                "configMapKeyRef": {
                                    "name": "kube-lego",
                                    "key": "lego.email"
                                }
                            }
                        },
                        {
                            "name": "LEGO_URL",
                            "valueFrom": {
                                "configMapKeyRef": {
                                    "name": "kube-lego",
                                    "key": "lego.url"
                                }
                            }
                        },
                        {
                            "name": "LEGO_NAMESPACE",
                            "valueFrom": {
                                "fieldRef": {
                                    "fieldPath": "metadata.namespace"
                                }
                            }
                        },
                        {
                            "name": "LEGO_POD_IP",
                            "valueFrom": {
                                "fieldRef": {
                                    "fieldPath": "status.podIP"
                                }
                            }
                        }
                    ],
                    "readinessProbe": {
                        "httpGet": {
                            "path": "/healthz",
                            "port": 8080
                        },
                        "initialDelaySeconds": 5,
                        "timeoutSeconds": 1
                    }
                }]
            }
        }
    }
}

The kube lego backend is not healthy though.

[domparry]

yeah, so it doesn't always use the readinessProbe when setting up the healthcheck.... You can go fix it under compute Engine -> Health Checks... It's a PITA, but I've learned that this is the main reason for your issue... Every time I change the ingress, I have to go re-point the health checks...

[ciokan]

Somehow the health check was checking the wrong port allocated.