kube-lego with multiple ingress-controllers

[krogon-dp]

I am using multiple ingress controllers. Moreover, those are multiple nginx controllers, with different ingress-class value.

I have changed LEGO_SUPPORTED_INGRESS_CLASS variable to satisfy my environment. When I changed ingress definition (tls-acme annotation + tls specification) kube-lego tries to request new certificate. But it does not create entry in kube-lego-nginx ingress. I guess it is due to the fact that this ingress has annotation "ingress.class: nginx" and kube-lego did not create another ingress definition.

When I created manually second ingress (kube-lego-otherclass-manual) and placed there rule for /.well-known/acme-challenge everything was successful (despite the fact I had to restart controller as part of https://github.com/jetstack/kube-lego/issues/212)

[vaijab]

I am hitting the same issue actually. I see that latest stable release of kube-lego (0.1.5) has a bug where it is unable to read LEGO_SUPPORTED_INGRESS_CLASS properly. The bug is fixed in https://github.com/jetstack/kube-lego/pull/223/commits/47ffbbd755247737b81de22abe6feb849716da00 on master.

We need a new release of kube-lego. Could you release a patch version, @munnerz ?

[krogon-dp]

I suppose the concept of separating ingress.class and ingress.provider implemented in #202 covers this use-case. Waiting for new release.

[fergusstrange]

Would be great to see a release of this soon as the feature would be really useful for us. Let me know if there's any help needed in testing / verifying to get this through.

[rowleyaj]

I have the same use case and issues.

As @krogon-dp mentioned #202 is relevant, #223 introduces the commit @vaijab mentions. It however also removes setting a default provider (LEGO_DEFAULT_INGRESS_PROVIDER) too, meaning that every ingress using nginx has to now have an additional annotation set.

@jackhopner raised this as an issue on #223 but after it was merged, so I think we will need another PR to add that functionality back in, without breaking GCE support.

For those waiting for a release I found that there are pre-release build tags (https://hub.docker.com/r/jetstack/kube-lego/tags/) that can be used to pull a more up to date image. I'm currently running tag master-2368 this starts up as kube-lego 0.1.6-dev-784fc8fe so commit 784fc8f (latest master at time of writing).

I believe we may also need https://github.com/kubernetes/ingress/pull/1434 to actually be able to successfully reload nginx with updated certs though.

[jackhopner]

@rowleyaj I'll submit a PR in the next few days

[hawky-4s-]

I've got the same problem. Is anyone working on it?

[jackhopner]

@hawky-4s- PR has been created see https://github.com/jetstack/kube-lego/pull/257

[tannerjfco]

Just wanted to note that I have the same issue as reported in OP even with using the latest development tag pushed to dockerhub (master-3163 pushed yesterday). So it doesn't seem like the referenced fixes that have come in are addressing the issue as originally reported.

I have the following set, if its helpful:

LEGO_SUPPORTED_INGRESS_CLASS: site-prod,site-stage
LEGO_DEFAULT_INGRESS_CLASS: site-prod
LEGO_DEFAULT_INGRESS_PROVIDER: nginx

kube-lego will detect ingresses with either "site-prod" or "site-stage" class and try to create/renew the cert, but kube-lego only creates 1 ingress resource for the acme challenge with the "site-prod" ingress class. It seems like there should be an ingress resource created for every supported ingress class, but I've only ever seen 1 resource created.

[eversC]

based on current master HEAD (f8cd69c), it looks as though kube-lego is still creating a single ingress per provider ("nginx" or "gce"), rather than per supported class, as @tannerjfco states

'per provider' ingress' won't work when using multiple ingress controllers that have different ingress classes, as the kube-lego ingress will be ignored by all but one of the controllers

the OP mentions creating a second ingress as their workaround. Sounds like having kube-lego create an ingress per supported class would work, though not sure if it's quite so simple as that?

[abh]

FWIW in our setup we solved this by just running two deployments of kube-lego, one for each ingress class. It works fine that way and was very easy to setup (copy the deployment and have the second one not read the environment variables from the config map for class and provider).

[eversC]

thanks @abh - I was going to attempt that next, good to know it works. It seems cleaner than creating additional ingress' outside of kube-lego.

I wonder if it's worth keeping this open - is using a single kube-lego deployment for multiple ingress classes an intended use-case?

[eversC]

I (finally) got round to trying out multiple kube-lego deployments, and it wasn't so straight forward (for me at least!)

the 2nd kube-lego deployment wasn't actually creating new ingress and service resources, as kube-lego was just using it's default names for them, and detecting they already existed.

It also didn't have the selector (in the created service) set to the 2nd kube-lego app, which led to the acme challenge coming into the 1st kube-lego deployment, and subsequently the error:

msg="error while authorizing: reachability test failed: received id (<actual_id>) did not match expected (<expected_id>)"

setting these env vars on one of the kube-lego deployments resolved it for me:

LEGO_SERVICE_NAME_NGINX  (to something other than default "kube-lego-nginx")
LEGO_INGRESS_NAME_NGINX  (to something other than default "kube-lego-nginx")
LEGO_SERVICE_SELECTOR  (to something other than default "kube-lego", must match the label (app=) of your alternative kube-lego app)

see https://github.com/jetstack/kube-lego#environment-variables