Kubernetes Ingress Controller

[hunter]

Kubernetes has introduced an nginx based docker image as the frontend controller for their Ingress router (name based virtual hosting/load balancer)

The nginx implementation is basic. A small Go binary queries the Kubernetes APIs, pulls a list of Ingress resources, writes an nginx.conf and then reloads the server - https://github.com/kubernetes/contrib/tree/master/Ingress/controllers/nginx-alpha

Caddy seems like a better fit for this (with lets encrypt, graceful restarts and use in other Go apps coming in 0.8)

I'm just adding this as a future idea. Once 0.8 is out we might look at implementing something to give a Caddy alternative for Kubernetes Ingress (unless someone else beats us to it ;)

[mholt]

Sounds cool; I'm not really familiar with Ingress or Kubernetes (shame on me, I know) but am always happy to see how people build on Caddy! Love seeing useful stuff come out of it.

[jbeda]

(I'm a cofounder of Kubernetes and am looking at Caddy in other contexts)

This would be cool but the ingress systems that are being developed for Kubernetes are really about the load balancing/reverse proxy-ing. The idea is to serve public facing HTTP(S) backed by a dynamic set of containers scheduled on the cluster. From a quick look through the Caddy docs (I'm a noob here) it looks like doing that type of load balancing isn't a goal for Caddy right now.

But I do agree the TLS-by-default feature (and HTTP2) is an awesome combo.

Edit: I missed the proxy directive! To make this work we'd have to have something that can actively monitor the Kuberenetes API and configure sites and proxies automatically. Perhaps do this from the outside once there is a Caddy API?

[mholt]

You could do it either way. If you want it built right into the web server, a plugin is the way to go. If you prefer to keep it separate, you could use the Caddy API, coming in version 0.9. (I definitely anticipate that adding and removing proxy backends will be a very popular task for the API.)

[jordic]

I know a little of caddy and also a little of kubernetes.

I think the only rouge point can be the rate limits that LE has for issuing certs: currently 7 certs per week per domain. Also, I'm not sure if it also counts for the rate, when a cert is already issued and caddy asks again for it.. (Storage on kubernetes pods is efemeral).

For the last it can be also solved, but requires some persistent storage for the certs...

@mholt if caddy has an api, is not necessary to build the Ingress controller as a pluguin.. It can be a new binary that listens for events on kubernetes master and reconfigs caddy on the go. A pod can have multiple process.

[pwFoo]

Could also be interesting for other dicker setups without Kubernetes. Instead oft optimize it for Kubernetes it should be universal for docker.

[mholt]

@hunter @jbeda Have you seen #564? That's pretty close to what you're thinking of, no?

[jbeda]

Very cool -- would be great to get it hooked up end to end w/ k8s after that PR gets merged.

[bprashanth]

I think the gap that caddy would fill nicely is if there's an existing Ingress controller backed by, say haproxy, that wants real certs with the least amount of hassle. One way to address this (instead of making caddy listen to something like etcd and do things haproxy might be better suited for) is to have a caddy controller that just acquires certs for Ingresses like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
spec:
  tls:
  - hosts:
    - foo.bar.com
    - foobar.bar.com
    secretName: empty
  backend:
    serviceName: echoserver-https
    servicePort: 80

A cluster wide caddy controller would notice this, get the certs and populate the secret field. Only then would the haproxy/nginx/cloudlb pick the Ingress up.

Thoughts? (I feel like you might just tell me to use LEGO directly) Related discussion https://github.com/kubernetes/kubernetes/issues/19899.

[bprashanth]

A cluster wide caddy controller would notice this, get the certs and populate the secret field. Only then would the haproxy/nginx/cloudlb pick the Ingress up.

Or we make a caddy sidecar that one can add to any existing Ingress controller in https://github.com/kubernetes/contrib/tree/master/ingress/controllers

[mholt]

Caddy's not a certificate daemon/manager; for that, you want lego.

[bprashanth]

Yes that's what I suspected you'd say. Well, a kubernetes backend for caddy would still be cool. I'm willing to help with the Kubernetes bits if anyone is willing to drive the effort.

[faddat]

@jbeda @mholt How to go about getting #564 merge-ready? Speaking for myself, this would make caddy OMFG unstoppable. Right now cluster ingeress is really needlessly hard, weather it's a docker swarm cluster or a kube cluster, fact is it's pointlessly hard for both. #564 would seem to solve both use cases cleanly. I'm going to do some testing and will let you know if that ends up being the case.

[ultimateboy]

I've been working on a similar solution using caddy as a front end load balancer for kubernetes services, creating proxies in an automatically generated Caddyfile based on annotations on services. The majority of the code came from the Deis Router which uses nginx. The fork is in very early stages, but given the relevancy to this issue, I thought I'd post a link: https://github.com/drud/router

[jbeda]

@faddat I'd love to help but I'm over committed right now. 😞

[mholt]

@hunter, @jbeda - and everyone else in this issue - thanks for your comments! Just so you know, I have linked to this issue from a thread in our community forum where we are aggregating ideas/requests for Caddy plugins.

If you want to continue discussing the plugin before development begins on it, feel free to start a new topic on the forum. :+1: Hope to see you participate with us there!

If, however, this is something that is actively being developed as a Caddy plugin, feel free to re-open and discuss its development right here. :smile: Just trying to clear through my backlog, ya know...

For anyone looking for k8s ingress controller: https://github.com/wehco/caddy-ingress-controller

Update: We have started an official one, but we need the community's help to finish it: https://github.com/caddyserver/ingress