Public traffic routing

[lottspot]

Incoming HTTP and HTTPS traffic needs to be routed to the appropriate container.

[lottspot]

For HTTP traffic, the best solution will be to configure an ingress controller and configure project-specific ingress resources.

Unfortunately, no current ingress controller implementations include support for SNI, so for the time being, we should deploy a HAProxy container which can receive all incoming HTTPS traffic and handle SSL termination.

[lottspot]

For applications which wish to redirect all HTTP traffic over HTTPS, we should configure a nginx container whose sole job is to redirect all incoming HTTP requests to a matching HTTPS request and route all HTTP traffic for the application to this container.

[lottspot]

I previously took the following notes during my research of this problem:

Kubernetes provides a ingress resource which, in an ideal world, could be used to route inbound public traffic to the appropriate backend service without the need for any load balancers. However, as of February 2016, no ingress controller implementations appear to support SNI; an absolute necessity in order to properly suport multiple public HTTPS applications on the same cluster. Thus, as of time of writing, it makes the most sense to simply implement a HAProxy service which manually proxies public traffic to backend services.

The public HAProxy instances should be run on infra which is expected to have stable availability; this typically means avoiding running the HAProxy instances on worker nodes.

HAProxy itself does not necessarily need dynamic configuration; in order to successfully proxy to backend services, a HAProxy instance needs two capabilities:

1. Perform DNS resolution of service name
2. Route to IP address returned by name resolution

In the typical case, this means running in an environment which is networked with the cluster overlay network and which resolves against the cluster resolvers

[themightychris]

Would it be a best practice for our frontend router to just letsencrypt and force HTTPS on all traffic without any project configuration?

[lottspot]

I think that's a great point, and it is definitely what we should strive to be doing. Configuring our ingress traffic to be fully HTTPS with letsencrypt backed certificates is a distinct task from simply routing any HTTP/HTTPS traffic at all, so I'll go ahead and open up a new issue for it

[lottspot]

Knowing that we ultimately want to route public traffic in this way though does affect how much effort it makes sense for us to put into configuring HTTP traffic routing. That is to say, it wouldn't be worth it to put effort into configuring special kubernetes ingress controllers and resources, since they would just be thrown out for a simple nginx container that redirects to https.

[MooseQuest]

HAProxy can be scripted into a container which would allow the HAproxy to handle the SNI coming in to provide the SSL Termination... Are we doing one way SSL though?

[lottspot]

Worth heavily considering is envoy. Will likely be experimenting with the possibility.

[MooseQuest]

For some reason, I didn't think of this, but I have Layer 7 routing system called SHIFT:

Splunk High Ingest Flexible Transit.

It can be used for HTTP Proxying and L7 Routing.

I use it for moving data around the country when systems are down.

[lottspot]

Envoy is a really cool project, but for right now, it just makes more sense to go with what we know and what we can deploy quickly. An nginx DaemonSet has been deployed to the alpha cluster to handle routing incoming traffic.

[lottspot]

How rude of me-- I never noted what commit this was completed in. Done in ce335c4