Ingress controller evaluation

jgmize commented 7 years ago

https://medium.com/@cashisclay/kubernetes-ingress-82aa960f658e

bookshelfdave commented 7 years ago

jgmize commented 6 years ago

Blocks #448

jgmize commented 6 years ago

https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer

bookshelfdave commented 6 years ago

https://www.getambassador.io/

bookshelfdave commented 6 years ago

See https://github.com/mozmeao/infra-private/pull/132

bookshelfdave commented 6 years ago

https://github.com/appscode/voyager

bookshelfdave commented 6 years ago

The Problem

Bedrock currently uses a git-push based method for deploying demo branches to dynamic subdomains of moz.works with TLS support. For example, pushing a git branch named demo/feature would create a demo instance with the following URL: https://bedrock-demo-feature.us-west.moz.works/.

This feature is provided by the Deis Workflow router, which is now officially unsupported. To replace the functionality that Deis Workflow provides, we need a custom Kubernetes Ingress that can route traffic from a dynamic domain name to a Kubernetes deployment with TLS. Our evaluation has been primarily performed in GCP with GKE.

We'd also like to gain experience using Kubernetes Ingress and learn how it can help solve some of our infrastructure problems.

Requirements

In our evaluation, we looked for the following features:

easy to setup, configure and maintain
- SRE should spend no more than a few hours installing and understanding how the product works
the product must provide comprehensive and up-to-date documentation
the ability to dynamically route to new K8s deployments based on domain name (rather than /path)
LetsEncrypt certificate request and auto-renewal support
- it's preferrable to use ACME cert validation with Route53
  - .moz.works / .mozmar.org domains are hosted in AWS Route53, the product should be able to easily update hosted zones a with appropriate credentials
  - ACME http validation may require application request path modifications
- bonus: automatically request LetsEncrypt wildcard certificates

Evaluation

This section contains notes on products that were evaluated:

Ambassador
- the demo application failed to route traffic every few requests under extremely low load conditions.
- no LetsEncrypt integration, TLS is setup manually
  - manual cert renewal
Traefik
- Traefik is a general solution that supports many backends, including K8s, Docker, ECS, Mesos, Zookeeper etc.
- poor Kubernetes installation documentation
  - I had to dig through blog posts and Github issues to figure out how to install in K8s
- Traefik supports Let's Encrypt, however the documentation is limited.
- I was unable to get Traefik up and running after 1.5 days (without TLS/LetsEncrypt support)

Nginx Ingress Controller

from the docs: It is built around the Kubernetes Ingress resource, using a ConfigMap to store the NGINX configuration.

TLS is supported with kube-lego, which states:

kube-lego is in maintenance mode only. There is no plan to support any new features. 
The latest Kubernetes release that kube-lego officially supports is 1.8. The officially 
endorsed successor is cert-manager.

kube-lego's successor is cert-manager, which states the following in it's readme:

This project is not yet ready to be a component in a critical production stack, 
however it is at a point where it offers comparable features to other projects 
in the space. If you have a non-critical piece of infrastructure, or are feeling 
brave, please do try cert-manager and report your experience here in the issue section.

given that the two above certificate management solutions aren't suitable for production, we moved on to other products.

Heptio Contour
- From the readme:
```
This is an early release so that we can start sharing with the community.
```
- cert support is via cert-manager, which states that it's not ready for production use.
GKE Ingress + Certsbridge
- no LetsEncrypt support, however we successfully uploaded a manually-request LetsEncrypt wildcard certificate via the GUI.
- Certsbridge is a GCP product (made by Google)
  - it currently does not support wildcard certificates
  - a cert can only have a single name (it does not support Subject Alternative Names)
  - Certsbridge is currently in alpha
  - we are also evaluating Certsbridge for use with GKE-hosted prod/stage/dev environments
    - certificates in this case would be for Cloudflare to load balancer communications.
Voyager
- out of all ingress controllers evaluated, this was the easiest to get up and running with TLS and dynamic routing
- LetsEncrypt support is per-namespace
  - for example, bedrock-demo needs it's own LetsEncrypt config and requested certs are placed in this namespace
- LetsEncrypt wildcard support added in 7.0.0, which was released a few days ago
- uses K8s CustomResourceDefinitions (a custom Ingress object)
- Voyager uses HAProxy under the hood
  - configurable number of HA proxy pods per Ingress
- Route53 LetsEncrypt verification
- certs are automatically renewed 7 days before expiration
- Voyager uses an IAM account to manage a Route53 hosted zone. For security, we prefer to use a hosted zone other than moz.works. Testing was performed with mozmar.org.
- it takes < 1 minute for a new host to appear after being added to the ingress.

Recommendation

Voyager was the easier product to setup that met all of our requirements. A wildcard certificate per ingress seems to be the easiest way to dynamically host bedrock demo instances.

bookshelfdave commented 6 years ago

cc @bensternthal ^

bensternthal commented 6 years ago

Got it.. thanks for the explanations!

mozmeao / infra