search

GSA-TTS / datagov-brokerpak-eks

Broker AWS EKS instances using the OSBAPI (eg from cloud.gov)
Other
9 stars 7 forks source link

Ensure all inter-pod traffic uses TLS #8

Open mogul opened 3 years ago

mogul commented 3 years ago

User Story

In order to have TLS on every network hop between the outside world and individual pods, we want EKS clusters configured to use AWS App Mesh and cert-manager.

Acceptance Criteria

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

This work will help us meet our compliance requirements. See section 10.9.6.

Sketch

For this story, we only need to work up through step 4.1 of the referenced blog post... That is, we want to demonstrate mTLS between the nginx-ingress pod and the 2048 pod.

We can work up through step 5 (TLS between the ALB controller and nginx-ingress controller) in a separate/future story.

We're now considering 4 options going forward:

  1. Remove nginx-ingress to get as close to the AWS-supported configuration as possible (adds ALB costs)
  2. Try the new solr-operator support for inter-node TLS (solves for Solr, further work needed in future for other k8s services)
  3. Try the AWS+Kong documented method that uses Kong as the ingress controller (keeps single ALB)
  4. Keep trying to debug existing path

See also https://docs.aws.amazon.com/app-mesh/latest/userguide/getting-started-kubernetes.html

mogul commented 3 years ago

We have to figure out why our annotations aren't making it all the way from the helm_release resource to the pods.

adborden commented 3 years ago

As we're re-evaluating our approach, I wanted to mention another option, the Solr Operator has support for mutual TLS. It doesn't look simple to implement and it doesn't look like it covers ZooKeeper but it might end up being an overall simpler solution that meets the compliance requirements.

mogul commented 3 years ago

It appears this is new functionality as of version 0.3.0 of solr-operator, which is not yet released. However, it's due any day now! Here's the 6-day-old ChangeLog.

mogul commented 3 years ago

BTW, I think the inter-pod TLS functionality is relatively straightforward if cert-manager is installed in the k8s cluster. It's the mTLS (client-certificate) part that is more complex.

mogul commented 3 years ago

Srini is examining this alternate approach.

mogul commented 3 years ago

For the record: it turns out that if you're using Fargate, AWS AppMesh is the only viable service mesh option for now. Brought to you after I read up on linkerd and got excited about it being simpler, only to find this post saying Istio/App Mesh is the only game in Fargate town.

mogul commented 2 years ago

Cilium now supports a sidecar-free deployment mode using eBPF, and also provides ingress controller functionality! So we may want to switch to it instead of the ingress-nginx controller and Calico, and get secure east-west traffic to boot. https://github.com/cilium/cilium/releases/tag/v1.12.0