argoflow / argoflow-aws

Argoflow-AWS has been superseded by deployKF
GNU Affero General Public License v3.0
44 stars 29 forks source link

TLS Termination at Load Balancer #160

Open amybachir opened 3 years ago

amybachir commented 3 years ago

What changes do I need to make to terminate TLS at the load balancer?

I've already done the following:

  1. I'm using AWS certificate manager so I added my certs arn to istio-ingressgatway service.
  2. I removed 443 port config from the Gateways.

I noticed there are some EnvoyFilter objects. I think I might have a problem with the proxy protocol filter. Should I remove this? Is there anything else I should be aware of?

Thanks much!

davidspek commented 3 years ago

@amybachir Sorry for the slow reply, I've been very busy with something new that should make deploying Kubeflow a whole lot easier and more secure. I haven't tested this, but you should just need to remove port 443 and the redirect to it from the Gateways. You'd also need to set the correct loadbalancer type. Have you already tried removing the proxy protocol filter?

soleares commented 3 years ago

@amybachir Did you have any luck getting this to work? I'm trying to do something similar - use ACM certs with the NLB and re-encrypt with self-signed certs in the cluster for end-to-end encryption. I've added the annotations to the load balancer but I'm getting filter_chain_not_found in the envoy proxy logs.

amybachir commented 3 years ago

@soleares Yes, I got this working! However, I'm terminating tls at the AWS load balancer and using clear text for in-cluster communications so all communications past the load balancer are http. I can trace back my steps and provide a list of modifications I've done to get this working but I do remember having to remove the proxy protocol filter for sure and turning on insecure mode for argocd deployment.

davidspek commented 3 years ago

@soleares I don’t think you can do TLS termination on an NLB. You probably need to change to an ALB which is at layer 7, and as mentioned remove the proxy protocol.

What is the reason for wanting to use ACM certs and then self-signed certificates in the cluster?

amybachir commented 3 years ago

@soleares I don’t think you can do TLS termination on an NLB. You probably need to change to an ALB which is at layer 7, and as mentioned remove the proxy protocol.

What is the reason for wanting to use ACM certs and then self-signed certificates in the cluster?

@DavidSpek good catch! I didn't notice @soleares was referring to creating an NLB. Yes, definitely you need an ALB instead! I removed the service annotations from istio-ingressgateway service and created an ingress object with an ALB:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/load-balancer-name: <<__aws_load_balancer.name__>>
    alb.ingress.kubernetes.io/certificate-arn: <<__istio_ingressgateway_aws_load_balancer_ssl_cert__>>
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/security-groups: YOUR-SECURITY-GROUPS
    alb.ingress.kubernetes.io/subnets: YOUR-SUBNETS
    kubernetes.io/ingress.class: alb
    external-dns.alpha.kubernetes.io/hostname: <<__route53.hosted_zone_hostnames__>>
  finalizers:
  - ingress.k8s.aws/resources
  generation: 1
  labels:
    kustomize.component: istio-ingress
  name: istio-ingress
  namespace: istio-system
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
        path: /*
soleares commented 3 years ago

@amybachir @DavidSpek Thank you. It's good to know that the ALB setup works with this distribution. I have this setup running with Kubeflow 1.2 in production.

From what I'm reading NLB should support TLS termination: https://aws.amazon.com/blogs/aws/new-tls-termination-for-network-load-balancers/. It also supports adding multiple ACM certs and the AWS load balancer controller annotation supports passing a list. But I'm not having luck getting it to work with Istio ingress-gateway.

The reason I'm trying to get this to work is that:

So I'll probably either: