TLS Termination at Load Balancer

[amybachir]

What changes do I need to make to terminate TLS at the load balancer?

I've already done the following:

1. I'm using AWS certificate manager so I added my certs arn to istio-ingressgatway service.
2. I removed 443 port config from the Gateways.

I noticed there are some EnvoyFilter objects. I think I might have a problem with the proxy protocol filter. Should I remove this? Is there anything else I should be aware of?

Thanks much!

[davidspek]

@amybachir Sorry for the slow reply, I've been very busy with something new that should make deploying Kubeflow a whole lot easier and more secure. I haven't tested this, but you should just need to remove port 443 and the redirect to it from the Gateways. You'd also need to set the correct loadbalancer type. Have you already tried removing the proxy protocol filter?

[soleares]

@amybachir Did you have any luck getting this to work? I'm trying to do something similar - use ACM certs with the NLB and re-encrypt with self-signed certs in the cluster for end-to-end encryption. I've added the annotations to the load balancer but I'm getting filter_chain_not_found in the envoy proxy logs.

[amybachir]

@soleares Yes, I got this working! However, I'm terminating tls at the AWS load balancer and using clear text for in-cluster communications so all communications past the load balancer are http. I can trace back my steps and provide a list of modifications I've done to get this working but I do remember having to remove the proxy protocol filter for sure and turning on insecure mode for argocd deployment.

[davidspek]

@soleares I don’t think you can do TLS termination on an NLB. You probably need to change to an ALB which is at layer 7, and as mentioned remove the proxy protocol.

What is the reason for wanting to use ACM certs and then self-signed certificates in the cluster?

[amybachir]

@soleares I don’t think you can do TLS termination on an NLB. You probably need to change to an ALB which is at layer 7, and as mentioned remove the proxy protocol.

What is the reason for wanting to use ACM certs and then self-signed certificates in the cluster?

@DavidSpek good catch! I didn't notice @soleares was referring to creating an NLB. Yes, definitely you need an ALB instead! I removed the service annotations from istio-ingressgateway service and created an ingress object with an ALB:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/load-balancer-name: <<__aws_load_balancer.name__>>
    alb.ingress.kubernetes.io/certificate-arn: <<__istio_ingressgateway_aws_load_balancer_ssl_cert__>>
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/security-groups: YOUR-SECURITY-GROUPS
    alb.ingress.kubernetes.io/subnets: YOUR-SUBNETS
    kubernetes.io/ingress.class: alb
    external-dns.alpha.kubernetes.io/hostname: <<__route53.hosted_zone_hostnames__>>
  finalizers:
  - ingress.k8s.aws/resources
  generation: 1
  labels:
    kustomize.component: istio-ingress
  name: istio-ingress
  namespace: istio-system
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: istio-ingressgateway
          servicePort: 80
        path: /*

[soleares]

@amybachir @DavidSpek Thank you. It's good to know that the ALB setup works with this distribution. I have this setup running with Kubeflow 1.2 in production.

From what I'm reading NLB should support TLS termination: https://aws.amazon.com/blogs/aws/new-tls-termination-for-network-load-balancers/. It also supports adding multiple ACM certs and the AWS load balancer controller annotation supports passing a list. But I'm not having luck getting it to work with Istio ingress-gateway.

The reason I'm trying to get this to work is that:

• I need to encrypt traffic all the way through for compliance reasons.
• I can't use cert-manager to automatically issue certs - due to security restrictions on our domain. So I'd prefer ACM so that I don't have to manually renew/update certs every year.

So I'll probably either:

• Use manually issued certs for now and investigate this further later
• Go back to using an ALB
• Try this ingress approach with the NLB to see if I get a different result.