`istio-gateway` charm does not expose an option for internal LB when deploying on EKS

gustavosr98 commented 1 year ago

Bug Description

Deploying istio gateway as a LoadBalancer on AWS EKS will by default create an AWS Load Balancer that is exposed to the Internet by default

There are multiple cases where because of security constraints this needs to be only exposed to internal networks and not to the Internet. This also applies when deploying a private EKS that is only available on private VPCs and not publicly available VPCs.

I would like to have exposed the option to add the required annotation on the charm to deploy the LoadBalancer as a internal LoadBalancer rather than a private one

See https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/annotations/#lb-internal

To Reproduce

juju deploy --trust --debug istio-gateway \ --channel=1.16/stable \ --config kind=ingress \ --config gateway_service_type=LoadBalancer

Environment

AWS EKS Juju 2.9.45 Charm channel: 1.16/stable Charm revision: 663

Relevant log output

n/a

Additional context

Some additional external reference https://anshul-prakash17.medium.com/creating-an-internal-load-balancer-in-aws-eks-using-istio-3defcb51c6a2

Workaround

Create the service as a NodePort initially, so the LoadBalancer doesn't get created on AWS

juju deploy --trust --debug istio-gateway \
--channel=1.16/stable \
--config kind=ingress \
--config gateway_service_type=NodePort

Edit the manifest.yaml located on the charm source code to add the service annotation for internal lb

# [..]
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
labels:
app: istio-{{ kind }}gateway
install.operator.istio.io/owning-resource: unknown
istio: {{ kind }}gateway
istio.io/rev: default
operator.istio.io/component: "{{ kind|capitalize }}Gateways"
release: istio
name: istio-{{ kind }}gateway-workload
namespace: {{ namespace }}
spec:
ports:
- name: http2
  port: 80
  protocol: TCP
  targetPort: 8080
- name: https
  port: 443
  protocol: TCP
  targetPort: 8443
selector:
istio: {{ kind }}gateway
type: {{ 'ClusterIP' if kind == 'egress' else gateway_service_type }}

Push the new manifest.yaml into the deployed charm

juju scp new-manifest.yaml istio-gateway/0:/var/lib/juju/agents/unit-istio-gateway-0/charm/src/manifest.yaml

Change the service type to LoadBalancer

juju config istio-gateway gateway_service_type=LoadBalancer

gustavosr98 commented 1 year ago

Since there are going to be multiple annotations per cloud (AWS, GCP, Azure, etc) I believe an approach could be to expose on the charm an options like extra-annotations to solve any additional customization required per cloud

For this other too as well:

orfeas-k commented 10 months ago

Thank you @gustavosr98 for the detailed report. I will bring this up in one of the team's meeting in order to evaluate this feature, since this is something that was also requested some time ago in #66.

gustavosr98 commented 2 months ago

I have not tested but found these kubernetes-service-annotations, within those there is a kubernetes-service-type option worth testing during deployment type to avoid hacking into the charm code

# juju config istio-ingressgateway | grep kubernetes
  kubernetes-ingress-allow-http:
  kubernetes-ingress-class:
  kubernetes-ingress-ssl-passthrough:
  kubernetes-ingress-ssl-redirect:
  kubernetes-service-annotations:
  kubernetes-service-external-ips:
  kubernetes-service-externalname:
  kubernetes-service-loadbalancer-ip:
  kubernetes-service-loadbalancer-sourceranges:
  kubernetes-service-target-port:
  kubernetes-service-type:

canonical / istio-operators