Option to share LB between Ingresses

[GGotardo]

I want to organize my cluster into multiples namespaces (app1, app2) and work with Ingress to access each of them. Something like:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app1-ing
  namespace: app1
  annotations:
    kubernetes.io/ingress.global-static-ip-name: ip-ingress-backend
spec:
  rules:
  - host: app1-service1.example.com
    http:
      paths:
      - backend:
          serviceName: nginx-1
          servicePort: 80
        path: /service1
 - host: app1-service2.example.com
    http:
      paths:
      - backend:
          serviceName: nginx-2
          servicePort: 80
        path: /service2

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app2-ing
  namespace: app2
  annotations:
    kubernetes.io/ingress.global-static-ip-name: ip-ingress-backend
spec:
  rules:
  - host: app2-service1.example.com
    http:
      paths:
      - backend:
          serviceName: nginx-1
          servicePort: 80
        path: /service1
   - host: app2-service2.example.com
    http:
      paths:
      - backend:
          serviceName: nginx-2
          servicePort: 80
        path: /service2

But when I try to do so, the following error is showed while creating the second Ingress:

googleapi: Error 400: Invalid value for field 'resource.IPAddress': 'xxxx'. Specified IP address is in-use and would result in a conflict., invalid

It tries to create another LB, but it should share the same one, just creating new backends/frontends.

[Berndinox]

Oh, sorry.. seems like i dont got the hole thread.. lol at least i have the "base" and can play around a bit, i will let you know if i can find any updates regarding your goals.

[jglick]

create a controller to join the fragments into a single ingress resource

https://github.com/jakubkulhan/ingress-merge

My use case is a slight variant: several Ingress objects (perhaps in the same namespace) with the same host but binding different Services to different paths. ingress-nginx supports this.

[Andrewangeta]

create a controller to join the fragments into a single ingress resource

https://github.com/jakubkulhan/ingress-merge

My use case is a slight variant: several Ingress objects (perhaps in the same namespace) with the same host but binding different Services to different paths. ingress-nginx supports this.

Any steps on installing without Helm? I'm just using GHActions to do a kubectl apply at the moment but everyone is using helm in a lot of scenarios. I just don't need/want it right now.

[Andrewangeta]

Also the fact that AWS implemented this already.... https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/298#issuecomment-715532217

[tpokki]

Hey @tpokki do you know where I could find some tutorial/examples in order to implement that ?

This workaround/setup with traefik works for us (nginx setup has issues, as described above).

Install traefik:

helm install -f traefik-values.yaml traefik stable/traefik

# traefik-values.yaml
fullnameOverride: traefik
rbac:
  enabled: true
serviceType: NodePort
service:
  annotations:
    cloud.google.com/backend-config: '{"default": "backendconfig"}'

Rest of the objects without helm. N.b. you need to allocate the IP address external-lb-address separately.

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: backendconfig
spec:
  timeoutSec: 300
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: external-lb-address
    networking.gke.io/managed-certificates: managedcertificate
spec:
  backend:
    serviceName: traefik
    servicePort: 80
---
apiVersion: networking.gke.io/v1beta2
kind: ManagedCertificate
metadata:
  name: managedcertificate
spec:
  domains:
    - example.com

The individual Ingress resources are annotated with

kubernetes.io/ingress.class: traefik

... to make traefik process them.

[Aut0R3V]

Looks like this issue has been up for a long time. Anything I can do here?

[hermanbanken]

On a slightly related note: we are now working on our own MultiCluster LB controller that will utilize 2 or more existing LBs created by 2 different Kubernetes clusters that will merge the two URLMaps and create BackendServices with global Backends (instance groups from all clusters).

For us this is the easiest way, and cheapest (no Anthos Hub, as we don't need most of Anthos's features).

This controller would be similar to one that can create one LB from multiple Ingress resources, though simpler.

[tnaduc]

+1 for this feature. We have multiple deployments resided in different repos and it is great to be able to have one ingress resource for each deployment rather than have a long single ingress resource for all deployments. With this feature, we can use google's managed certificate for our LB. Please add this feature. Many thanks.

[sergeyshevch]

Are there any updates on it?

[Berndinox]

Seems like its coming with API-Gateway. A replacement for ingress: https://github.com/kubernetes/ingress-gce/issues/973#issuecomment-792771406

[bryanlarsen]

I didn't get this to work with traefik, since it 404's on '/' for me. It has a '/ping' route that'll 200, but that's on a different port, and I couldn't finagle BackendConfiguration to get the health check to work.

But I did get nginx to work. It has a default backend option that'll return a 200 to '/healthz' so that'll work for the health check.

helm3 install -f nginx-values.yaml ingress-nginx ingress-nginx/ingress-nginx

#nginx-values.yaml

controller:
  service:
    type: NodePort
    annotations:
      cloud.google.com/backend-config: '{"default": "backendconfig"}'
  admissionWebhooks:
    enabled: false
defaultBackend:
  enabled: true

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: backendconfig
spec:
  timeoutSec: 300
  healthCheck:
    requestPath: /healthz

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    networking.gke.io/managed-certificates: foo,bar
spec:
  backend:
    serviceName: ingress-nginx-controller
    servicePort: 80

And then you can create a ManagedCertificate and an Ingress for foo and bar, and they'll share the same LoadBalancer.

But in the end it doesn't really solve the underlying problem. Sure, it lets us create separate Ingress's for foo and bar, but we have to list the certificate names in the annotation on the loadbalancer ingress, so it's not significantly more convenient than putting all the ingresses into a single manifest.

[adrian-gierakowski]

@bryanlarsen you can use the GCLB in L4 proxy mode and terminate tls on the nginx ingress, and instead of using google managed certificates have them automatically created by cert-manager. This would allow you to have services which are fully specified independently of one another and avoid having to modify a shared resource when adding/removing services exposed to the outside world. With nginx ingress you also get more flexibility as you can route based on host name etc, whereas GCLB can only route based on path

[bryanlarsen]

@adrian-gierakowski this is for a situation where cert-manager cannot be used due to https://github.com/jetstack/cert-manager/issues/3717

[adrian-gierakowski]

@bryanlarsen you could try https://github.com/caddyserver/ingress

[slayer]

I want to use IAP for multiple ingresses (multiple namespaces) on the shared LB. It is impossible now, isn't ?

[bowei]

I would take a look at https://cloud.google.com/blog/products/containers-kubernetes/new-gke-gateway-controller-implements-kubernetes-gateway-api for a multi-role use case

[Andrewangeta]

I didn't realize they had a pre release version available?!?!?! ive been waiting for years @bowei thanks for sharing!

[rdxmb]

I cannot believe gce is the only one who does not support this yet...

[rymnc]

any updates here?

[CarpathianUA]

it's strange that at the end of 2021 there is no way to share LB between Ingresses ...

[Andrewangeta]

@CarpathianUA was just about to comment the same thing yesterday. Supposedly the new gateway API would help with that. But it's not available with GKE Autopilot.

[swetharepakula]

This is a limitation of the Ingress API. The Gateway API allows for this feature and we will look to support this feature there.

[adrian-gierakowski]

@swetharepakula how is this completed? Also what is the limitation of the ingress api? Nginx ingress supports multiple ingress objects controlling a single gateway

[mgoodness]

@swetharepakula how is this completed? Also what is the limitation of the ingress api? Nginx ingress supports multiple ingress objects controlling a single gateway

I think the more accurate explanation is that it's a limitation of the GCE implementation of the Ingress API.

[swetharepakula]

The Ingress API does not natively support this. A single Ingress is meant to represent a single load balancer. Nginx is completely different architecture and what works there does not necessarily work for ingress-gce. In ingress-gce, we have many other concerns such as namespaces, permissions etc that cannot be expressed clearly in the Ingress API. This feature is one of the core reasons the Gateway API was created. The Gateway API will allow a single LB to be shared and has the necessary permissions and namespaces pieces built into it. This feature will be supported through the Gateway API, but we will not be adding it to Ingress.

@adrian-gierakowski , I was not aware Github had different close options. I will close this as not planned.

[adrian-gierakowski]

@swetharepakula thank you for the clarification 👍