CFP: allow using NodePort type service for Gateway API's Gateway

[Cajga]

Cilium Feature Proposal

Is your proposed feature related to a problem? We are on bare metal and have external, HW load balancer clusters that are not integrated into our Kubernetes cluster (in short: LoadBalancer type services are not useful for us). This prevents us to use the Gateway API implementation of Cilium (and forces us to keep using Istio for ingress/service mesh).

Describe the feature you'd like We would like to be able to use NodePort service type for Gateways.

(Optional) Describe your proposed solution For example, Istio implemented this with an annotation on the Gateway resource called networking.istio.io/service-type that would also allow the use of ClusterIP.

While we know that this is not how one should extend Gateway API resources, but sounds like an easy/simple temporary solution until a solution is rolled out to the Gateway API itself.

[Cajga]

This ticket may be relevant: https://github.com/cilium/cilium/issues/21923

[teknologista]

+1

[ml-twi]

+1

[kahirokunn]

+1

[arouene]

I'm surprised that the GatewayAPI implementation doesn't use the same settings as the ingressController :

ingressController:
  enabled: true
  loadbalancerMode: Shared
  service:
    type: NodePort
    insecureNodePort: 30080
    secureNodePort: 30443

Is there any bypass/settings to use a node port for the GatewayAPI ?

[bdalpe]

You should be able to do this in 1.16 with the functionality added in https://github.com/cilium/cilium/pull/30840. The new functionality allows for listening on the host network and setting a configurable port.

[networkhermit]

You should be able to do this in 1.16 with the functionality added in #30840. The new functionality allows for listening on the host network and setting a configurable port.

Is there any major tradeoff between NodePort and HostNetwork listener? e.g. security, network policy, host requirement, direct routing.

I think they are not strictly equivalent.

[kahirokunn]

@bdalpe https://github.com/cilium/cilium/issues/27273#issuecomment-2027330604

helm upgrade --install cilium cilium/cilium --version 1.16.0-pre.1 -n kube-system --values ./cilium-values.yaml --wait

kubeProxyReplacement: true
hostServices:
  enabled: false
externalIPs:
  enabled: true
nodePort:
  enabled: true
hostPort:
  enabled: true
image:
  pullPolicy: IfNotPresent
ipam:
  mode: kubernetes
operator:
  replicas: 1
hubble:
  dropEventEmitter:
    enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
    service:
      type: NodePort
      nodePort: 30002
hostFirewall:
  enabled: true
gatewayAPI:
  enabled: true
  hostNetwork:
    enabled: true
    nodes:
      matchLabels:
        kubernetes.io/os: linux

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1","kind":"Gateway","metadata":{"annotations":{},"name":"my-gateway","namespace":"default"},"spec":{"gatewayClassName":"cilium","listeners":[{"allowedRoutes":{"namespaces":{"from":"Same"}},"name":"web-gw","port":80,"protocol":"HTTP"}]}}
  creationTimestamp: "2024-04-24T05:56:54Z"
  generation: 1
  name: my-gateway
  namespace: default
  resourceVersion: "8979"
  uid: e89cf7a9-f05f-404f-b094-d1b27dc1672b
spec:
  gatewayClassName: cilium
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: web-gw
    port: 80
    protocol: HTTP
status:
  conditions:
  - lastTransitionTime: "2024-04-24T05:56:54Z"
    message: Gateway successfully scheduled
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-04-24T05:56:54Z"
    message: Gateway successfully reconciled
    observedGeneration: 1
    reason: Programmed
    status: "True"
    type: Programmed
  listeners:
  - attachedRoutes: 0
    conditions:
    - lastTransitionTime: "2024-04-24T06:20:22Z"
      message: Listener Programmed
      observedGeneration: 1
      reason: Programmed
      status: "True"
      type: Programmed
    - lastTransitionTime: "2024-04-24T06:20:22Z"
      message: Listener Accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-04-24T06:20:22Z"
      message: Resolved Refs
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: web-gw
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute

I tried to configure the GatewayAPI, but the Gateway is Listen on any Port on the host and from within the cluster I could not tell which DNS I could connect to. The .status.address is not filled in either. Is there a complete example where the GatewayAPI is exposed on a nodePort and works? Thx.

[kahirokunn]

I read and understand this. Listeners now points to the hostport, and if I specify a port of 1023 or higher, it works fine. https://github.com/cilium/cilium/pull/31839/files

[kahirokunn]

@bdalpe https://github.com/cilium/cilium/issues/27273#issuecomment-2027330604

A mode has recently been added where the listener works with the host machine, but it still does not work with ALB Ingress. This is because the NodePort service is not created. A ClusterIP service could be used, but without labelSelector, Endpoint, etc. in that case, there is no connection. 😢

[aleksandarilic96]

A bit confused here so we can use it the same way as we use Ingress with NodePort? If so do you have an example?

[kahirokunn]

For ALB Ingress, pods matching the ClusterIP selector can be directly registered as ALB targets. In the case of NodePort, the traffic is accessed by a random EC2 instance and then forwarded to the Pod.

[kahirokunn]

This is a sample of the use of the NodePort.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/healthcheck-port: "30267"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/subnets: subnet-xxx
    alb.ingress.kubernetes.io/target-type: instance
  name: nodeport-sample
  namespace: default
spec:
  defaultBackend:
    service:
      name: nodeport-sample
      port:
        name: http
  ingressClassName: alb

[kahirokunn]

Certainly, you could use ClusterIP, but unfortunately, accessing a Cilium ClusterIP will not establish a connection. This is because the ClusterIP is not an ENI. 😕

For AWS users, it's disappointing that the current implementation of Cilium's Gateway API for ClusterIP, as well as implementations that listen on the host, are not compatible with ALB Ingress. If Cilium's Gateway API were to support the creation of ALBs or TargetGroups, it might mitigate these compatibility issues even if it isn't directly compatible with ALB Ingress.

[aleksandarilic96]

Ok from the PR you mentioned above https://github.com/cilium/cilium/pull/31839/files

It would seem that you would be able to expose it on a host network like with ingress.

So when you define a Gateway that port should be exposed on the host network?

If we would use this as an example

apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: my-gateway spec: gatewayClassName: cilium listeners:

• protocol: HTTP port: 1055 (since it would be needed to be higher than 1023) name: web-gw

This would expose port 1055 on the host network, right or I'm totally misunderstanding it?

[kahirokunn]

Port 1055 will be exposed on the host network. However, the configuration of ALB Ingress -> Gateway is not available. This is because the exposure of the node's port at 1055 is not represented as a NodePortService.

[kahirokunn]

In addition to the unavailability of the ALB Ingress -> Cilium Gateway configuration, the configurations for Istio -> Cilium Gateway and Contour -> Cilium Gateway are also not available. This is because, while they have a service selector, they lack a node selector.

Currently, Cilium's Gateway finds it challenging to integrate with other CNCF projects. Of course, it is permissible not to have integration, but in that case, wouldn't it be difficult to utilize in a public cloud without taking additional steps like creating ALBs or target groups?

[aleksandarilic96]

Where able to get it working on the host network?

I was testing it and its always stuck in Pending Message: Waiting for controller Reason: Pending Status: Unknown Type: Accepted

This should be the correct configuration.

kubeProxyReplacement: true k8sClientRateLimit: qps: 50 burst: 200 operator: replicas: 1 rollOutPods: true hostServices: enabled: false externalIPs: enabled: true nodePort: enabled: true hostPort: enabled: true hostFirewall: enabled: true gatewayAPI: enabled: true hostNetwork: enabled: true

apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: tls-gateway spec: gatewayClassName: cilium listeners:

• name: https-1 protocol: HTTPS port: 1254 hostname: "works.theoshomelab.xyz" tls: certificateRefs:

  • kind: Secret name: test-cert

[kahirokunn]

@aleksandarilic96

kubeProxyReplacement: true
hostServices:
  enabled: false
externalIPs:
  enabled: true
nodePort:
  enabled: true
hostPort:
  enabled: true
  pullPolicy: IfNotPresent
ipam:
  mode: kubernetes
operator:
  replicas: 1
  dashboards:
    enabled: true
prometheus:
  enabled: true
  serviceMonitor:
    enabled: true
hubble:
  dropEventEmitter:
    enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
    frontend:
    backend:
    service:
      type: NodePort
      nodePort: 30002
hostFirewall:
  enabled: true
envoy:
  enabled: true
gatewayAPI:
  enabled: true
  hostNetwork:
    enabled: true
    sharedHTTPPort: 30080
    sharedTLSPassthroughPort: 30443
    nodes:
      matchLabels:
        kubernetes.io/os: linux

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: example
  namespace: example-local-gateway
spec:
  gatewayClassName: cilium
  listeners:
  - protocol: HTTP
    port: 30080
    name: http
    allowedRoutes:
      namespaces:
        from: All

The 30080 port is now published on host. 👍

[aleksandarilic96]

Thanks for the example I had an issue with cilium and needed to restart the operator to pull the changes.

[joseguerrero]

Hi @kahirokunn, in your example here, where are the gatewayAPI.hostNetwork.sharedHTTPPort and gatewayAPI.hostNetwork.sharedTLSPassthroughPort settings coming from? I don't see any values like that even in the latest version of the values files here.

[Silvest89]

Hi @kahirokunn, in your example here, where are the gatewayAPI.hostNetwork.sharedHTTPPort and gatewayAPI.hostNetwork.sharedTLSPassthroughPort settings coming from? I don't see any values like that even in the latest version of the values files here.

They are part of the ingress controller and not of the GatewayAPI. You can omit them for GatewayAPI https://artifacthub.io/packages/helm/cilium/cilium/1.16.0-pre.3?modal=values