search

kubernetes-sigs / gateway-api

Repository for the next iteration of composite service (e.g. Ingress) and load balancing APIs.
https://gateway-api.sigs.k8s.io
Apache License 2.0
1.75k stars 454 forks source link

AWS load balancer access log annotations do not have effect #3232

Closed hongbo-miao closed 1 month ago

hongbo-miao commented 1 month ago

What happened

I am using v1.1.0 experimental Gateway API CRDs:

https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_gatewayclasses.yaml
https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_gateways.yaml
https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_httproutes.yaml
https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_referencegrants.yaml
https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml
https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml

I am trying to enable load balancer access log:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: hm-gateway-api-gateway
  namespace: production-hm-gateway-api
  labels:
    app.kubernetes.io/name: hm-gateway-api-gateway
    app.kubernetes.io/part-of: production-hm-gateway-api
spec:
  gatewayClassName: cilium
  infrastructure:
    annotations:
      # https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations
      # https://cloud-provider-aws.sigs.k8s.io/service_controller
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xx,subnet-xx,subnet-xx,subnet-xx
      # New load balancer access log annotation
      service.beta.kubernetes.io/aws-load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=hm-kubernetes-load-balancer-log-archer,access_logs.s3.prefix=hm-gateway-api-gateway
      # Legacy load balancer access log annotation
      # service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: "5"
      # service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
      # service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: hm-kubernetes-load-balancer-log-archer
      # service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: hm-gateway-api-gateway
  listeners:
    - name: http
      protocol: HTTP
      hostname: "*.internal.mydomain.com"
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      protocol: HTTPS
      hostname: "*.internal.mydomain.com"
      port: 443
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: staging-hm-gateway-api-certificate
            namespace: production-hm-gateway-api

I can confirm these annotations work well

      service.beta.kubernetes.io/aws-load-balancer-type: nlb
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xx,subnet-xx,subnet-xx,subnet-xx

However, both new load balancer access log annotation

      # New load balancer access log annotation
      service.beta.kubernetes.io/aws-load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=hm-kubernetes-load-balancer-log-archer,access_logs.s3.prefix=hm-gateway-api-gateway

and legacy load balancer access log annotation

      # Legacy load balancer access log annotation
      service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: "5"
      service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: hm-kubernetes-load-balancer-log-archer
      service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: hm-gateway-api-gateway

do not work. 🥲

image

What you expected to happen

I was hoping the new load balancer access log annotation work well.

Anything else we need to know?

https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.8/guide/service/annotations/ mentions legacy and new annonations:

image

https://cloud-provider-aws.sigs.k8s.io/service_controller only has legacy annotations. I am not sure which doc is source of truth, but anyway I tried both new and legacy annotations, and none of them work.

robscott commented 1 month ago

Hey @hongbo-miao, this repo is only for issues with the API itself, not implementations of the API. In this case, I'd recommend filing an issue with Cilium.

hongbo-miao commented 1 month ago

Thank you @robscott for the info! I opened at https://github.com/cilium/cilium/issues/34116 and will track there ☺️