Kong can't proxy grpc via AWS NLB.

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

I tested by using this setting and work with using RESTApi request.

KIC helm configuration

proxy:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp" # only tcp or ssl
      service.beta.kubernetes.io/aws-load-balancer-alpn-policy: None
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: |-
        preserve_client_ip.enabled=true
      service.beta.kubernetes.io/aws-load-balancer-name: "kong-pvt-eks-infra-pgd"
      service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"

      # ssl setting
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:..."
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
      service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-FS-1-2-2019-08

    http:
      enabled: true

    tls:
      enabled: true

env:
  trusted_ips: 0.0.0.0/0,::0
  proxy_listen: "0.0.0.0:8000 proxy_protocol, 0.0.0.0:8443 proxy_protocol"
  real_ip_header: proxy_protocol

Service

apiVersion: v1
kind: Service
metadata:
  name: cqrsdotnet-svc
  namespace: cqrs-app-ns
  annotations:
    konghq.com/protocol: grpcs
spec:
  selector:
    app: cqrsdotnet
  ports:
    - name: http
      port: 8080
      targetPort: 9000
    - name: grpc
      port: 8081
      targetPort: 9001

Ingress for api

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cqrsdotnet-ingress
  namespace: cqrs-app-ns
spec:
  ingressClassName: kong
  rules:
  - host: web-kong-ex.abc.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cqrsdotnet-svc
            port:
              number: 8080

Ingress for grpc

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cqrsdotnet-grpc-ingress
  namespace: cqrs-app-ns
  annotations:
    konghq.com/protocols: "grpc,grpcs"
spec:
  ingressClassName: kong
  rules:
    - host: web-kong-grpc.abc.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: cqrsdotnet-svc
              port:
                number: 8081

curl -i "https://web-kong-ex.abc.com/api/v1/SendMessage?message=abc"
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Date: Fri, 09 Jun 2023 14:50:14 GMT
Server: Kestrel
api-supported-versions: 1
X-Kong-Upstream-Latency: 3
X-Kong-Proxy-Latency: 1
Via: kong/3.3.0

{"message":"received: abc"}

Expected Behavior

Need to proxy grpc protocal through AWS NLB -> Kong ingress controller

Steps To Reproduce

but it's not work when using grpc through 443 which tls terminating on NLB.

grpcurl "web-kong-grpc.abc.com:443" list
Failed to dial target host "web-kong-grpc.abc.com:443": context deadline exceeded

I have tested call the svc in the k8s cluster, it's work as well.

The log in proxy container shown.

- - [09/Jun/2023:14:59:27 +0000] "PRI * HTTP/2.0" 400 12 "-" "-"

Kong Ingress Controller version

v2.10.0

Kubernetes version

EKS 1.24

Kong / kubernetes-ingress-controller

Kong can't proxy grpc via AWS NLB. #4177