Issue Apache Apisix gRPC transcode with AWS Networkloadbalancer

[bakuppus]

upstream and route with gRPC transcode and scheme grpc in APISIX:

Upstream:

curl http://127.0.0.1:9180/apisix/admin/upstreams/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
  "id": 1,
  "nodes": [
    {
      "host": "grpc-pyserver.default.svc.cluster.local",
      "port": 50053,
      "weight": 1
    }
  ],
  "scheme": "grpc",
  "type": "roundrobin",
  "retries": 3,
  "timeout": {
    "connect": 6,
    "send": 6,
    "read": 6
  }
}
'

Route:

curl http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
  "id": 1,
  "uri": "/grpc",
  "host": "grpc-pyserver.poc.kubelancer.net",
  "name": "grpc-route",
  "plugins": {
    "grpc-transcode": {
      "proto_id": "example.proto",
      "method": "SayHello",
      "service": "ExampleService"
    }
  },
  "upstream_id": 1,
  "status": 1
}
'

I created an upstream with the id 1 that points to a gRPC server at grpc-server:50053. We then create a route with the id 1 that uses the grpc-transcode plugin to transcode the gRPC request to the SayHello method of the ExampleService service, as defined in the example.proto file. The route is configured to use the upstream with the id 1.

GRPC application service is grpc-server exposing hostname "host": "grpc-pyserver.poc.kubelancer.net",

created the example.proto file and loaded it into APISIX using the proto plugin.

Helm Deployed on EKS with Network LoadBalancer

helm install apisix apisix/apisix \
  --set allow.ipList="{0.0.0.0/0}" \
  --set service.type=LoadBalancer \
  --set ingress-controller.enabled=true \
  --set ingress-controller.config.kubernetes.enableGatewayAPI=true\
  --set dashboard.enabled=true \
  --create-namespace \
  --namespace ingress-apisix \
  --set ingress-controller.config.apisix.serviceNamespace=ingress-apisix \
  --set ingress-controller.config.apisix.adminAPIVersion=v3 \
  --set apisix.ssl.enabled=true \
  --set service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"=nlb \
  --set service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-internal"=false 

Internally within cluster getting the expected result

  bala@kubelancer app % grpcurl -plaintext -d '{"name": "World"}' localhost:50053 example.ExampleService/SayHello

{
  "message": "Hello, World!"
}
bala@kubelancer app % 

Issue: While externally mapping the network loadbalancer to "host": "grpc-pyserver.poc.kubelancer.net", which not works.

bala@kubelancer ~ % grpcurl -plaintext -d '{"name": "World"}' grpc-pyserver.poc.kubelancer.net:443 example.ExampleService/SayHello
Failed to dial target host "grpc-pyserver.poc.kubelancer.net:443": context deadline exceeded

But internal works.

bala@kubelancer ~ % grpcurl -plaintext -d '{"name": "World"}' localhost:50053 example.ExampleService/SayHello
{
  "message": "Hello, World!"
}

bala@kubelancer ~ % grpcurl -plaintext localhost:50053 list
example.ExampleService
grpc.reflection.v1alpha.ServerReflection
bala@kubelancer ~ % 

Let me know if anyone using Apache Apisix with AWS Network loadBalancer for GRPC application. Guide me what's wrong.

[thapaaaa]

I am having the same issue. It works internally within the cluster, but when trying to map it to the network load balancer to host externally, it didn't work. Apache apisix is not a good option if you try to use GRPC externally. Try exploring other alternatives , Good luck !

[palungtar]

I did this got into something similar issue and was unable to determine why it was failing externally. Unsure if it is the mapping issue or any other issue. But as additional checks, you might want to review the APISIX configuration for gRPC and TLS settings. Also, validate the NLB configuration (the target group settings and health checks). Are your security groups rules in place correctly to allow traffic? An NLB with gRPC, you usually need to use TLS passthrough and handle the TLS termination at the APISIX level.

I was suggested alternative solution to use ALB as it natively supports gRPC traffic and also can handle TLS termination, making it simpler to expose gRPC externally. Please update back if you can make the above work or figure out what's wrong. I am curious. I am unsure if it will work. I spent a lot of time on it.