Use XDS server with proxyless GRPC XDS client

[arthurh4]

Hello, We are currently using envoy gateway for some of our small services and want to expand envoy gateway to high throughput, low latency GRPC services. We tested GCP traffic director in the past with proxyless but we want to explore options to leverage envoy gateway in order to only have one tool to support and have multi cloud support (+ OSS is always a plus to review code and bugs).

I've been playing over the weekend with simple golang GRPC client/server setup with an envoy-gateway GRPCRoute setup but got kinda stuck on getting and parsing ADS response from the XDS server.

I get to the point of sending the follow ADS request:

2024/07/07 01:42:00 INFO: [xds] [xds-client 0xc00035ac80] [envoy-gateway.envoy-gateway-system.svc.cluster.local:18000] ADS request sent: {
  "node":  {
    "id":  "envoy-envoy-gateway-system-envoy-acme-network-grpc-839693ea-65cc58fd86-j8qnc",
    "locality":  {
      "zone":  "us-central1-a"
    },
    "userAgentName":  "gRPC Go",
    "userAgentVersion":  "1.63.0",
    "clientFeatures":  [
      "envoy.lb.does_not_support_overprovisioning",
      "xds.config.resource-in-sotw"
    ]
  },
  "resourceNames":  [
    "envoy-gateway-system/envoy-acme-network-grpc/grpc"
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.listener.v3.Listener"
}

Which kinda works but I get the following error:

2024/07/07 01:42:00 could not greet2: rpc error: code = Unavailable desc = network filters {[]} had invalid server side HTTP Filters {[name:"envoy.filters.http.grpc_web"  typed_config:{type_url:"type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb"} name:"envoy.filters.http.grpc_stats"  typed_config:{type_url:"type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig"  value:"\x08\x01\x1a\x02\x08\x01"} name:"envoy.filters.http.router"  typed_config:{[type.googleapis.com/envoy.extensions.filters.http.router.v3.Router]:{suppress_envoy_headers:true}}]}: no filter implementation found for "type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb"

I've read the https://github.com/grpc/proposal/blob/master/A39-xds-http-filters.md doc but i'm not really sure if there is a a workaround, it seems like there is no option to completely ignore non optionnal with no implementation filters.

I'm fairly new to XDS and envoy-gateway so trying to see if anyone tried to acheive this setup before and some leads on where to go. I there any other request/endpoint I could query to get the GRPC endpoints behind the GRPCRoute in order to enable proxyless with envoy-gateway ?

I'm simply trying to get a POC to review that it could in theory works and perhaps build some extension in future ?

ADS response I got

2024/07/07 01:42:00 INFO: [xds] [xds-client 0xc00035ac80] [envoy-gateway.envoy-gateway-system.svc.cluster.local:18000] ADS response received: version_info:"1"  resources:{[type.googleapis.com/envoy.config.listener.v3.Listener]:{name:"envoy-gateway-system/envoy-acme-network-grpc/grpc"  address:{socket_address:{address:"0.0.0.0"  port_value:10080}}  default_filter_chain:{filters:{name:"envoy.filters.network.http_connection_manager"  typed_config:{[type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager]:{stat_prefix:"http"  rds:{config_source:{ads:{}  resource_api_version:V3}  route_config_name:"envoy-gateway-system/envoy-acme-network-grpc/grpc"}  http_filters:{name:"envoy.filters.http.grpc_web"  typed_config:{type_url:"type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb"}}  http_filters:{name:"envoy.filters.http.grpc_stats"  typed_config:{type_url:"type.googleapis.com/envoy.extensions.filters.http.grpc_stats.v3.FilterConfig"  value:"\x08\x01\x1a\x02\x08\x01"}}  http_filters:{name:"envoy.filters.http.router"  typed_config:{[type.googleapis.com/envoy.extensions.filters.http.router.v3.Router]:{suppress_envoy_headers:true}}}  common_http_protocol_options:{headers_with_underscores_action:REJECT_REQUEST}  http2_protocol_options:{max_concurrent_streams:{value:100}  initial_stream_window_size:{value:65536}  initial_connection_window_size:{value:1048576}}  server_header_transformation:PASS_THROUGH  access_log:{name:"envoy.access_loggers.file"  typed_config:{type_url:"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"  value:"\n\x0b/dev/stdout*\x87\x08*\x84\x08\x1a\x81\x08{\"start_time\":\"%START_TIME%\",\"method\":\"%REQ(:METHOD)%\",\"x-envoy-origin-path\":\"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\"protocol\":\"%PROTOCOL%\",\"response_code\":\"%RESPONSE_CODE%\",\"response_flags\":\"%RESPONSE_FLAGS%\",\"response_code_details\":\"%RESPONSE_CODE_DETAILS%\",\"connection_termination_details\":\"%CONNECTION_TERMINATION_DETAILS%\",\"upstream_transport_failure_reason\":\"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\"bytes_received\":\"%BYTES_RECEIVED%\",\"bytes_sent\":\"%BYTES_SENT%\",\"duration\":\"%DURATION%\",\"x-envoy-upstream-service-time\":\"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\"x-forwarded-for\":\"%REQ(X-FORWARDED-FOR)%\",\"user-agent\":\"%REQ(USER-AGENT)%\",\"x-request-id\":\"%REQ(X-REQUEST-ID)%\",\":authority\":\"%REQ(:AUTHORITY)%\",\"upstream_host\":\"%UPSTREAM_HOST%\",\"upstream_cluster\":\"%UPSTREAM_CLUSTER%\",\"upstream_local_address\":\"%UPSTREAM_LOCAL_ADDRESS%\",\"downstream_local_address\":\"%DOWNSTREAM_LOCAL_ADDRESS%\",\"downstream_remote_address\":\"%DOWNSTREAM_REMOTE_ADDRESS%\",\"requested_server_name\":\"%REQUESTED_SERVER_NAME%\",\"route_name\":\"%ROUTE_NAME%\"}\n"}}  use_remote_address:{value:true}  normalize_path:{value:true}  merge_slashes:true  path_with_escaped_slashes_action:UNESCAPE_AND_REDIRECT}}}  name:"envoy-gateway-system/envoy-acme-network-grpc/grpc"}  per_connection_buffer_limit_bytes:{value:32768}  drain_type:MODIFY_ONLY  access_log:{name:"envoy.access_loggers.file"  filter:{response_flag_filter:{flags:"NR"}}  typed_config:{type_url:"type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"  value:"\n\x0b/dev/stdout*\x87\x08*\x84\x08\x1a\x81\x08{\"start_time\":\"%START_TIME%\",\"method\":\"%REQ(:METHOD)%\",\"x-envoy-origin-path\":\"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\"protocol\":\"%PROTOCOL%\",\"response_code\":\"%RESPONSE_CODE%\",\"response_flags\":\"%RESPONSE_FLAGS%\",\"response_code_details\":\"%RESPONSE_CODE_DETAILS%\",\"connection_termination_details\":\"%CONNECTION_TERMINATION_DETAILS%\",\"upstream_transport_failure_reason\":\"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\"bytes_received\":\"%BYTES_RECEIVED%\",\"bytes_sent\":\"%BYTES_SENT%\",\"duration\":\"%DURATION%\",\"x-envoy-upstream-service-time\":\"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\"x-forwarded-for\":\"%REQ(X-FORWARDED-FOR)%\",\"user-agent\":\"%REQ(USER-AGENT)%\",\"x-request-id\":\"%REQ(X-REQUEST-ID)%\",\":authority\":\"%REQ(:AUTHORITY)%\",\"upstream_host\":\"%UPSTREAM_HOST%\",\"upstream_cluster\":\"%UPSTREAM_CLUSTER%\",\"upstream_local_address\":\"%UPSTREAM_LOCAL_ADDRESS%\",\"downstream_local_address\":\"%DOWNSTREAM_LOCAL_ADDRESS%\",\"downstream_remote_address\":\"%DOWNSTREAM_REMOTE_ADDRESS%\",\"requested_server_name\":\"%REQUESTED_SERVER_NAME%\",\"route_name\":\"%ROUTE_NAME%\"}\n"}}}}  type_url:"type.googleapis.com/envoy.config.listener.v3.Listener"  nonce:"1"

K8s resource:

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"GRPCRoute","metadata":{"annotations":{},"name":"grpc-server","namespace":"grpc-test"},"spec":{"hostnames":["test-grpc.acme.io"],"parentRefs":[{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"envoy-acme-network-grpc","namespace":"envoy-gateway-system"}],"rules":[{"backendRefs":[{"name":"grpc-server","port":80}],"matches":[{"method":{"method":"SayHello","service":"helloworld.Greeter"}}]}]}}
  creationTimestamp: "2024-07-05T19:25:34Z"
  generation: 3
  name: grpc-server
  namespace: grpc-test
  resourceVersion: "484271068"
  uid: c76167ac-0265-4cf4-a5e8-eecd57e2ee50
spec:
  hostnames:
  - test-grpc.acme.io
  - test2-grpc.acme.io
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: envoy-acme-network-grpc
    namespace: envoy-gateway-system
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: grpc-server
      port: 80
      weight: 1
    matches:
    - method:
        method: SayHello
        service: helloworld.Greeter
        type: Exact
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-07-07T01:29:42Z"
      message: Route is accepted
      observedGeneration: 3
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-07-07T01:29:42Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 3
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: envoy-acme-network-grpc
      namespace: envoy-gateway-system

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

sorry for the late reply @arthurh4, you could try to use EnvoyPatchPolicy to edit the xds config and rm grpc_web from the config https://gateway.envoyproxy.io/docs/tasks/extensibility/envoy-patch-policy/

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.