Open 2dev2 opened 5 months ago
cc @nareddyt
gRPC transcoder with ext_authz should work well, I believe I personally ran a similar configuration in the past.
Your config LGTM. I noticed the following:
To help further debug, please provide the following details:
POST /http_reuest_url/xyz/abc?a=3&b=4
with 0 body bytes, which I am suspicious about. gRPC transcoder might be expecting a GET request. Please share the full HTTP request (curl command would be nice) and the HTTP annotation on your gRPC service definition.--log-level debug
and share the application logs. gRPC transcoder outputs many logs which would help debug here.2.a). Trying not clearing the the route cache, clear_route_cache
FYI gRPC transcoder will automatically clear the route cache once it translates the HTTP request unless match_incoming_request_route=true
. Let's debug with the default behavior where match_incoming_request_route=false.
Your route config looks correct for this case.
@nareddyt Thanks for looking out, from long time we got stuck on this. if possible please suggest further.
curl request here: curl --location 'local.api.com/sms_grpc/v1/sms/v1/api/v1/org/sendsms?message=hello&senderId=fg&number=6778&templateId=123' \ --header 'X-API-KEY: abcdefgh'.
which is able to match the envoy grpc service and without envoy ext_auth and it gives response.
grpc service anotation: service SmsApiService { rpc SendSmsExternalGETV1(SendSmsExternalGETV1Request) returns (SendSmsExternalGETV1Response){ option (google.api.http) = { get: "/sms_grpc/v1/sms/v1/api/v1/org/sendsms" }; } } client response in postman:
2 envoy debug log and screen shot where we are able to get the details its making POST API call from inside of ext_auth_filter, envoy_debuglog.txt
Thanks for providing the logs and configuration files, that gave me everything I needed to debug the issue. I'll walk you through it.
Logs show the request is following the request path correctly:
Client --> grpc transcoder --> ext_authz --> upstream cluster new_api_grpc_cluster
In fact, the HTTP 501 is from your own backend. Notice the response has header x-envoy-upstream-service-time
which indicates it tried reaching out to your backend (upstream).
[2024-04-25 22:21:48.961][15][debug][http] [source/common/http/conn_manager_impl.cc:1838] [Tags: "ConnectionId":"0","StreamId":"18191014628947526340"] encoding headers via codec (end_stream=false):
':status', '501'
'content-type', 'application/json'
'x-envoy-upstream-service-time', '1'
'content-length', '167'
'date', 'Thu, 25 Apr 2024 22:21:48 GMT'
'server', 'envoy'
'x-request-id', '3241b8e0-c1eb-46d8-8f1d-fce751fbad69'
Obviously I don't have your backend to verify this. But you can try making the same request Envoy makes directly to your backend (host.docker.internal:8897
, assuming you expose that container port) and observe the response. Request:
[2024-04-25 22:21:48.960][15][debug][router] [source/common/router/router.cc:738] [Tags: "ConnectionId":"0","StreamId":"18191014628947526340"] router decoding headers:
':scheme', 'http'
':method', 'POST'
':path', '/edugo.api.services.sms.v1.SmsApiService/SendSmsExternalGETV1?org_id=5b00abbab9'
':authority', 'local.api.com'
'x-api-key', 'abcdefgh'
'x-org-id', '5b00abbab9'
'x-forwarded-proto', 'http'
'postman-token', 'ec105acf-c79b-4ffd-930d-4a07d9480a79'
'te', 'trailers'
'x-envoy-auth-partial-body', 'false'
'x-envoy-original-method', 'GET'
'x-envoy-original-path', '/sms_grpc/v1/sms/v1/api/v1/org/sendsms?message=hello&senderId=fg&number=6778&templateId=123'
'x-request-id', '3241b8e0-c1eb-46d8-8f1d-fce751fbad69'
'content-type', 'application/grpc'
'x-envoy-expected-rq-timeout-ms', '60000'
I do have a guess for why your backend is sending back HTTP 501. Take a look at the HTTP path that envoy sends to your backend:
':path', '/edugo.api.services.sms.v1.SmsApiService/SendSmsExternalGETV1?org_id=5b00abbab9'
IIUC the query parameter ?org_id=5b00abbab9
in the path is not compliant with the gRPC over HTTP2 specification. All metadata you wish to forward to gRPC server should be as gRPC metadata, i.e. lowercase HTTP headers like x-api-key
and x-org-id
.
Why is Envoy sending this query parameter to the backend? It is because of the ext_authz response from your ext_authz server:
status {
}
ok_response {
headers {
header {
key: "X-ORG-ID"
value: "5b00abbab9"
}
}
headers {
header {
key: ":scheme"
value: "http"
}
keep_empty_value: true
}
headers {
header {
key: ":method"
value: "POST"
}
keep_empty_value: true
}
headers {
header {
key: ":path"
value: "/edugo.api.services.sms.v1.SmsApiService/SendSmsExternalGETV1"
}
keep_empty_value: true
}
headers {
header {
key: "x-forwarded-proto"
value: "http"
}
keep_empty_value: true
}
headers {
header {
key: "postman-token"
value: "ec105acf-c79b-4ffd-930d-4a07d9480a79"
}
keep_empty_value: true
}
headers {
header {
key: "te"
value: "trailers"
}
keep_empty_value: true
}
headers {
header {
key: "x-envoy-auth-partial-body"
value: "false"
}
keep_empty_value: true
}
headers {
header {
key: ":authority"
value: "local.api.com"
}
keep_empty_value: true
}
headers {
header {
key: "x-envoy-original-method"
value: "GET"
}
keep_empty_value: true
}
headers {
header {
key: "x-envoy-original-path"
value: "/sms_grpc/v1/sms/v1/api/v1/org/sendsms?message=hello&senderId=fg&number=6778&templateId=123"
}
keep_empty_value: true
}
headers {
header {
key: "x-request-id"
value: "3241b8e0-c1eb-46d8-8f1d-fce751fbad69"
}
keep_empty_value: true
}
headers {
header {
key: "content-type"
value: "application/grpc"
}
keep_empty_value: true
}
headers_to_remove: "apikey"
query_parameters_to_set {
key: "org_id"
value: "5b00abbab9"
}
}
dynamic_metadata {
fields {
key: "x-edu-org-id"
value {
string_value: "5b00abbab9"
}
}
}
The following part should be removed, as it is not compliant with gRPC over HTTP2:
query_parameters_to_set {
key: "org_id"
value: "5b00abbab9"
}
In general, i don't recommend you rewrite the entire request via the ext_authz response. Just add on the headers you need, like x-org-id
, and omit the rest of the fields.
TLDR - there is no issue with gRPC JSON transcoder or ext_authz filter integration. It is due to malformed ext_authz response causing the user's backend to respond with 501.
i have removed query_parameters_to_set and make path as grpc compliant and its working fine. Thanks a lot, for giving the details insight and suggestion.
We have used query params to use this feature of grpc-gateway: https://github.com/googleapis/googleapis/blob/master/google/api/http.proto#L223 Means if we pass some field as query params and they are mentioned/member of in body proto struct, it will be filled with passed query data.
Title: GRPC-JSON transcoder is ignoring ext_authz filter, can we have documentation/example updates in which both filter are working together
Description:
Expectation: in envoy ext_authz control plane authenticate and pass the CheckResponse to upstream grpc-service. how to apply missing envoy-config or how we can resolve the 501 error code ( as auth is passing request to grpc but not in correct url )
Based on suggestion Tried Solution/steps :
Listener envoy config:
envoy 1.30 listener config: web_lb_grpc2 .txt