Closed Sanskarzz closed 3 months ago
Thanks for reporting this @Sanskarzz. If you'd like to contribute a fix that would be great! Thanks.
Hey @ashutosh-narkar Yes, I would like to contribute. However, I have tried debugging and checking the Envoy logs to identify the problem, but I couldn't find a solution. It would be great if you could guide me or provide me with the steps to follow to resolve this issue.
Here are the logs of envoy when i made curl request
[2024-05-25 06:47:51.313][1][debug][dns] [source/common/network/dns_impl.cc:270] dns resolution for 127.0.0.1 started
[2024-05-25 06:47:51.313][1][debug][dns] [source/common/network/dns_impl.cc:188] dns resolution for 127.0.0.1 completed with status 0
[2024-05-25 06:47:51.313][1][debug][upstream] [source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address 127.0.0.1:8080
[2024-05-25 06:47:51.313][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:177] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
[2024-05-25 06:47:51.377][1][debug][main] [source/server/server.cc:229] flushing stats
[2024-05-25 06:47:56.309][1][debug][dns] [source/common/network/dns_impl.cc:270] dns resolution for 127.0.0.1 started
[2024-05-25 06:47:56.309][1][debug][dns] [source/common/network/dns_impl.cc:188] dns resolution for 127.0.0.1 completed with status 0
[2024-05-25 06:47:56.309][1][debug][upstream] [source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address 127.0.0.1:8080
[2024-05-25 06:47:56.309][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:177] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
[2024-05-25 06:47:56.378][1][debug][main] [source/server/server.cc:229] flushing stats
[2024-05-25 06:47:56.723][22][debug][conn_handler] [source/server/active_tcp_listener.cc:140] [C2] new connection from 10.244.0.1:46174
[2024-05-25 06:47:56.723][22][debug][http] [source/common/http/conn_manager_impl.cc:274] [C2] new stream
[2024-05-25 06:47:56.723][22][debug][http] [source/common/http/conn_manager_impl.cc:867] [C2][S1304557893114863817] request headers complete (end_stream=true):
':authority', '192.168.49.2:32286'
':path', '/people'
':method', 'GET'
'user-agent', 'curl/7.81.0'
'accept', '*/*'
'authorization', 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoiZ3Vlc3QiLCJzdWIiOiJZV3hwWTJVPSIsIm5iZiI6MTUxNDg1MTEzOSwiZXhwIjoxNjQxMDgxNTM5fQ.K5DnnbbIOspRbpCr2IKXE9cPVatGOCBrBQobQmBmaeU'
[2024-05-25 06:47:56.723][22][debug][http] [source/common/http/filter_manager.cc:835] [C2][S1304557893114863817] request end stream
[2024-05-25 06:47:56.726][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:363] Finish with grpc-status code 0
[2024-05-25 06:47:56.726][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:215] notifyRemoteClose 0
[2024-05-25 06:47:56.726][22][debug][http] [source/common/http/filter_manager.cc:947] [C2][S1304557893114863817] Sending local reply with details ext_authz_denied
[2024-05-25 06:47:56.726][22][debug][http] [source/common/http/conn_manager_impl.cc:1467] [C2][S1304557893114863817] encoding headers via codec (end_stream=true):
':status', '403'
'date', 'Sat, 25 May 2024 06:47:56 GMT'
'server', 'envoy'
[2024-05-25 06:47:56.726][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:404] Stream cleanup with 0 in-flight tags
[2024-05-25 06:47:56.726][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:393] Deferred delete
[2024-05-25 06:47:56.726][22][debug][grpc] [source/common/grpc/google_async_client_impl.cc:165] GoogleAsyncStreamImpl destruct
[2024-05-25 06:47:56.726][22][debug][connection] [source/common/network/connection_impl.cc:640] [C2] remote close
[2024-05-25 06:47:56.726][22][debug][connection] [source/common/network/connection_impl.cc:249] [C2] closing socket: 0
[2024-05-25 06:47:56.726][22][debug][conn_handler] [source/server/active_stream_listener_base.cc:120] [C2] adding to cleanup list
[2024-05-25 06:48:01.314][1][debug][dns] [source/common/network/dns_impl.cc:270] dns resolution for 127.0.0.1 started
[2024-05-25 06:48:01.314][1][debug][dns] [source/common/network/dns_impl.cc:188] dns resolution for 127.0.0.1 completed with status 0
[2024-05-25 06:48:01.314][1][debug][upstream] [source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address 127.0.0.1:8080
[2024-05-25 06:48:01.314][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:177] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
[2024-05-25 06:48:01.383][1][debug][main] [source/server/server.cc:229] flushing stats
[2024-05-25 06:48:06.310][1][debug][dns] [source/common/network/dns_impl.cc:270] dns resolution for 127.0.0.1 started
[2024-05-25 06:48:06.310][1][debug][dns] [source/common/network/dns_impl.cc:188] dns resolution for 127.0.0.1 completed with status 0
[2024-05-25 06:48:06.310][1][debug][upstream] [source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address 127.0.0.1:8080
[2024-05-25 06:48:06.310][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:177] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
[2024-05-25 06:48:06.382][1][debug][main] [source/server/server.cc:229] flushing stats
Hey @ashutosh-narkar Are you sure grpc_service.google_grpc.target_uri
field in envoy config supports unix domain sockets (uds) , just asking because i have not found any documentation on that .
- name: envoy.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
transport_api_version: V3
with_request_body:
max_request_bytes: 8192
allow_partial_message: true
pack_as_bytes: true
failure_mode_allow: false
grpc_service:
google_grpc:
stat_prefix: ext_authz
target_uri: unix:///run/opa/sockets/auth.sock
timeout: 0.5s
It was working before so I would imagine the target_uri
setting should be fine. I would check if there's some new config setting in Envoy that maybe causing this or if something changed in the Envoy config. Also I would check if OPA's getting the expected request.
Thank you, @ashutosh-narkar , for your response. If there is an opportunity for me to contribute, please let me know where the issue lies, and I would be happy to assist. Actually i'm LFX mentee currently working on the kyverno-envoy-plugin, I have learned a great deal from your work on the OPA-envoy-plugin. I appreciate your contributions to open source; they have been incredibly helpful and inspiring. Thanks for doing open source.
If you control the envoy CLI args, try adding --component-log-level ext_authz:trace
and see what it logs then.
@srenatus @ashutosh-narkar
Here the request/log info after adding --component-log-level ext_authz:trace
in args
sanskar@sanskar-HP-Laptop-15s-du1xxx:~$ kubectl logs "$(kubectl get pod -l app=example-app -o jsonpath={.items..metadata.name})" -c envoy -f
[2024-05-29 12:26:19.095][23][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:42] Sending CheckRequest: attributes {
source {
address {
socket_address {
address: "10.244.0.1"
port_value: 31005
}
}
}
destination {
address {
socket_address {
address: "10.244.0.5"
port_value: 8000
}
}
}
request {
time {
seconds: 1716985579
nanos: 93830000
}
http {
id: "6536821954363734"
method: "GET"
headers {
key: ":authority"
value: "192.168.49.2:31814"
}
headers {
key: ":method"
value: "GET"
}
headers {
key: ":path"
value: "/people"
}
headers {
key: ":scheme"
value: "http"
}
headers {
key: "accept"
value: "*/*"
}
headers {
key: "authorization"
value: "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyb2xlIjoiZ3Vlc3QiLCJzdWIiOiJZV3hwWTJVPSIsIm5iZiI6MTUxNDg1MTEzOSwiZXhwIjoxNjQxMDgxNTM5fQ.K5DnnbbIOspRbpCr2IKXE9cPVatGOCBrBQobQmBmaeU"
}
headers {
key: "user-agent"
value: "curl/7.81.0"
}
headers {
key: "x-forwarded-proto"
value: "http"
}
headers {
key: "x-request-id"
value: "65f25374-f403-4fbf-8840-008f5e490844"
}
path: "/people"
host: "192.168.49.2:31814"
scheme: "http"
protocol: "HTTP/1.1"
}
}
metadata_context {
}
}
[2024-05-29 12:26:19.102][23][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:48] Received CheckResponse: status {
code: 7
}
dynamic_metadata {
fields {
key: "decision_id"
value {
string_value: "ac1824e9-6d1f-43f2-931f-69aa8f106e40"
}
}
}
curl request
sanskar@sanskar-HP-Laptop-15s-du1xxx:~/opa-envoy-plugin/examples/envoy-uds$ curl -i -H "Authorization: Bearer "$ALICE_TOKEN"" http://$SERVICE_URL/people
HTTP/1.1 403 Forbidden
date: Wed, 29 May 2024 12:26:19 GMT
server: envoy
content-length: 0
[2024-05-29 12:26:19.102][23][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:48] Received CheckResponse: status {
code: 7
}
dynamic_metadata {
fields {
key: "decision_id"
value {
string_value: "ac1824e9-6d1f-43f2-931f-69aa8f106e40"
}
}
}
So that's definitely a response from opa-envoy-plugin, meaning the UDS communication works. The problem thus has something to do with you config and policy. Can you share them?
I don't think so this proves the response from opa-envoy-plugin it can be sent by envoy filter also. I am using same config and policy as provided in example demo checkout this.
Are you sure? That decision ID in dynamic metadata is generated by opa-envoy-plugin and sent as part of the response. Envoy doesn't make this up.
Not fully sure leave it. I found where was the error the ALICE_TOKEN was provided in the already expired I will PR this soon. But can you please help me with these log error in envoy I could not find where is the problem i don't have much experience of envoy .
sanskar@sanskar-HP-Laptop-15s-du1xxx:~$ kubectl logs "$(kubectl get pod -l app=testapp -o jsonpath={.items..metadata.name})" -c envoy -f
[2024-05-29 18:36:58.404][15][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:111] [Tags: "ConnectionId":"0","StreamId":"566655657751400563"] ext_authz filter calling authorization server
[2024-05-29 18:36:58.405][15][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:42] Sending CheckRequest: attributes {
source {
address {
socket_address {
address: "10.244.0.1"
port_value: 40835
}
}
}
destination {
address {
socket_address {
address: "10.244.0.6"
port_value: 7000
}
}
}
request {
time {
seconds: 1717007818
nanos: 389537000
}
http {
id: "566655657751400563"
method: "GET"
headers {
key: ":authority"
value: "192.168.49.2:32430"
}
headers {
key: ":method"
value: "GET"
}
headers {
key: ":path"
value: "/book"
}
headers {
key: ":scheme"
value: "http"
}
headers {
key: "accept"
value: "*/*"
}
headers {
key: "authorization"
value: "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjIyNDEwODE1MzksIm5iZiI6MTUxNDg1MTEzOSwicm9sZSI6Imd1ZXN0Iiwic3ViIjoiWVd4cFkyVT0ifQ.ja1bgvIt47393ba_WbSBm35NrUhdxM4mOVQN8iXz8lk"
}
headers {
key: "user-agent"
value: "curl/7.81.0"
}
headers {
key: "x-forwarded-proto"
value: "http"
}
headers {
key: "x-request-id"
value: "eecc8745-23d9-4851-9e6b-13d274972058"
}
path: "/book"
host: "192.168.49.2:32430"
scheme: "http"
protocol: "HTTP/1.1"
}
}
metadata_context {
}
route_metadata_context {
}
}
[2024-05-29 18:36:58.406][15][trace][ext_authz] [source/extensions/filters/common/ext_authz/ext_authz_grpc_impl.cc:116] CheckRequest call failed with status: Internal
[2024-05-29 18:36:58.406][15][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:468] [Tags: "ConnectionId":"0","StreamId":"566655657751400563"] ext_authz filter rejected the request with an error. Response status code: 403
[2024-05-29 18:36:58.412][15][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:221] [Tags: "ConnectionId":"0","StreamId":"566655657751400563"] ext_authz filter has 0 response header(s) to add and 0 response header(s) to set to the encoded response:
But can you please help me with these log error in envoy I could not find where is the problem i don't have much experience of envoy .
Now it looks like a problem calling opa-envoy-plugin. Or rather, opa-envoy-plugin seems to have hit some error. Can you check and share its logs, too?
Also, it might help to enable decision logs with opa-envoy-plugin
: you'll see exactly when it has gotten a request, with which inputs, and what the result was.
decision_logs:
console: true
This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.
Short description
This example tutorial on opa-envoy-plugin with UDS does't work .
cc @ashutosh-narkar
it returns 403 forbidden on both get and post curl request.
Steps To Reproduce
Expected behavior
Additional context