Internal listener access logging in Envoy

istio / istio

Connect, secure, control, and observe services.

https://istio.io

Apache License 2.0

35.93k stars 7.75k forks source link

Internal listener access logging in Envoy #50321

Open howardjohn opened 7 months ago

howardjohn commented 7 months ago

When we use HBONE in envoy, we get access logs showing unexpected info

Without hbone:

[2024-04-08T20:53:05.142Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 1648 0 0 "10.244.0.1" "curl/8.7.1" "d68f23d1-02a2-4999-8572-6f154573c74c" "172.18.0.101" "10.244.0.10:80" outbound|80||echo.default.svc.cluster.local 10.244.0.7:53060 10.244.0.7:8080 10.244.0.1:59040 -

With hbone:

[2024-04-08T20:44:12.363Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 1648 3 2 "10.244.0.1" "curl/8.7.1" "66dc2115-b7f5-45eb-96c2-2a2b221559f7" "172.18.0.101" "envoy://connect_originate/10.244.0.10:80" outbound|80||echo.default.svc.cluster.local envoy://internal_client_address/ 10.244.0.7:8080 10.244.0.1:27113 -

Probably we can tweak these to avoid exposing internals

zirain commented 7 months ago

I recall istio only print listener access log when 404 in the past?

howardjohn commented 7 months ago

This isn't about listener access logs, but for hbone the destination endpoint is an internal listener instead of ip

zirain commented 7 months ago

you can either change the hardcode log format or just remove it(this option may cause difficult to debug).

howardjohn commented 7 months ago

I think we want to log the real source and dest IP. This may just be changing the format to access some other info, but we might need to make changes in Envoy/istio-proxy to expose that info if its not available (not sure what info we have)

howardjohn commented 6 months ago

There are two parts here:

Source ip is envoy://internal_client_address/. This I think requires us to either change envoy core, or pass "real source ip" as metadata across the internal listener
Destination IP is "envoy://connect_originate/10.244.0.10:80". This seems easier, since the original IP is known to be in EDS metadata and access logs can access it. However, we would want to do something like metadata.ip OR upstream_addr for the non HBONE cases; I don't see how to do that in envoy. %CEL(...) seemed promising but has a lot of bugs

zirain commented 4 months ago

envoy://connect_originate/10.244.0.10:80 is from %UPSTREAM_HOST% which related to https://github.com/envoyproxy/envoy/blob/main/source/common/network/address_impl.cc#L470. envoy://internal_client_address/ is from %UPSTREAM_LOCAL_ADDRESS% which related to https://github.com/envoyproxy/envoy/blob/main/source/extensions/bootstrap/internal_listener/client_connection_factory.cc#L30

keithmattix commented 1 month ago

Not stale