Linkerd-proxy logging full header contents of incoming http requests for log level debug and trace.

bhanuprakash-1 commented 1 month ago

What is the issue?

I have set log level to debug for my linkerd proxy configuration. But we observed that, this debug log level is resulting in linkerd-proxy containers logging whole incoming request headers. This is a serious security issue as headers contains bearer token, secrets etc and ideally proxy pod should not have logged any http request payload/header. The documentation for log levels says nothing about logging customer data: https://linkerd.io/2.15/reference/proxy-log-level/

Why is linkerd proxy logging contents of the incoming http request as part of application logs?? Do not log the contents/payload/headers of the incoming and outgoing requests ever even for debug or trace log levels. Only log your linkerd application flow traces/strings in trace logs with no content from requests.

Linkerd-version: 2.13.1

Example Console log:

[305438.538290s] DEBUG ThreadId(01) inbound:accept{client.addr=172.18.37.97:46520}:server{port=80}:http:http{name=data-plane--weather-data-provider-service:80}:profile:http1: linkerd_proxy_http::client: headers={"host": "data-plane--weather-data-provider-service", "user-agent": "",........ and all the headers including all secrets and "traceparent": "00-fc9fcfda6d04569e26aa0f002a24d6ed-ccf325f9bce1e444-00", "content-type": "application/json; charset=utf-8", "content-length": "3009", "l5d-client-id": "default.default.serviceaccount.identity.linkerd.cluster.local"}

The request's source is an internal pod which also has linkerd-proxy side-car container injected.

How can it be reproduced?

Set linkerd-proxy log level to debug.

Logs, error output, etc

Added above

output of `linkerd check -o short`

 linkerd check -o short
Linkerd core checks
===================

linkerd-version
---------------
‼ cli is up-to-date
    unsupported version channel: stable-2.11.0
    see https://linkerd.io/2.11/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    failed to parse actual version: unsupported version format: 2.13.1
    see https://linkerd.io/2.11/checks/#l5d-version-control for hints
‼ control plane and cli versions match
    control plane running 2.13.1 but cli running stable-2.11.0
    see https://linkerd.io/2.11/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
        * linkerd-destination-855c469f76-8rlrg (2.13.1)
        * linkerd-destination-855c469f76-h96wg (2.13.1)
        * linkerd-destination-855c469f76-hcx2h (2.13.1)
        * linkerd-identity-5b76b77589-jhxqm (2.13.1)
        * linkerd-identity-5b76b77589-sj2p6 (2.13.1)
        * linkerd-identity-5b76b77589-xv8rw (2.13.1)
        * linkerd-proxy-injector-6b48b56cd-6fnrw (2.13.1)
        * linkerd-proxy-injector-6b48b56cd-724z6 (2.13.1)
        * linkerd-proxy-injector-6b48b56cd-8pnrx (2.13.1)
    see https://linkerd.io/2.11/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
    linkerd-destination-855c469f76-8rlrg running 2.13.1 but cli running stable-2.11.0
    see https://linkerd.io/2.11/checks/#l5d-cp-proxy-cli-version for hints

linkerd-ha-checks
-----------------
‼ pod injection disabled on kube-system
    kube-system namespace needs to have the label config.linkerd.io/admission-webhooks: disabled if injector webhook failure policy is Fail
    see https://linkerd.io/2.11/checks/#l5d-injection-disabled for hints

Status check results are √

Environment

Kubernetes Version: 1.27.9
Cluster Environment: AKS
Host OS: Node OS is CBL-Mariner/Linux, kernel version 5.15.153.1-2.cm2, containerd://1.6.26
Linkerd version: 2.13.1

Possible solution

Do not log the contents/payload/headers of the incoming and outgoing requests ever even for debug or trace log levels. Only log your linkerd application code trace logs with no content from requests.

Think of linkerd-proxy containers as an external service or SDK, which is handling customer data and so do not log customer's data in logs or give warning in the documentation explicitly here: https://linkerd.io/2.15/reference/proxy-log-level/ and give expose a config to scrub any customer data.

Additional context

No response

Would you like to work on fixing this bug?

None

bhanuprakash-1 commented 1 month ago

Also, can we know for sure that for other log levels like warn or error, linkerd-proxy won't log requests contents?? If this is documented somewhere, can you please point me to it :).

Request contents, including headers, body, query and path params are often highly sensitive and we take special care to not log them in our application logs and even disable core dumps as they might contain sensitive data. But linkerd-proxy casually logging request headers seems to be a big issue from my perspective.

bhanuprakash-1 commented 1 month ago

Hi team,

Any update :) ?

@wmorgan @admc

wmorgan commented 1 month ago

@bhanuprakash-1 Thanks for pointing this out. We have a fix in progress.

linkerd / linkerd2