[Elasticsearch] Specify the sensitivity of `elasticsearch.customHeaders`

[afharo]

The setting elasticsearch.customHeaders allows users to append any headers to the requests in Elasticsearch.

In some cases, users may set these additional headers to authenticate/pass through a proxy or a firewall of some sort. The values of those headers may be considered sensitive in some scenarios, and we should redact them when logging them.

However, we currently don't have a way to specify which of those customHeaders are sensitive and which ones are OK to appear in the log entries.

We need to find a way to specify the level of sensitivity of those headers. Potentially, we could accept the customHeaders to be either the value (for BWC reasons) or an object with { sensitive: true, value: <actual_value> } (sensitive is up to discussion 😇 ).

Workaround

Users can use a logging RewriteAppender to filter those entries in the logs. kibana.yml would look like:

elasticsearch.customHeaders:
  my-custom-secret-header: super-secret-value

logging:
  appenders:
    json-layout:
      type: console
      layout:
        type: json
    # Add RewriteAppender to redact the header
    censor-my-custom-secret-header:
      type: rewrite
      appenders: [ "json-layout" ]
      policy:
        type: meta
        mode: update
        properties:
          - path: "http.request.headers.my-custom-secret-header"
            value: "[REDACTED]"

# Use the rewrite appender.
logging.root.appenders: [ default, "censor-my-custom-secret-header" ]

However, this configuration is cumbersome and probably, easy to forget to apply, potentially leaking secrets before applying the config.

On top of that, developers may still log it in other ways we may not be able to control (i.e.: logger.warn(`Something wrong happened ${JSON.stringify(error)}`);).

I'd suggest we allow setting the sensitivity of the headers and use it in both: the logger filter and a new custom Serializer provided to the ES Client constructor.

WDYT?

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[lukeelmers]

On top of that, developers may still log it in other ways we may not be able to control (i.e.: logger.warn(Something wrong happened ${JSON.stringify(error)});).

++ IMO this is the primary reason to consider adding a feature like this. I think it is useful to take care of filtering sensitive data "at the source" whenever possible, vs relying on folks applying special logging configurations which aren't guaranteed to be bulletproof.

And since elasticsearch.customHeaders feels like something that would be commonly used for sensitive data, I think it would make a lot of sense here.

(sensitive is up to discussion 😇 )

My +1 to sensitive, as it is consistent with the similar option we expose when registering uiSettings.

[pgayvallet]

elasticsearch.customHeaders is already somewhat of an edge case / work-around for some very specific (and usually bad) customer infrastructures. Given we already have a workaround (for the workaround) via the redaction appender, I'd say we don't want to do anything more, and will close the issue.

As always, please feel free to reopen if anyone thinks otherwise.