Metricbeat doesn't work properly with `api_key` in metricbeat elasticsearch module but work ok with `username`\`password`

[gitpel]

Metricbeat doesn't work properly with api_key in metricbeat elasticsearch module but work ok with username\password the same api_key works fine in the output.elasticsearch.api_key

/etc/metricbeat/modules.d/elasticsearch-xpack.yml

---
- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://fqdn:8443"]
  ssl:
   certificate_authorities:
      - /etc/ssl/certs/ca-certificates.crt
  api_key: xxxxxxx:yyyyyyyy

! note that the same api_key works fine in /etc/metricbeat/metricbeat.yml

---
output:
  elasticsearch:
    hosts: ['https://server-01:9200', 'https://server-02:9200', ...]
    ssl:
      certificate_authorities: ['/etc/ssl/certs/ca-certificates.crt']
    api_key: xxxxxxx:yyyyyyyy

metricbeat:
  config:
    modules:
      path: ${path.config}/modules.d/*.yml
      reload:
        enabled: false

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

error log:

....
Dec  4 21:12:00 elasticsearch-01 metricbeat[214918]: 2021-12-04T21:12:00.767Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.node_stats: HTTP error 401 in : 401 Unauthorized
Dec  4 21:12:10 elasticsearch-01 metricbeat[214918]: 2021-12-04T21:12:10.606Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.enrich: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized
Dec  4 21:12:10 elasticsearch-01 metricbeat[214918]: 2021-12-04T21:12:10.610Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized
Dec  4 21:12:10 elasticsearch-01 metricbeat[214918]: 2021-12-04T21:12:10.632Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized
Dec  4 21:12:10 elasticsearch-01 metricbeat[214918]: 2021-12-04T21:12:10.660Z#011INFO#011module/wrapper.go:259#011Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized
....

• Version: metricbeat/stable,now 7.15.2 amd64 [installed]
• Operating System: Debian GNU/Linux 11 (bullseye)

solution: use username\password instead of api_key in /etc/metricbeat/modules.d/elasticsearch-xpack.yml

[gitpel]

update: The same behaviour with metricbeat kibana module api_key results HTTP error 401 in : 401 Unauthorized, but works fine with username\password, the same api_key from the same user works fine with output.elasticsearch.api_key

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[rseldner]

Since the modules support these standard http config options, we can use a header to pass the base64 ApiKey

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://my-elasticsearch.url:9200"]
  headers:
    Authorization: ApiKey XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX==

Or if they ApiKey is in the keystore...

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://my-elasticsearch.url:9200"]
  headers:
    Authorization: ApiKey ${api_key}