search

uken / fluent-plugin-elasticsearch

Apache License 2.0
890 stars 310 forks source link

elasticsearch_dynamic ignores api_key #950

Closed nimaaj93 closed 2 years ago

nimaaj93 commented 2 years ago

(check apply)

Problem

elasticsearch_dynamic ignores api_key. I have a working fluentd configuration with a static index name with @type elasticsearch that ships logs to Elastic Cloud from Kubernetes. After I tried to set index name dynamically with elasticsearch_dynamic, authentication header seems to be omitted from requests and now I am getting 401 errors:

Could not communicate to Elasticsearch, resetting connection and trying again. [401] {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

...

Steps to replicate

This is my configuration for fluentd-output.conf file for the Helm chart:

<match kubernetes.var.log.containers.**ingress**.log>
        @type elasticsearch
        host "xxx.elastic-cloud.com"
        port "xxxx"
        custom_headers {"Authorization":"ApiKey xxx"}
        scheme https
        index_name "ingress-logs"
        <buffer>
          @type file
          path /opt/bitnami/fluentd/logs/buffers/logs-ingress.buffer
          flush_thread_count 2
          flush_interval 5s
        </buffer>
      </match>

changing @type elasticsearch to @type elasticsearch_dynamic will cause 401:

<match kubernetes.var.log.containers.**ingress**.log>
        @type elasticsearch_dynamic
        host "xxx.elastic-cloud.com"
        port "xxxx"
        custom_headers {"Authorization":"ApiKey xxx"}
        scheme https
        index_name "ingress-logs"
        <buffer>
          @type file
          path /opt/bitnami/fluentd/logs/buffers/logs-ingress.buffer
          flush_thread_count 2
          flush_interval 5s
        </buffer>
      </match>

I tried the same config with api_key property and got the same error.

Expected Behavior or What you need to ask

Set api_key (or custom auth headers) for the requests to Elasticsearch.

...

Using Fluentd and ES plugin versions

activesupport (6.1.3.2) addressable (2.7.0) aws-eventstream (1.1.1) aws-partitions (1.465.0) aws-sdk-core (3.114.0) aws-sdk-kms (1.43.0) aws-sdk-s3 (1.95.1) aws-sdk-sqs (1.39.0) aws-sigv4 (1.2.3) bigdecimal (default: 1.4.1) bundler (2.2.18, 2.2.15) cmath (default: 1.0.0) concurrent-ruby (1.1.8) cool.io (1.7.1) csv (default: 3.0.9) date (default: 2.0.0) did_you_mean (1.3.0) digest-crc (0.6.3) domain_name (0.5.20190701) e2mmap (default: 0.1.0) elasticsearch (7.13.0) elasticsearch-api (7.13.0) elasticsearch-transport (7.13.0) elasticsearch-xpack (7.13.0) etc (default: 1.0.1) excon (0.82.0) faraday (1.4.2) faraday-em_http (1.0.0) faraday-em_synchrony (1.0.0) faraday-excon (1.1.0) faraday-net_http (1.0.1) faraday-net_http_persistent (1.1.0) fcntl (default: 1.0.0) ffi (1.15.1) ffi-compiler (1.0.1) fiddle (default: 1.0.0) fileutils (default: 1.1.0) fluent-config-regexp-type (1.0.0) fluent-plugin-concat (2.5.0) fluent-plugin-detect-exceptions (0.0.13) fluent-plugin-elasticsearch (5.0.3) fluent-plugin-grafana-loki (1.2.16) fluent-plugin-kafka (0.16.3) fluent-plugin-kubernetes_metadata_filter (2.7.0) fluent-plugin-multi-format-parser (1.0.0) fluent-plugin-prometheus (2.0.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-s3 (1.6.0) fluent-plugin-systemd (1.0.5) fluentd (1.13.0, 1.12.4) forwardable (default: 1.2.0) http (4.4.1) http-accept (1.7.0) http-cookie (1.0.3) http-form_data (2.3.0) http-parser (1.2.3) http_parser.rb (0.6.0) i18n (1.8.10) io-console (default: 0.4.7) ipaddr (default: 1.2.2) irb (default: 1.0.0) jmespath (1.4.0) json (2.1.0) jsonpath (1.1.0) kubeclient (4.9.1) logger (default: 1.3.0) lru_redux (1.1.0) ltsv (0.1.2) matrix (default: 0.1.0) mime-types (3.3.1) mime-types-data (3.2021.0225) minitest (5.14.4, 5.11.3) msgpack (1.4.2) multi_json (1.15.0) multipart-post (2.1.1) mutex_m (default: 0.1.0) net-telnet (0.2.0) netrc (0.11.0) oj (3.3.10) openssl (default: 2.1.2) ostruct (default: 0.1.0) power_assert (1.1.3) prime (default: 0.1.0) prometheus-client (2.1.0) psych (default: 3.1.0) public_suffix (4.0.6) rake (13.0.3, 12.3.3) rdoc (default: 6.1.2) recursive-open-struct (1.1.3) rest-client (2.1.0) rexml (default: 3.1.9.1) rss (default: 0.2.7) ruby-kafka (1.3.0) ruby2_keywords (0.0.4) rubygems-update (3.2.15) scanf (default: 1.0.0) sdbm (default: 1.0.0) serverengine (2.2.4) shell (default: 0.7) sigdump (0.2.4) stringio (default: 0.0.2) strptime (0.2.5) strscan (default: 1.0.0) sync (default: 0.5.0) systemd-journal (1.4.2) test-unit (3.2.9) thwait (default: 0.1.0) tracer (default: 0.1.0) tzinfo (2.0.4) tzinfo-data (1.2021.1) unf (0.1.4) unf_ext (0.0.7.7) webrick (1.7.0, default: 1.4.4) xmlrpc (0.3.0) yajl-ruby (1.4.1) zeitwerk (2.4.2) zlib (default: 1.0.0)


* ES version (optional)
* ES template(s) (optional)
cosmo0920 commented 2 years ago

elasticsearch_dynamic will be deprecated and not actively maintained. If you send a patch to handle this issue, we can review a PR.

cosmo0920 commented 2 years ago

Now, elasticsearch_dynamic has been marked as deprecated. No longer actively maintained anymore.