search

uken / fluent-plugin-elasticsearch

Apache License 2.0
891 stars 310 forks source link

index pattern isn't create in ElasticSearch-oss:v7.9.3 #839

Open ghost opened 3 years ago

ghost commented 3 years ago

(check apply)

Problem

Trying to upgrade github kubernetes fluentd-elasticsearch addon with the elasticsearch-oss:v7.9.3 image. After deployment, there is no logstash- index pattern created in elasticsearch. ...

Steps to replicate

  1. fluentd:v3.1.0 configMap from kubernetes fluentd-elasticsearch configMap. Modified @log_level to debug and added reconnect/reload options 
    output.conf: |-
<match **>
  @id elasticsearch
  @type elasticsearch
  @log_level debug
  include_tag_key true
  with_transporter_log true
  host elasticsearch-logging
  port 9200
  logstash_format true
  include_timestamp true
  index_name fluentd
  type_name fluentd
  reload_connections false
  reconnect_on_error true
  reload_on_failure true
  <buffer>
    @type file
    path /var/log/fluentd-buffers/kubernetes.system.buffer
    flush_mode interval
    retry_type exponential_backoff
    flush_thread_count 2
    flush_interval 5s
    retry_forever
    retry_max_interval 30
    chunk_limit_size 2M
    total_limit_size 500M
    overflow_action block
  </buffer>
</match>
  2. fluentd:v3.1.0 daemonSet from kubernetes fluentd-elasticsearch daemonSet
  3. The fluentd pod's runtime log indicates it connects to elasticsearch successfully, 
    2020-11-23 05:48:25 +0000 [info]: fluent/log.rb:329:info: adding match pattern="**" type="elasticsearch"
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] 'host elasticsearch-logging' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-logging' doesn't have tag placeholder
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] Need substitution: false
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] 'host_placeholder elasticsearch-logging' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-logging' doesn't have tag placeholder
2020-11-23 05:48:26 +0000 [info]: [elasticsearch] GET http://elasticsearch-logging:9200/ [status:200, request:0.009s, query:n/a]
2020-11-23 05:48:26 +0000 [debug]: [elasticsearch] < {
"name" : "elasticsearch-logging-0",
"cluster_name" : "kubernetes-logging",
"cluster_uuid" : "qiyyUKcYRHiRH45aURNX8A",
"version" : {
"number" : "7.9.3",
"build_flavor" : "oss",
"build_type" : "docker",
"build_hash" : "c4138e51121ef06a6404866cddc601906fe5c868",
"build_date" : "2020-10-16T10:36:16.141335Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
2020-11-23 05:48:26 +0000 [warn]: [elasticsearch] Detected ES 7.x: `_doc` will be used as the document `_type`.
  4. elasticsearch-oss:v7.9.3 pod's log shows, 
    + export NODE_NAME=elasticsearch-logging-0
+ NODE_NAME=elasticsearch-logging-0
+ export NODE_MASTER=true
+ NODE_MASTER=true
+ export NODE_DATA=true
+ NODE_DATA=true
+ export HTTP_PORT=9200
+ HTTP_PORT=9200
+ export TRANSPORT_PORT=9300
+ TRANSPORT_PORT=9300
+ export MINIMUM_MASTER_NODES=1
+ MINIMUM_MASTER_NODES=1
+ chown -R elasticsearch:elasticsearch /data
+ ./bin/elasticsearch_logging_discovery
I1123 05:36:41.578941      10 elasticsearch_logging_discovery.go:86] Kubernetes Elasticsearch logging discovery
I1123 05:36:41.598541      10 elasticsearch_logging_discovery.go:143] Found ["10.1.0.134"]
I1123 05:36:41.598714      10 elasticsearch_logging_discovery.go:154] Endpoints = ["10.1.0.134"]
+ exec su elasticsearch -c /usr/local/bin/docker-entrypoint.sh
[2020-11-23T05:36:48,296][WARN ][o.e.c.l.LogConfigurator  ] [elasticsearch-logging-0] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
/usr/share/elasticsearch/config/log4j2.properties
[2020-11-23T05:36:49,058][INFO ][o.e.n.Node               ] [elasticsearch-logging-0] version[7.9.3], pid[15], build[oss/docker/c4138e51121ef06a6404866cddc601906fe5c868/2020-10-16T10:36:16.141335Z], OS[Linux/5.4.39-linuxkit/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/15/15+36-1562]
[2020-11-23T05:36:49,059][INFO ][o.e.n.Node               ] [elasticsearch-logging-0] JVM home [/usr/share/elasticsearch/jdk]
[2020-11-23T05:36:49,059][INFO ][o.e.n.Node               ] [elasticsearch-logging-0] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-15647596668025494642, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=docker, -Des.bundled_jdk=true]
[2020-11-23T05:36:52,389][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [aggs-matrix-stats]
[2020-11-23T05:36:52,389][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [analysis-common]
[2020-11-23T05:36:52,390][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [geo]
[2020-11-23T05:36:52,390][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [ingest-common]
[2020-11-23T05:36:52,390][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [ingest-geoip]
[2020-11-23T05:36:52,390][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [ingest-user-agent]
[2020-11-23T05:36:52,390][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [kibana]
[2020-11-23T05:36:52,390][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [lang-expression]
[2020-11-23T05:36:52,390][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [lang-mustache]
[2020-11-23T05:36:52,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [lang-painless]
[2020-11-23T05:36:52,457][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [mapper-extras]
[2020-11-23T05:36:52,458][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [parent-join]
[2020-11-23T05:36:52,458][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [percolator]
[2020-11-23T05:36:52,458][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [rank-eval]
[2020-11-23T05:36:52,458][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [reindex]
[2020-11-23T05:36:52,458][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [repository-url]
[2020-11-23T05:36:52,459][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [tasks]
[2020-11-23T05:36:52,459][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] loaded module [transport-netty4]
[2020-11-23T05:36:52,460][INFO ][o.e.p.PluginsService     ] [elasticsearch-logging-0] no plugins loaded
[2020-11-23T05:36:52,574][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-logging-0] using [1] data paths, mounts [[/data (/dev/vda1)]], net usable_space [41.2gb], net total_space [58.4gb], types [ext4]
[2020-11-23T05:36:52,574][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-logging-0] heap size [1gb], compressed ordinary object pointers [true]
[2020-11-23T05:36:52,579][WARN ][o.e.d.c.s.Settings       ] [elasticsearch-logging-0] [node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2020-11-23T05:36:52,580][WARN ][o.e.d.c.s.Settings       ] [elasticsearch-logging-0] [node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2020-11-23T05:36:52,767][INFO ][o.e.n.Node               ] [elasticsearch-logging-0] node name [elasticsearch-logging-0], node ID [58lqrPUJShO8oUNfH_63Qg], cluster name [kubernetes-logging]
[2020-11-23T05:37:02,752][INFO ][o.e.t.NettyAllocator     ] [elasticsearch-logging-0] creating NettyAllocator with the following configs: [name=unpooled, factors={es.unsafe.use_unpooled_allocator=false, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2020-11-23T05:37:02,955][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-logging-0] using discovery type [zen] and seed hosts providers [settings]
[2020-11-23T05:37:03,950][WARN ][o.e.g.DanglingIndicesState] [elasticsearch-logging-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2020-11-23T05:37:04,449][INFO ][o.e.n.Node               ] [elasticsearch-logging-0] initialized
[2020-11-23T05:37:04,449][INFO ][o.e.n.Node               ] [elasticsearch-logging-0] starting ...
[2020-11-23T05:37:04,838][INFO ][o.e.t.TransportService   ] [elasticsearch-logging-0] publish_address {10.1.0.134:9300}, bound_addresses {10.1.0.134:9300}
[2020-11-23T05:37:05,253][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-logging-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-11-23T05:37:05,263][INFO ][o.e.c.c.Coordinator      ] [elasticsearch-logging-0] setting initial configuration to VotingConfiguration{58lqrPUJShO8oUNfH_63Qg}
[2020-11-23T05:37:05,638][INFO ][o.e.c.s.MasterService    ] [elasticsearch-logging-0] elected-as-master ([1] nodes joined)[{elasticsearch-logging-0}{58lqrPUJShO8oUNfH_63Qg}{CPL16lo_SaicixbVuTjrtA}{10.1.0.134}{10.1.0.134:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{elasticsearch-logging-0}{58lqrPUJShO8oUNfH_63Qg}{CPL16lo_SaicixbVuTjrtA}{10.1.0.134}{10.1.0.134:9300}{dimr}]}
[2020-11-23T05:37:05,751][INFO ][o.e.c.c.CoordinationState] [elasticsearch-logging-0] cluster UUID set to [qiyyUKcYRHiRH45aURNX8A]
[2020-11-23T05:37:05,840][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-logging-0] master node changed {previous [], current [{elasticsearch-logging-0}{58lqrPUJShO8oUNfH_63Qg}{CPL16lo_SaicixbVuTjrtA}{10.1.0.134}{10.1.0.134:9300}{dimr}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2020-11-23T05:37:05,855][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-logging-0] publish_address {10.1.0.134:9200}, bound_addresses {10.1.0.134:9200}
[2020-11-23T05:37:05,856][INFO ][o.e.n.Node               ] [elasticsearch-logging-0] started
[2020-11-23T05:37:05,945][INFO ][o.e.g.GatewayService     ] [elasticsearch-logging-0] recovered [0] indices into cluster_state

Expected Behavior or What you need to ask

If restored to elasticsearch-oss:v7.4.3, then in the elasticsearch-oss pod runtime log, the expected index pattern is created.

[2020-11-23T00:51:48,455][INFO ][o.e.c.m.MetaDataCreateIndexService] [elasticsearch-logging-0] [logstash-2020.11.23] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-11-23T00:51:48,712][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch-logging-0] [logstash-2020.11.23/0ywWgSf8So-JmHHJFPtX-g] create_mapping [_doc]
[2020-11-23T00:51:48,746][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch-logging-0] [logstash-2020.11.23/0ywWgSf8So-JmHHJFPtX-g] update_mapping [_doc]

...

Using Fluentd and ES plugin versions

gem 'activesupport', '6.0.3.4'
gem 'elasticsearch-xpack', '7.9.0'
gem 'fluentd', '1.11.4'
gem 'fluent-plugin-concat', '2.4.0'
gem 'fluent-plugin-detect-exceptions', '0.0.13'
gem 'fluent-plugin-elasticsearch', '4.2.2'
gem 'fluent-plugin-kubernetes_metadata_filter', '2.5.2'
gem 'fluent-plugin-multi-format-parser', '1.0.0'
gem 'fluent-plugin-prometheus', '1.8.4'
gem 'fluent-plugin-systemd', '1.0.2'
gem 'oj', '3.10.15'
cosmo0920 commented 3 years ago

Trying to upgrade github kubernetes fluentd-elasticsearch addon with the elasticsearch-oss:v7.9.3 image. After deployment, there is no logstash- index pattern created in elasticsearch.

First, this addon is not our repository. You should report this issue on kubernetes repository.

Not using our fluentd kubernetes daemonset, our issue handling priority is low.

[2020-11-23T05:37:05,856][INFO ][o.e.n.Node ] [elasticsearch-logging-0] started [2020-11-23T05:37:05,945][INFO ][o.e.g.GatewayService ] [elasticsearch-logging-0] recovered [0] indices into cluster_state

Your ES 7.9.3 cluster's log just says Elasticsearch was started....