"Validation Failed: 1: index is missing;" after updating fluentd and os plugin

toby181 commented 1 year ago

(check apply)

[x] read the contribution guideline
[ ] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

After updating fluentd from 1.14.6 to 1.15.3 and updating fluent-plugin-opensearch from 1.0.7 to 1.0.8 I'm facing this issue: "OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (): [400] {\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is missing;2: index is missing;3: index is missing;\"}],"

Full message:

2022-11-09 15:27:17 +0000 [info]: #0 starting fluentd worker pid=19 ppid=7 worker=0
2022-11-09 15:27:17 +0000 [debug]: #0 [out_os_ds] buffer started instance=2980 stage_size=0 queue_size=0
2022-11-09 15:27:17 +0000 [debug]: #0 [out_os_ds] flush_thread actually running
2022-11-09 15:27:17 +0000 [debug]: #0 [out_os_ds] flush_thread actually running
2022-11-09 15:27:17 +0000 [debug]: #0 [out_os_ds] flush_thread actually running
2022-11-09 15:27:17 +0000 [debug]: #0 [out_os_ds] flush_thread actually running
2022-11-09 15:27:17 +0000 [debug]: #0 [out_os_ds] flush_thread actually running
2022-11-09 15:27:17 +0000 [debug]: #0 [out_os_ds] flush_thread actually running
2022-11-09 15:27:17 +0000 [debug]: #0 [out_os_ds] enqueue_thread actually running
2022-11-09 15:27:17 +0000 [debug]: #0 [out_os_ds] flush_thread actually running
2022-11-09 15:27:17 +0000 [debug]: #0 [out_os_ds] flush_thread actually running
2022-11-09 15:27:17 +0000 [info]: #0 Subscribe to topic cke-infra
2022-11-09 15:27:17 +0000 [info]: #0 fluentd worker is now running worker=0
2022-11-09 15:27:22 +0000 [debug]: #0 [out_os_ds] taking back chunk for errors. chunk="5ed0b4ab6ca34535356201d8194918d1"
2022-11-09 15:27:22 +0000 [warn]: #0 [out_os_ds] failed to flush the buffer. retry_times=0 next_retry_time=2022-11-09 15:27:24 +0000 chunk="5ed0b4ab6ca34535356201d8194918d1" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (): [400] {\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is missing;2: index is missing;3: index is missing;4: index is missing;5: index is missing;6: index is missing;7: index is missing;8: index is missing;9: index is missing;10: index is missing;\"}],\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is missing;2: index is missing;3: index is missing;4: index is missing;5: index is missing;6: index is missing;7: index is missing;8: index is missing;9: index is missing;10: index is missing;\"},\"status\":400}"
  2022-11-09 15:27:22 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.8/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'
  2022-11-09 15:27:22 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.8/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'
  2022-11-09 15:27:22 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1180:in `try_flush'
  2022-11-09 15:27:22 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1501:in `flush_thread_run'
  2022-11-09 15:27:22 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
  2022-11-09 15:27:22 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-11-09 15:27:23 +0000 [debug]: #0 [out_os_ds] taking back chunk for errors. chunk="5ed0b4ab6c9fbb8ea1c780a7006fc4ea"
2022-11-09 15:27:23 +0000 [debug]: #0 [out_os_ds] taking back chunk for errors. chunk="5ed0b4ab6c914f51511f02dc83e65556"
2022-11-09 15:27:23 +0000 [warn]: #0 [out_os_ds] failed to flush the buffer. retry_times=0 next_retry_time=2022-11-09 15:27:24 +0000 chunk="5ed0b4ab6c914f51511f02dc83e65556" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (): [400] {\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is missing;\"}],\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is missing;\"},\"status\":400}"
  2022-11-09 15:27:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.8/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'
  2022-11-09 15:27:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.8/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'
  2022-11-09 15:27:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1180:in `try_flush'
  2022-11-09 15:27:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1501:in `flush_thread_run'
  2022-11-09 15:27:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
  2022-11-09 15:27:23 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-11-09 15:27:23 +0000 [warn]: #0 [out_os_ds] failed to flush the buffer. retry_times=0 next_retry_time=2022-11-09 15:27:24 +0000 chunk="5ed0b4ab6c9fbb8ea1c780a7006fc4ea" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (): [400] {\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is missing;2: index is missing;3: index is missing;\"}],

I do not see anything related to this error on OpenSearch side. After reverting the changes and starting the fluentd pod (k8s) again, logs are successfully shipped.

Expected Behavior or What you need to ask

Logs are shipped as usual into DataStreams.

Using Fluentd and OpenSearch plugin versions

OS version
Bare Metal or within Docker or Kubernetes or others? Kubernetes
Fluentd v1.0 or later
- paste result of fluentd --version or td-agent --version fluentd 1.15.3

OpenSearch plugin version

paste boot log of fluentd or td-agent

<source>
kafka
</source>
<match **>
@type opensearch_data_stream
@id out_os_ds
@log_level "debug"
data_stream_name "${logstash_prefix}"
data_stream_template_name ""
include_tag_key true
host "opensearch-cluster-prod-master"
port 9200
path ""
scheme https
ssl_verify false
ssl_version TLSv1_2
user "fluentd-k8scluster1"
password xxxxxx
reload_connections false
reconnect_on_error true
reload_on_failure true
log_os_400_reason true
logstash_format true
include_timestamp false
sniffer_class_name "Fluent::Plugin::OpenSearchSimpleSniffer"
request_timeout 120s
suppress_type_name false
<buffer logstash_prefix, tag>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 64M
total_limit_size 512MB
retry_max_interval 30
retry_forever true
</buffer>
</match>

paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

abbrev (default: 0.1.0)
async (1.30.3)
async-http (0.56.6)
async-io (1.34.0)
async-pool (0.3.12)
aws-eventstream (1.2.0)
aws-partitions (1.658.0)
aws-sdk-core (3.166.0)
aws-sigv4 (1.5.2)
base64 (default: 0.1.1)
benchmark (default: 0.2.0)
bigdecimal (default: 3.1.1)
bundler (default: 2.3.7)
cgi (default: 0.3.1)
concurrent-ruby (1.1.10)
console (1.16.2)
cool.io (1.7.1)
csv (default: 3.2.2)
date (default: 3.2.2)
debug (1.4.0)
delegate (default: 0.2.0)
did_you_mean (default: 1.6.1)
digest (default: 3.1.0)
digest-crc (0.6.4)
drb (default: 2.1.0)
english (default: 0.7.1)
erb (default: 2.2.3)
error_highlight (default: 0.3.0)
etc (default: 1.3.0)
excon (0.94.0)
faraday (1.10.2)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.1)
fiber-local (1.0.0)
fiddle (default: 1.1.0)
fileutils (default: 1.6.0)
find (default: 0.1.1)
fluent-plugin-kafka (0.18.1)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.8)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-record-modifier (2.1.1)
fluentd (1.15.3)
forwardable (default: 1.3.2)
getoptlong (default: 0.1.1)
http_parser.rb (0.8.0)
io-console (default: 0.5.11)
io-nonblock (default: 0.1.0)
io-wait (default: 0.2.1)
ipaddr (default: 1.2.4)
irb (default: 1.4.1)
jmespath (1.6.1)
json (2.6.2, default: 2.6.1)
logger (default: 1.5.0)
ltsv (0.1.2)
matrix (0.4.2)
minitest (5.15.0)
msgpack (1.6.0)
multi_json (1.15.0)
multipart-post (2.2.3)
mutex_m (default: 0.1.1)
net-ftp (0.1.3)
net-http (default: 0.2.0)
net-imap (0.2.3)
net-pop (0.1.1)
net-protocol (default: 0.1.2)
net-smtp (0.3.1)
nio4r (2.5.8)
nkf (default: 0.1.1)
observer (default: 0.1.1)
oj (3.13.22)
open-uri (default: 0.2.0)
open3 (default: 0.1.1)
opensearch-api (2.0.2)
opensearch-ruby (2.0.3)
opensearch-transport (2.0.1)
openssl (default: 3.0.0)
optparse (default: 0.2.0)
ostruct (default: 0.5.2)
pathname (default: 0.2.0)
power_assert (2.0.1)
pp (default: 0.3.0)
prettyprint (default: 0.1.1)
prime (0.1.2)
prometheus-client (4.0.0)
protocol-hpack (1.4.2)
protocol-http (0.22.9)
protocol-http1 (0.14.6)
protocol-http2 (0.14.2)
pstore (default: 0.1.1)
psych (default: 4.0.3)
racc (default: 1.6.0)
rake (13.0.6)
rbs (2.1.0)
rdoc (default: 6.4.0)
readline (default: 0.0.3)
readline-ext (default: 0.1.4)
reline (default: 0.3.0)
resolv (default: 0.2.1)
resolv-replace (default: 0.1.0)
rexml (3.2.5)
rinda (default: 0.1.1)
rss (0.2.9)
ruby-kafka (1.5.0)
ruby2_keywords (default: 0.0.5)
securerandom (default: 0.1.1)
serverengine (2.3.0)
set (default: 1.0.2)
shellwords (default: 0.1.0)
sigdump (0.2.4)
singleton (default: 0.1.1)
stringio (default: 3.0.1)
strptime (0.2.5)
strscan (default: 3.0.1)
syslog (default: 0.1.0)
tempfile (default: 0.1.2)
test-unit (3.5.3)
time (default: 0.2.0)
timeout (default: 0.2.0)
timers (4.3.5)
tmpdir (default: 0.1.2)
traces (0.4.1)
tsort (default: 0.1.0)
typeprof (0.21.2)
tzinfo (2.0.5)
tzinfo-data (1.2022.6)
un (default: 0.2.0)
uri (default: 0.11.0)
weakref (default: 0.1.1)
webrick (1.7.0)
yajl-ruby (1.4.3)
yaml (default: 0.2.0)
zlib (default: 2.1.1)

OpenSearch version (optional) 2.3.0

OpenSearch template(s) (optional)


PUT _index_template/logstream_k8s_template
{
"index_patterns": [
"logstream-k8scluster1-*"
],
"data_stream": { },
"priority": 200,
"template": {
"settings": {
  "index": {
    "number_of_shards": "1",
    "number_of_replicas": "1",
    "mapping": {
      "total_fields": {
        "limit": "2000"
      }
    }
  }
},
"mappings": {
  "properties": {
    "timestamp": {
      "type": "date"
    }
  }
}
},
"composed_of": [
"k8s_component_template"
]
}

toby181 commented 1 year ago

I've already added the fluentd user to the all_access group on OpenSearch to eliminate possible authorization issues.

toby181 commented 1 year ago

The combination of fluent/fluentd:v1.14-debian-1 + fluent-plugin-opensearch -v 1.0.7 worked fine a few weeks ago, now, I'm getting errors as well.

Exception:

2022-12-13 12:47:19 +0000 [error]: #0 unexpected error error_class=Faraday::Error error=":excon is not registered on Faraday::Adapter"                                                                                                                                       │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.7.1/lib/faraday/middleware_registry.rb:57:in `lookup_middleware'                                                                                                                                    │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.7.1/lib/faraday/rack_builder.rb:113:in `adapter'                                                                                                                                                    │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/forwardable.rb:235:in `adapter'                                                                                                                                                                            │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:633:in `block in client'                                                                                                                   │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.7.1/lib/faraday/connection.rb:91:in `initialize'                                                                                                                                                    │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.7.1/lib/faraday.rb:98:in `new'                                                                                                                                                                      │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.7.1/lib/faraday.rb:98:in `new'                                                                                                                                                                      │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.1.0/lib/opensearch/transport/transport/http/faraday.rb:72:in `__build_connection'                                                                                                      │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.1.0/lib/opensearch/transport/transport/base.rb:166:in `block in __build_connections'                                                                                                   │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.1.0/lib/opensearch/transport/transport/base.rb:158:in `map'                                                                                                                            │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.1.0/lib/opensearch/transport/transport/base.rb:158:in `__build_connections'                                                                                                            │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.1.0/lib/opensearch/transport/transport/base.rb:69:in `initialize'                                                                                                                      │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:650:in `new'                                                                                                                               │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:650:in `client'                                                                                                                            │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/opensearch_index_template.rb:62:in `host_unreachable_exceptions'                                                                                             │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/opensearch_index_template.rb:71:in `rescue in retry_operate'                                                                                                 │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/opensearch_index_template.rb:69:in `retry_operate'                                                                                                           │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:521:in `handle_last_seen_os_major_version'                                                                                                 │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:400:in `configure'                                                                                                                         │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:20:in `configure'                                                                                                              │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin.rb:187:in `configure'                                                                                                                                                        │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:132:in `add_match'                                                                                                                                                         │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'                                                                                                                                                 │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'                                                                                                                                                               │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'                                                                                                                                                          │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:149:in `configure'                                                                                                                                                    │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'                                                                                                                                                        │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'                                                                                                                                                     │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:719:in `block in run_worker'                                                                                                                                          │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in `main_process'                                                                                                                                                 │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in `run_worker'                                                                                                                                                   │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in `<top (required)>'                                                                                                                                        │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'                                                                                                                                                        │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'                                                                                                                                                        │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'                                                                                                                                                           │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `load'                                                                                                                                                                                           │
│   2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `<main>'

gem list

async (1.30.3)
async-http (0.54.0)
async-io (1.33.0)
async-pool (0.3.10)
aws-eventstream (1.2.0)
aws-partitions (1.675.0)
aws-sdk-core (3.168.4)
aws-sigv4 (1.5.2)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (default: 2.1.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.10)
console (1.15.3)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
digest-crc (0.6.4)
etc (default: 1.1.0)
excon (0.95.0)
ext_monitor (0.1.2)
faraday (2.7.1)
faraday-net_http (3.0.2)
faraday_middleware-aws-sigv4 (1.0.1)
fcntl (default: 1.0.0)
fiber-local (1.0.0)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-plugin-kafka (0.18.1)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.7)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-record-modifier (2.1.1)
fluentd (1.14.6)
forwardable (default: 1.3.1)
gdbm (default: 2.1.0)
getoptlong (default: 0.1.0)
http_parser.rb (0.8.0)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.6.2)
json (2.4.1, default: 2.3.0)
logger (default: 1.4.2)
ltsv (0.1.2)
matrix (default: 0.2.0)
minitest (5.13.0)
msgpack (1.5.2)
multi_json (1.15.0)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
net-telnet (0.2.0)
nio4r (2.5.8)
observer (default: 0.1.0)
oj (3.10.18)
open3 (default: 0.1.0)
opensearch-api (2.1.0)
opensearch-ruby (2.1.0)
opensearch-transport (2.1.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
power_assert (1.1.7)
prime (default: 0.1.1)
prometheus-client (4.0.0)
protocol-hpack (1.4.2)
protocol-http (0.21.0)
protocol-http1 (0.13.2)
protocol-http2 (0.14.2)
pstore (default: 0.1.0)
psych (default: 3.1.0)
racc (default: 1.4.16)
rake (13.0.1)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
reline (default: 0.1.5)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby-kafka (1.5.0)
ruby2_keywords (0.0.5)
sdbm (default: 1.0.0)
serverengine (2.3.0)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
test-unit (3.3.4)
timeout (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
uri (default: 0.10.0)
webrick (default: 1.6.1)
xmlrpc (0.3.0)
yajl-ruby (1.4.3)
yaml (default: 0.1.0)
zlib (default: 1.1.0)

toby181 commented 1 year ago

using version 1.0.8 of the plugin results in the "index missing" error

toby181 commented 1 year ago

I got a working fluentd, which was build in September 2022, with those gems installed

*** LOCAL GEMS ***

async (1.30.3)
async-http (0.54.0)
async-io (1.33.0)
async-pool (0.3.10)
aws-eventstream (1.2.0)
aws-partitions (1.602.0)
aws-sdk-core (3.131.2)
aws-sigv4 (1.5.0)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (default: 2.1.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.10)
console (1.15.3)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
digest-crc (0.6.4)
etc (default: 1.1.0)
excon (0.92.3)
ext_monitor (0.1.2)
faraday (1.10.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
fiber-local (1.0.0)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-plugin-kafka (0.17.5)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.7)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-record-modifier (2.1.0)
fluentd (1.14.6)
forwardable (default: 1.3.1)
gdbm (default: 2.1.0)
getoptlong (default: 0.1.0)
http_parser.rb (0.8.0)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.6.1)
json (2.4.1, default: 2.3.0)
logger (default: 1.4.2)
ltsv (0.1.2)
matrix (default: 0.2.0)
minitest (5.13.0)
msgpack (1.5.2)
multi_json (1.15.0)
multipart-post (2.2.3)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
net-telnet (0.2.0)
nio4r (2.5.8)
observer (default: 0.1.0)
oj (3.10.18)
open3 (default: 0.1.0)
opensearch-api (2.0.2)
opensearch-ruby (2.0.2)
opensearch-transport (2.0.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
power_assert (1.1.7)
prime (default: 0.1.1)
prometheus-client (4.0.0)
protocol-hpack (1.4.2)
protocol-http (0.21.0)
protocol-http1 (0.13.2)
protocol-http2 (0.14.2)
pstore (default: 0.1.0)
psych (default: 3.1.0)
racc (default: 1.4.16)
rake (13.0.1)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
reline (default: 0.1.5)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby-kafka (1.5.0)
ruby2_keywords (0.0.5)
sdbm (default: 1.0.0)
serverengine (2.3.0)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
test-unit (3.3.4)
timeout (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
uri (default: 0.10.0)
webrick (default: 1.6.1)
xmlrpc (0.3.0)
yajl-ruby (1.4.3)
yaml (default: 0.1.0)
zlib (default: 1.1.0)

When I try to build that image, with the exact gem versions again, it's failing. This is the part of my Dockerfile that takes care about the gems. Here I try to rebuild my working fluentd instance.

RUN buildDeps="sudo make gcc g++ libc-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && gem install fluent-plugin-kafka -v 0.17.5 \
 && gem install fluent-plugin-opensearch -v 1.0.7 \
 && gem install fluent-plugin-record-modifier -v 2.1.0 \
 && gem install fluent-plugin-multi-format-parser -v 1.0.0 \
 && gem install fluent-plugin-prometheus -v 2.0.2 \
 && gem uninstall -I aws-partitions \
 && gem uninstall -I aws-sdk-core \
 && gem uninstall -I aws-sigv4 \
 && gem uninstall -I jmespath \
 && gem uninstall -I excon \
 && gem uninstall -I faraday \
 && gem uninstall -I opensearch-api \
 && gem uninstall -I opensearch-ruby \
 && gem uninstall -I opensearch-transport \
 && gem install aws-partitions -v 1.602.0 \
 && gem install aws-sdk-core -v 3.131.2 \
 && gem install aws-sigv4 -v 1.5.0 \
 && gem install jmespath -v 1.6.1 \
 && gem install excon -v 0.92.3 \
 && gem install faraday -v 1.10.0 \
 && gem install opensearch-api -v 2.0.2 \
 && gem install opensearch-ruby -v 2.0.2 \
 && gem install opensearch-transport -v 2.0.0 \

In this case, the instance is not even starting, because:

2022-12-13 14:27:16 +0000 [info]: gem 'fluentd' version '1.14.6'                                                                                                                                                                                                             │
│ /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate faraday_middleware-aws-sigv4-1.0.1, because faraday-1.10.0 conflicts with faraday (>= 2.0, < 3) (Gem::ConflictError)                                                    │
│     from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate'                                                                                                                                                                                              │
│     from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1403:in `block in activate_dependencies'                                                                                                                                                                        │
│     from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1389:in `each'                                                                                                                                                                                                  │
│     from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1389:in `activate_dependencies'                                                                                                                                                                                 │
│     from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1371:in `activate'                                                                                                                                                                                              │
│     from /usr/local/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate'                                                                                                                                                                                               │
│     from /usr/local/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate'                                                                                                                                                                                                         │
│     from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require'                                                                                                                                                                            │
│     from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require'                                                                                                                                                                                      │
│     from /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>'                                                                                                                                                  │
│     from /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:2:in `require_relative'                                                                                                                                       │
│     from /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:2:in `<top (required)>'                                                                                                                                       │
│     from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'                                                                                                                                                                                       │
│     from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'                                                                                                                                                                                       │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search'                                                                                                                                                                               │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each'                                                                                                                                                                                          │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search'                                                                                                                                                                                        │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup'                                                                                                                                                                                         │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl'                                                                                                                                                                                        │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output'                                                                                                                                                                                      │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match'                                                                                                                                                                                        │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure'                                                                                                                                                                                │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each'                                                                                                                                                                                              │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure'                                                                                                                                                                                         │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:149:in `configure'                                                                                                                                                                                   │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure'                                                                                                                                                                                       │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure'                                                                                                                                                                                    │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor'                                                                                                                                                                              │
│     from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>'                                                                                                                                                                       │
│     from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'                                                                                                                                                                                       │
│     from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'                                                                                                                                                                                       │
│     from /usr/local/bundle/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'                                                                                                                                                                                          │
│     from /usr/local/bundle/bin/fluentd:23:in `load'                                                                                                                                                                                                                          │
│     from /usr/local/bundle/bin/fluentd:23:in `<main>'

With version 1.0.9 of fluent-plugin-opensearch the error is:

2022-12-13 14:30:02 +0000 [warn]: #0 [out_os_ds] failed to flush the buffer. retry_times=1 next_retry_time=2022-12-13 14:30:04 +0000 chunk="5efb674374c619703ee9684f5523e3d5" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push  │
│ logs to OpenSearch cluster (): [400] {\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is missing;\"}],\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is mis │
│ sing;\"},\"status\":400}"                                                                                                                                                                                                                                                    │
│   2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.9/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write'                                                                                                        │
│   2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.9/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write'                                                                                                                  │
│   2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush'                                                                                                                                                 │
│   2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run'                                                                                                                                          │
│   2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'                                                                                                                                  │
│   2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

gem list

*** LOCAL GEMS ***

async (1.30.3)
async-http (0.54.0)
async-io (1.33.0)
async-pool (0.3.10)
aws-eventstream (1.2.0)
aws-partitions (1.602.0)
aws-sdk-core (3.131.2)
aws-sigv4 (1.5.2, 1.5.0)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (default: 2.1.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.10)
console (1.15.3)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
digest-crc (0.6.4)
etc (default: 1.1.0)
excon (0.92.3)
ext_monitor (0.1.2)
faraday (1.10.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
fiber-local (1.0.0)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-plugin-kafka (0.17.5)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.9)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-record-modifier (2.1.0)
fluentd (1.14.6)
forwardable (default: 1.3.1)
gdbm (default: 2.1.0)
getoptlong (default: 0.1.0)
http_parser.rb (0.8.0)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.6.2, 1.6.1)
json (2.4.1, default: 2.3.0)
logger (default: 1.4.2)
ltsv (0.1.2)
matrix (default: 0.2.0)
minitest (5.13.0)
msgpack (1.5.2)
multi_json (1.15.0)
multipart-post (2.2.3)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
net-telnet (0.2.0)
nio4r (2.5.8)
observer (default: 0.1.0)
oj (3.10.18)
open3 (default: 0.1.0)
opensearch-api (2.0.2)
opensearch-ruby (2.0.2)
opensearch-transport (2.0.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
power_assert (1.1.7)
prime (default: 0.1.1)
prometheus-client (4.0.0)
protocol-hpack (1.4.2)
protocol-http (0.21.0)
protocol-http1 (0.13.2)
protocol-http2 (0.14.2)
pstore (default: 0.1.0)
psych (default: 3.1.0)
racc (default: 1.4.16)
rake (13.0.1)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
reline (default: 0.1.5)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby-kafka (1.5.0)
ruby2_keywords (0.0.5)
sdbm (default: 1.0.0)
serverengine (2.3.0)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
test-unit (3.3.4)
timeout (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
uri (default: 0.10.0)
webrick (default: 1.6.1)
xmlrpc (0.3.0)
yajl-ruby (1.4.3)
yaml (default: 0.1.0)
zlib (default: 1.1.0)

toby181 commented 1 year ago

I just recognized that although the "index is missing" appears, logs are still shipped to OS but I'm not aware which index is missing nor whether there's might some OS400 exception somewhere.

toby181 commented 1 year ago

Did another test today with the latest plugin version, latest fluentd, etc and discovered something... When the 'data_stream_name' is prefixed with a string, like "logstream-${logstash_prefix}" the log shipping is working again. Unfortunately the variable 'logstash_prefix' already consists of the string 'logstream-', so my DataStream name is in the end 'logstream-logstream-k8scluster1-'...

Anyway I would still be glad about some support!

toby181 commented 1 year ago

I was able to resolve this issue with a second match directive for OpenSearch.

The first match directive filters for the tags "kube host" and uses the variable for creating the data_stream_name. If any of the tags is available, it's guaranteed (from our side) that the 'logstash_prefix' field is available within the log message.

The second match directive filters for what has no tag at all and uses a fix data_stream_name.

Together with this configuration change I was able to update fluentd and the opensearch plugin to the latest version v1.16.1-debian-amd64-1.0 and 1.1.0.

  <match kube** host**>
  @type opensearch_data_stream
  @id out_os_ds
  @log_level "debug"
  data_stream_name "${logstash_prefix}"
  data_stream_template_name ""
  include_tag_key true
  host "opensearch-cluster-prod-master"
  port 9200
  path ""
  scheme https
  ssl_verify false
  ssl_version TLSv1_2
  user "fluentd-k8scluster1"
  password xxxxxx
  reload_connections false
  reconnect_on_error true
  reload_on_failure true
  log_os_400_reason true
  logstash_format true
  include_timestamp false
  sniffer_class_name "Fluent::Plugin::OpenSearchSimpleSniffer"
  request_timeout 120s
  suppress_type_name false
  <buffer logstash_prefix, tag>
    flush_thread_count 8
    flush_interval 5s
    chunk_limit_size 64M
    total_limit_size 512MB
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

  <match **>
  @type opensearch_data_stream
  @id out_os_ds
  @log_level "debug"
  data_stream_name "logs-leftover"
  data_stream_template_name ""
  include_tag_key true
  host "opensearch-cluster-prod-master"
  port 9200
  path ""
  scheme https
  ssl_verify false
  ssl_version TLSv1_2
  user "fluentd-k8scluster1"
  password xxxxxx
  reload_connections false
  reconnect_on_error true
  reload_on_failure true
  log_os_400_reason true
  logstash_format true
  include_timestamp false
  sniffer_class_name "Fluent::Plugin::OpenSearchSimpleSniffer"
  request_timeout 120s
  suppress_type_name false
  <buffer tag>
    flush_thread_count 8
    flush_interval 5s
    chunk_limit_size 64M
    total_limit_size 512MB
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

a-b-v commented 1 year ago

Hello, @toby181. Are you initially creating index_template and datastream by yourself? I trying to use opensearch_data_stream, but it doesn't create a datastream, only an index

toby181 commented 1 year ago

Hey @a-b-v You have to create the index template in advance but not the datastream. Your template has to have the '"data_stream": { },' set and certainly that your shipper got the right permissions

a-b-v commented 1 year ago

Hi, @toby181. I have already solved the problem, thanks. My template was wrong.

fluent / fluent-plugin-opensearch