Closed toby181 closed 1 year ago
I've already added the fluentd user to the all_access group on OpenSearch to eliminate possible authorization issues.
The combination of fluent/fluentd:v1.14-debian-1 + fluent-plugin-opensearch -v 1.0.7 worked fine a few weeks ago, now, I'm getting errors as well.
Exception:
2022-12-13 12:47:19 +0000 [error]: #0 unexpected error error_class=Faraday::Error error=":excon is not registered on Faraday::Adapter" │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.7.1/lib/faraday/middleware_registry.rb:57:in `lookup_middleware' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.7.1/lib/faraday/rack_builder.rb:113:in `adapter' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/forwardable.rb:235:in `adapter' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:633:in `block in client' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.7.1/lib/faraday/connection.rb:91:in `initialize' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.7.1/lib/faraday.rb:98:in `new' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/faraday-2.7.1/lib/faraday.rb:98:in `new' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.1.0/lib/opensearch/transport/transport/http/faraday.rb:72:in `__build_connection' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.1.0/lib/opensearch/transport/transport/base.rb:166:in `block in __build_connections' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.1.0/lib/opensearch/transport/transport/base.rb:158:in `map' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.1.0/lib/opensearch/transport/transport/base.rb:158:in `__build_connections' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/opensearch-transport-2.1.0/lib/opensearch/transport/transport/base.rb:69:in `initialize' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:650:in `new' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:650:in `client' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/opensearch_index_template.rb:62:in `host_unreachable_exceptions' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/opensearch_index_template.rb:71:in `rescue in retry_operate' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/opensearch_index_template.rb:69:in `retry_operate' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:521:in `handle_last_seen_os_major_version' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:400:in `configure' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:20:in `configure' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin.rb:187:in `configure' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:132:in `add_match' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:149:in `configure' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:719:in `block in run_worker' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in `main_process' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in `run_worker' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in `<top (required)>' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `load' │
│ 2022-12-13 12:47:19 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `<main>'
gem list
async (1.30.3)
async-http (0.54.0)
async-io (1.33.0)
async-pool (0.3.10)
aws-eventstream (1.2.0)
aws-partitions (1.675.0)
aws-sdk-core (3.168.4)
aws-sigv4 (1.5.2)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (default: 2.1.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.10)
console (1.15.3)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
digest-crc (0.6.4)
etc (default: 1.1.0)
excon (0.95.0)
ext_monitor (0.1.2)
faraday (2.7.1)
faraday-net_http (3.0.2)
faraday_middleware-aws-sigv4 (1.0.1)
fcntl (default: 1.0.0)
fiber-local (1.0.0)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-plugin-kafka (0.18.1)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.7)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-record-modifier (2.1.1)
fluentd (1.14.6)
forwardable (default: 1.3.1)
gdbm (default: 2.1.0)
getoptlong (default: 0.1.0)
http_parser.rb (0.8.0)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.6.2)
json (2.4.1, default: 2.3.0)
logger (default: 1.4.2)
ltsv (0.1.2)
matrix (default: 0.2.0)
minitest (5.13.0)
msgpack (1.5.2)
multi_json (1.15.0)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
net-telnet (0.2.0)
nio4r (2.5.8)
observer (default: 0.1.0)
oj (3.10.18)
open3 (default: 0.1.0)
opensearch-api (2.1.0)
opensearch-ruby (2.1.0)
opensearch-transport (2.1.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
power_assert (1.1.7)
prime (default: 0.1.1)
prometheus-client (4.0.0)
protocol-hpack (1.4.2)
protocol-http (0.21.0)
protocol-http1 (0.13.2)
protocol-http2 (0.14.2)
pstore (default: 0.1.0)
psych (default: 3.1.0)
racc (default: 1.4.16)
rake (13.0.1)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
reline (default: 0.1.5)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby-kafka (1.5.0)
ruby2_keywords (0.0.5)
sdbm (default: 1.0.0)
serverengine (2.3.0)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
test-unit (3.3.4)
timeout (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
uri (default: 0.10.0)
webrick (default: 1.6.1)
xmlrpc (0.3.0)
yajl-ruby (1.4.3)
yaml (default: 0.1.0)
zlib (default: 1.1.0)
using version 1.0.8 of the plugin results in the "index missing" error
I got a working fluentd, which was build in September 2022, with those gems installed
*** LOCAL GEMS ***
async (1.30.3)
async-http (0.54.0)
async-io (1.33.0)
async-pool (0.3.10)
aws-eventstream (1.2.0)
aws-partitions (1.602.0)
aws-sdk-core (3.131.2)
aws-sigv4 (1.5.0)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (default: 2.1.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.10)
console (1.15.3)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
digest-crc (0.6.4)
etc (default: 1.1.0)
excon (0.92.3)
ext_monitor (0.1.2)
faraday (1.10.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
fiber-local (1.0.0)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-plugin-kafka (0.17.5)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.7)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-record-modifier (2.1.0)
fluentd (1.14.6)
forwardable (default: 1.3.1)
gdbm (default: 2.1.0)
getoptlong (default: 0.1.0)
http_parser.rb (0.8.0)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.6.1)
json (2.4.1, default: 2.3.0)
logger (default: 1.4.2)
ltsv (0.1.2)
matrix (default: 0.2.0)
minitest (5.13.0)
msgpack (1.5.2)
multi_json (1.15.0)
multipart-post (2.2.3)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
net-telnet (0.2.0)
nio4r (2.5.8)
observer (default: 0.1.0)
oj (3.10.18)
open3 (default: 0.1.0)
opensearch-api (2.0.2)
opensearch-ruby (2.0.2)
opensearch-transport (2.0.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
power_assert (1.1.7)
prime (default: 0.1.1)
prometheus-client (4.0.0)
protocol-hpack (1.4.2)
protocol-http (0.21.0)
protocol-http1 (0.13.2)
protocol-http2 (0.14.2)
pstore (default: 0.1.0)
psych (default: 3.1.0)
racc (default: 1.4.16)
rake (13.0.1)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
reline (default: 0.1.5)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby-kafka (1.5.0)
ruby2_keywords (0.0.5)
sdbm (default: 1.0.0)
serverengine (2.3.0)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
test-unit (3.3.4)
timeout (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
uri (default: 0.10.0)
webrick (default: 1.6.1)
xmlrpc (0.3.0)
yajl-ruby (1.4.3)
yaml (default: 0.1.0)
zlib (default: 1.1.0)
When I try to build that image, with the exact gem versions again, it's failing. This is the part of my Dockerfile that takes care about the gems. Here I try to rebuild my working fluentd instance.
RUN buildDeps="sudo make gcc g++ libc-dev" \
&& apt-get update \
&& apt-get install -y --no-install-recommends $buildDeps \
&& gem install fluent-plugin-kafka -v 0.17.5 \
&& gem install fluent-plugin-opensearch -v 1.0.7 \
&& gem install fluent-plugin-record-modifier -v 2.1.0 \
&& gem install fluent-plugin-multi-format-parser -v 1.0.0 \
&& gem install fluent-plugin-prometheus -v 2.0.2 \
&& gem uninstall -I aws-partitions \
&& gem uninstall -I aws-sdk-core \
&& gem uninstall -I aws-sigv4 \
&& gem uninstall -I jmespath \
&& gem uninstall -I excon \
&& gem uninstall -I faraday \
&& gem uninstall -I opensearch-api \
&& gem uninstall -I opensearch-ruby \
&& gem uninstall -I opensearch-transport \
&& gem install aws-partitions -v 1.602.0 \
&& gem install aws-sdk-core -v 3.131.2 \
&& gem install aws-sigv4 -v 1.5.0 \
&& gem install jmespath -v 1.6.1 \
&& gem install excon -v 0.92.3 \
&& gem install faraday -v 1.10.0 \
&& gem install opensearch-api -v 2.0.2 \
&& gem install opensearch-ruby -v 2.0.2 \
&& gem install opensearch-transport -v 2.0.0 \
In this case, the instance is not even starting, because:
2022-12-13 14:27:16 +0000 [info]: gem 'fluentd' version '1.14.6' │
│ /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:2247:in `raise_if_conflicts': Unable to activate faraday_middleware-aws-sigv4-1.0.1, because faraday-1.10.0 conflicts with faraday (>= 2.0, < 3) (Gem::ConflictError) │
│ from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1369:in `activate' │
│ from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1403:in `block in activate_dependencies' │
│ from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1389:in `each' │
│ from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1389:in `activate_dependencies' │
│ from /usr/local/lib/ruby/2.7.0/rubygems/specification.rb:1371:in `activate' │
│ from /usr/local/lib/ruby/2.7.0/rubygems.rb:217:in `rescue in try_activate' │
│ from /usr/local/lib/ruby/2.7.0/rubygems.rb:210:in `try_activate' │
│ from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:151:in `rescue in require' │
│ from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:147:in `require' │
│ from /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch.rb:45:in `<top (required)>' │
│ from /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:2:in `require_relative' │
│ from /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.7/lib/fluent/plugin/out_opensearch_data_stream.rb:2:in `<top (required)>' │
│ from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' │
│ from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/registry.rb:103:in `block in search' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `each' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/registry.rb:100:in `search' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/registry.rb:44:in `lookup' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin.rb:169:in `new_impl' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin.rb:114:in `new_output' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:130:in `add_match' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:74:in `block in configure' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `each' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/agent.rb:64:in `configure' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:149:in `configure' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/engine.rb:105:in `configure' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/engine.rb:80:in `run_configure' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:668:in `run_supervisor' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:356:in `<top (required)>' │
│ from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' │
│ from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require' │
│ from /usr/local/bundle/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>' │
│ from /usr/local/bundle/bin/fluentd:23:in `load' │
│ from /usr/local/bundle/bin/fluentd:23:in `<main>'
With version 1.0.9 of fluent-plugin-opensearch the error is:
2022-12-13 14:30:02 +0000 [warn]: #0 [out_os_ds] failed to flush the buffer. retry_times=1 next_retry_time=2022-12-13 14:30:04 +0000 chunk="5efb674374c619703ee9684f5523e3d5" error_class=Fluent::Plugin::OpenSearchOutput::RecoverableRequestFailure error="could not push │
│ logs to OpenSearch cluster (): [400] {\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is missing;\"}],\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is mis │
│ sing;\"},\"status\":400}" │
│ 2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.9/lib/fluent/plugin/out_opensearch_data_stream.rb:208:in `rescue in write' │
│ 2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-opensearch-1.0.9/lib/fluent/plugin/out_opensearch_data_stream.rb:202:in `write' │
│ 2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1179:in `try_flush' │
│ 2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:1500:in `flush_thread_run' │
│ 2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start' │
│ 2022-12-13 14:30:02 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
gem list
*** LOCAL GEMS ***
async (1.30.3)
async-http (0.54.0)
async-io (1.33.0)
async-pool (0.3.10)
aws-eventstream (1.2.0)
aws-partitions (1.602.0)
aws-sdk-core (3.131.2)
aws-sigv4 (1.5.2, 1.5.0)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (default: 2.1.4)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.10)
console (1.15.3)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
digest-crc (0.6.4)
etc (default: 1.1.0)
excon (0.92.3)
ext_monitor (0.1.2)
faraday (1.10.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.0)
fiber-local (1.0.0)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-plugin-kafka (0.17.5)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.9)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-record-modifier (2.1.0)
fluentd (1.14.6)
forwardable (default: 1.3.1)
gdbm (default: 2.1.0)
getoptlong (default: 0.1.0)
http_parser.rb (0.8.0)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.6.2, 1.6.1)
json (2.4.1, default: 2.3.0)
logger (default: 1.4.2)
ltsv (0.1.2)
matrix (default: 0.2.0)
minitest (5.13.0)
msgpack (1.5.2)
multi_json (1.15.0)
multipart-post (2.2.3)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
net-telnet (0.2.0)
nio4r (2.5.8)
observer (default: 0.1.0)
oj (3.10.18)
open3 (default: 0.1.0)
opensearch-api (2.0.2)
opensearch-ruby (2.0.2)
opensearch-transport (2.0.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
power_assert (1.1.7)
prime (default: 0.1.1)
prometheus-client (4.0.0)
protocol-hpack (1.4.2)
protocol-http (0.21.0)
protocol-http1 (0.13.2)
protocol-http2 (0.14.2)
pstore (default: 0.1.0)
psych (default: 3.1.0)
racc (default: 1.4.16)
rake (13.0.1)
rdoc (default: 6.2.1.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
reline (default: 0.1.5)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby-kafka (1.5.0)
ruby2_keywords (0.0.5)
sdbm (default: 1.0.0)
serverengine (2.3.0)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
test-unit (3.3.4)
timeout (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
uri (default: 0.10.0)
webrick (default: 1.6.1)
xmlrpc (0.3.0)
yajl-ruby (1.4.3)
yaml (default: 0.1.0)
zlib (default: 1.1.0)
I just recognized that although the "index is missing" appears, logs are still shipped to OS but I'm not aware which index is missing nor whether there's might some OS400 exception somewhere.
Did another test today with the latest plugin version, latest fluentd, etc and discovered something... When the 'data_stream_name' is prefixed with a string, like "logstream-${logstash_prefix}" the log shipping is working again. Unfortunately the variable 'logstash_prefix' already consists of the string 'logstream-', so my DataStream name is in the end 'logstream-logstream-k8scluster1-'...
Anyway I would still be glad about some support!
I was able to resolve this issue with a second match directive for OpenSearch.
The first match directive filters for the tags "kube host" and uses the variable for creating the data_stream_name. If any of the tags is available, it's guaranteed (from our side) that the 'logstash_prefix' field is available within the log message.
The second match directive filters for what has no tag at all and uses a fix data_stream_name.
Together with this configuration change I was able to update fluentd and the opensearch plugin to the latest version v1.16.1-debian-amd64-1.0 and 1.1.0.
<match kube** host**>
@type opensearch_data_stream
@id out_os_ds
@log_level "debug"
data_stream_name "${logstash_prefix}"
data_stream_template_name ""
include_tag_key true
host "opensearch-cluster-prod-master"
port 9200
path ""
scheme https
ssl_verify false
ssl_version TLSv1_2
user "fluentd-k8scluster1"
password xxxxxx
reload_connections false
reconnect_on_error true
reload_on_failure true
log_os_400_reason true
logstash_format true
include_timestamp false
sniffer_class_name "Fluent::Plugin::OpenSearchSimpleSniffer"
request_timeout 120s
suppress_type_name false
<buffer logstash_prefix, tag>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 64M
total_limit_size 512MB
retry_max_interval 30
retry_forever true
</buffer>
</match>
<match **>
@type opensearch_data_stream
@id out_os_ds
@log_level "debug"
data_stream_name "logs-leftover"
data_stream_template_name ""
include_tag_key true
host "opensearch-cluster-prod-master"
port 9200
path ""
scheme https
ssl_verify false
ssl_version TLSv1_2
user "fluentd-k8scluster1"
password xxxxxx
reload_connections false
reconnect_on_error true
reload_on_failure true
log_os_400_reason true
logstash_format true
include_timestamp false
sniffer_class_name "Fluent::Plugin::OpenSearchSimpleSniffer"
request_timeout 120s
suppress_type_name false
<buffer tag>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 64M
total_limit_size 512MB
retry_max_interval 30
retry_forever true
</buffer>
</match>
Hello, @toby181. Are you initially creating index_template and datastream by yourself? I trying to use opensearch_data_stream, but it doesn't create a datastream, only an index
Hey @a-b-v You have to create the index template in advance but not the datastream. Your template has to have the '"data_stream": { },' set and certainly that your shipper got the right permissions
Hi, @toby181. I have already solved the problem, thanks. My template was wrong.
(check apply)
Steps to replicate
After updating fluentd from 1.14.6 to 1.15.3 and updating fluent-plugin-opensearch from 1.0.7 to 1.0.8 I'm facing this issue: "OpenSearchOutput::RecoverableRequestFailure error="could not push logs to OpenSearch cluster (): [400] {\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: index is missing;2: index is missing;3: index is missing;\"}],"
Full message:
I do not see anything related to this error on OpenSearch side. After reverting the changes and starting the fluentd pod (k8s) again, logs are successfully shipped.
Expected Behavior or What you need to ask
Logs are shipped as usual into DataStreams.
Using Fluentd and OpenSearch plugin versions
OS version
Bare Metal or within Docker or Kubernetes or others? Kubernetes
Fluentd v1.0 or later
fluentd --version
ortd-agent --version
fluentd 1.15.3OpenSearch plugin version
fluent-gem list
,td-agent-gem list
or your Gemfile.lockOpenSearch version (optional) 2.3.0
OpenSearch template(s) (optional)