NameError error="uninitialized constant Fluent::Plugin::ElasticsearchOutput::Elasticsearch::UnsupportedProductError"

anand3493 commented 2 years ago

(check apply)

[x] read the contribution guideline
[ ] (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Problem

Getting error with fluentd - aws elasticsearch setup [error]: #0 unexpected error error_class=NameError error="uninitialized constant Fluent::Plugin::ElasticsearchOutput::Elasticsearch::UnsupportedProductError"

Fluentd Pods are running. But logs are not flowing. FLuentd Logs reported error with stack trace:

2022-03-29 09:55:46 +0000 [error]: #0 unexpected error error_class=NameError error="uninitialized constant Fluent::Plugin::ElasticsearchOutput::Elasticsearch::UnsupportedProductError"
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/out_elasticsearch.rb:497:in `rescue in detect_es_major_version'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/out_elasticsearch.rb:495:in `detect_es_major_version'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/out_elasticsearch.rb:487:in `block in handle_last_seen_es_major_version'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/elasticsearch_index_template.rb:56:in `retry_operate'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/out_elasticsearch.rb:484:in `handle_last_seen_es_major_version'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/out_elasticsearch.rb:336:in `configure'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/plugin.rb:187:in `configure'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/agent.rb:132:in `add_match'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/agent.rb:74:in `block in configure'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/agent.rb:64:in `each'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/agent.rb:64:in `configure'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/root_agent.rb:149:in `configure'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/engine.rb:105:in `configure'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/engine.rb:80:in `run_configure'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/supervisor.rb:716:in `block in run_worker'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/supervisor.rb:968:in `main_process'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/supervisor.rb:708:in `run_worker'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
  2022-03-29 09:55:46 +0000 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
  2022-03-29 09:55:46 +0000 [error]: #0 /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.14.1/bin/fluentd:15:in `<top (required)>'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd:23:in `load'
  2022-03-29 09:55:46 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd:23:in `<main>'

...

Steps to replicate

Fluentd daemonset with aws elasticsearch plugin is configured to push logs

Expected Behavior or What you need to ask

Fluentd to push logs to ES ...

Using Fluentd and ES plugin versions

Kubernetes
Fluentd v.14.1
fluent-plugin-elasticsearch 5.2.1
fluent-plugin-aws-elasticsearch-service 2.4.1
ES version 7.10

anand3493 commented 2 years ago

Verified the fix in 5.2.2 release. working fine. Thank you for fixing the issue @cosmo0920

cosmo0920 commented 2 years ago

Wow, thanks for the conformation.

anand3493 commented 2 years ago

@cosmo0920 In one of the environment am getting the below error for the same packages:

Can u please check on this and let me know. As the stack trace suggests, it starting from fluent-plugin-elasticsearch only

cosmo0920 commented 2 years ago

This could be caused from non-existent [default] profile on ~/.aws/credentials or /root/.aws/config.

anand3493 commented 2 years ago

@cosmo0920 I have not configured assume_role_arn for the fluentd as the destination elasticsearch is open to public. The stack trace suggests it looks for Elasticsearch major version and then trying to hit some aws api to get the same. However I have even tried to supply : verify_es_version_at_startup false default_elasticsearch_version 7 - these 2 fields so that it doesn't require to do so. But still am getting the error

cosmo0920 commented 2 years ago

I'm not sure why you are using almost obsoleted plugin that is fluent-plugin-aws-elasticsearch-service. We recommend to use fluent-plugin-opensearch. That plugin can handle Fluentd logs ingestion into the recent AWS OpenSearch service.

uken / fluent-plugin-elasticsearch