search

atomita / fluent-plugin-aws-elasticsearch-service

This repository is extends of https://github.com/uken/fluent-plugin-elasticsearch/ which made connectable to Amazon Elassticsearch Service using Aws Signers V4. Please check the requirements to connect to Amazon Elassticsearch Service without using Aws Signers V4 for before starting use. (reference https://groups.google.com/forum/#!msg/fluentd/uW87VAOqxeE/cfhenicEBAAJ)
MIT License
122 stars 49 forks source link

Error starting FluentD #72

Open AnakinPt opened 3 years ago

AnakinPt commented 3 years ago

Problem

When I start the fluentD with this plugin, I get an error
2021-08-06 16:36:11 +0000 [info]: gem 'fluentd' version '1.13.3'
2021-08-06T16:36:12.215656300Z /usr/local/bundle/gems/fluent-plugin-aws-elasticsearch-service-2.4.1/lib/fluent/plugin/out_aws-elasticsearch-service.rb:144:in `include': wrong argument type Class (expected Module) (TypeError)
2021-08-06T16:36:12.215693000Z  from /usr/local/bundle/gems/fluent-plugin-aws-elasticsearch-service-2.4.1/lib/fluent/plugin/out_aws-elasticsearch-service.rb:144:in `<module:Client>'
2021-08-06T16:36:12.215695800Z  from /usr/local/bundle/gems/fluent-plugin-aws-elasticsearch-service-2.4.1/lib/fluent/plugin/out_aws-elasticsearch-service.rb:143:in `<module:Elasticsearch>'
2021-08-06T16:36:12.215698000Z  from /usr/local/bundle/gems/fluent-plugin-aws-elasticsearch-service-2.4.1/lib/fluent/plugin/out_aws-elasticsearch-service.rb:141:in `<class:ElasticsearchOutput>'
2021-08-06T16:36:12.215710700Z  from /usr/local/bundle/gems/fluent-plugin-aws-elasticsearch-service-2.4.1/lib/fluent/plugin/out_aws-elasticsearch-service.rb:140:in `<module:Plugin>'
2021-08-06T16:36:12.215712900Z  from /usr/local/bundle/gems/fluent-plugin-aws-elasticsearch-service-2.4.1/lib/fluent/plugin/out_aws-elasticsearch-service.rb:9:in `<top (required)>'
2021-08-06T16:36:12.215714900Z  from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2021-08-06T16:36:12.215716600Z  from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2021-08-06T16:36:12.215718300Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/registry.rb:102:in `block in search'
2021-08-06T16:36:12.215720000Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `each'
2021-08-06T16:36:12.215721700Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/registry.rb:99:in `search'
2021-08-06T16:36:12.215723400Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/registry.rb:44:in `lookup'
2021-08-06T16:36:12.215725100Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/plugin.rb:160:in `new_impl'
2021-08-06T16:36:12.215726700Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/plugin.rb:109:in `new_output'
2021-08-06T16:36:12.215728500Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/agent.rb:130:in `add_match'
2021-08-06T16:36:12.215730200Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in `block in configure'
2021-08-06T16:36:12.215731900Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `each'
2021-08-06T16:36:12.215733600Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `configure'
2021-08-06T16:36:12.215736000Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:146:in `configure'
2021-08-06T16:36:12.215737800Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in `configure'
2021-08-06T16:36:12.215739400Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in `run_configure'
2021-08-06T16:36:12.215741100Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:663:in `run_supervisor'
2021-08-06T16:36:12.215742900Z  from /usr/local/bundle/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:344:in `<top (required)>'
2021-08-06T16:36:12.215744800Z  from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2021-08-06T16:36:12.215746800Z  from /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2021-08-06T16:36:12.215748600Z  from /usr/local/bundle/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
2021-08-06T16:36:12.215750300Z  from /usr/local/bundle/bin/fluentd:23:in `load'
2021-08-06T16:36:12.215751900Z  from /usr/local/bundle/bin/fluentd:23:in `<main>'

Steps to replicate

I'm using this Dockerfile: FROM fluent/fluentd:edge-debian

USER root RUN ["gem", "install", "fluent-plugin-aws-elasticsearch-service"] RUN ["gem", "install", "fluent-plugin-record-modifier"] RUN ["gem", "install", "fluent-plugin-http-healthcheck"] RUN ["apt-get", "update"] RUN ["apt-get", "-y", "install", "curl"]

HEALTHCHECK --interval=30s --timeout=3s CMD curl -v http://localhost:8888 || exit 1

COPY entrypoint.sh /bin/

USER fluent

Provide example config and message

<match gelf-logging>
  @type "aws-elasticsearch-service"
  type_name "access_log"
  logstash_format true
  include_tag_key true
  tag_key "@log_name"
  logstash_prefix dummy
  templates { "dummy": "/fluentd/etc/dummy.json" }
  flush_interval 10s

  <endpoint>
    url "https://myElasticSearchInAws.eu-central-1.es.amazonaws.com"
    region "eu-central-1"
  </endpoint>
</match>

Expected Behavior or What you need to ask

I expect the application to start normally ...

Using Fluentd and ES plugin versions

Everything was provided in the error

bloodguard commented 3 years ago

Same issue here. Started to happen friday. There is no new version of the ES plugin so i don't why it started to happen out of the blue like that.

bloodguard commented 3 years ago

@AnakinPt Found the issue. I would recommend to read this issue: https://github.com/uken/fluent-plugin-elasticsearch/issues/912