search

atomita / fluent-plugin-aws-elasticsearch-service

This repository is extends of https://github.com/uken/fluent-plugin-elasticsearch/ which made connectable to Amazon Elassticsearch Service using Aws Signers V4. Please check the requirements to connect to Amazon Elassticsearch Service without using Aws Signers V4 for before starting use. (reference https://groups.google.com/forum/#!msg/fluentd/uW87VAOqxeE/cfhenicEBAAJ)
MIT License
122 stars 49 forks source link

Getting "undefined method `[]`" when using this with ES 6.0 on AWS #42

Open daviddyball opened 6 years ago

daviddyball commented 6 years ago

Trying to post to an ES 6.0 domain in AWS and getting the following error. I can't find any information on the undefined method \[]`` error it's complaining about though. Is there any additional logging I can enable to make debugging this any easier?

2018-03-01 19:20:39 +0000 [info]: #0 fluentd worker is now running worker=0
2018-03-01 19:20:44 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2018-03-01 19:20:45 +0000 chunk="5665ec1468f652a7261c16755f23a988" error_class=NoMethodError error="undefined method `[]' for nil:NilClass"
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/excon-0.60.0/lib/excon/connection.rb:244:in `request'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/faraday-0.14.0/lib/faraday/adapter/excon.rb:49:in `call'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/faraday_middleware-aws-signers-v4-0.1.1/lib/faraday_middleware/request/aws_signers_v4.rb:41:in `call'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/faraday-0.14.0/lib/faraday/rack_builder.rb:143:in `build_response'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/faraday-0.14.0/lib/faraday/connection.rb:387:in `run_request'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-6.0.1/lib/elasticsearch/transport/transport/http/faraday.rb:23:in `block in perform_request'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-6.0.1/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-6.0.1/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/elasticsearch-transport-6.0.1/lib/elasticsearch/transport/client.rb:131:in `perform_request'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/elasticsearch-api-6.0.1/lib/elasticsearch/api/actions/ping.rb:20:in `ping'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.0.1/lib/fluent/plugin/out_elasticsearch.rb:191:in `client'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.0.1/lib/fluent/plugin/out_elasticsearch.rb:425:in `rescue in send_bulk'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.0.1/lib/fluent/plugin/out_elasticsearch.rb:420:in `send_bulk'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.0.1/lib/fluent/plugin/out_elasticsearch.rb:407:in `write'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-aws-elasticsearch-service-1.0.0/lib/fluent/plugin/out_aws-elasticsearch-service.rb:61:in `write'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.25/lib/fluent/plugin/output.rb:1093:in `try_flush'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.25/lib/fluent/plugin/output.rb:1318:in `flush_thread_run'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.25/lib/fluent/plugin/output.rb:439:in `block (2 levels) in start'
  2018-03-01 19:20:44 +0000 [warn]: #0 /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.14.25/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
daviddyball commented 6 years ago

Linking to https://github.com/atomita/fluent-plugin-aws-elasticsearch-service/issues/41 which is a general query about 6.0 support in the plugin.

daviddyball commented 6 years ago

I will try spinning up an ES 5.5 cluster and see if my configuration gets the same errors.

daviddyball commented 6 years ago

Nope, same with a 5.5 cluster. Any help is appreciated.