search

atomita / fluent-plugin-aws-elasticsearch-service

This repository is extends of https://github.com/uken/fluent-plugin-elasticsearch/ which made connectable to Amazon Elassticsearch Service using Aws Signers V4. Please check the requirements to connect to Amazon Elassticsearch Service without using Aws Signers V4 for before starting use. (reference https://groups.google.com/forum/#!msg/fluentd/uW87VAOqxeE/cfhenicEBAAJ)
MIT License
122 stars 49 forks source link

Using example config, queues but never sends logs to Elasticsearch #47

Closed ilovemysillybanana closed 6 years ago

ilovemysillybanana commented 6 years ago

I'm trying to send my logs using the default example configuration shown here in the logs but it's not working, it just says it's constantly en-queuing but never sends anything out. I installed fluentd from the ruby gem and run it this way: nohup fluentd -c $HOME/fluent/fluent.conf -vv&>$HOME/fluent/logs/fluentd.log&

Here's an example of how I changed my:

<source>
  @type tail
  format apache2
  time_format "%d/%b/%Y:%T %z"
  path "/var/log/httpd/access_log_20180702-190810.log"
  pos_file "/home/ec2-user/fluent/pos/apachev2.access.pos"
  tag "jose.apache.access"
</source>

<match jose.**>
  @type aws-elasticsearch-service
  type_name "access_log"
  logstash_format true
  include_tag_key true
  tag_key "@log_name"
  flush_interval 1s
  <endpoint>
    url "https://myurl.us-east-2.es.amazonaws.com"
    region "us-east-2"
    access_key_id "ACCESS_KEY"
    secret_access_key "SECRET_ACCESS_KEY"
  </endpoint>
</match>

The logs don't show any errors to my understanding, but I do see that it constantly is adding things to a queue but never sends them out.

[ec2-user@ip-10-0-0-xxx logs]$ tail -100 fluentd.log
2018-07-03 13:40:57 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
2018-07-03 13:40:58 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
2018-07-03 13:40:59 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
2018-07-03 13:41:00 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
2018-07-03 13:41:01 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
2018-07-03 13:41:02 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
2018-07-03 13:41:03 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
2018-07-03 13:41:04 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
2018-07-03 13:41:05 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
2018-07-03 13:41:06 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
2018-07-03 13:41:07 +0000 [trace]: #0 fluent/log.rb:281:trace: enqueueing all chunks in buffer instance=46954251059480
ilovemysillybanana commented 6 years ago

UPDATE:

My same configuration up there ^ works fine if you don't use the pattern matching jose.**, you have to use ** or it won't work.

This is the same issue mentioned here:

https://github.com/atomita/fluent-plugin-aws-elasticsearch-service/issues/6

So I'm going to close this issue as it's a duplicate. But we really need better logging in this.