specifying aws-es index name

ranajit-jana commented 4 years ago

specifying index name

In the project below we are able to specify index name

https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.9/debian-elasticsearch7/conf/fluent.conf

I tried to specify the index name in the same lines

<match es.**>
  @type "aws-elasticsearch-service"
  type_name "access_log"
  logstash_format true
  include_tag_key true
  tag_key "@log_name"
  flush_interval 1s
  index_name my-index
</match>

but index-name is not taking effect.

Steps to replicate

<match es.**>
  @type "aws-elasticsearch-service"
  type_name "access_log"
  logstash_format true
  include_tag_key true
  tag_key "@log_name"
  flush_interval 1s
  index_name customname
  <endpoint>
    url "#{ENV['AWS_ES_CLUSTER_ENDPOINT']}"
    region "#{ENV['AWS_ES_CLUSTER_REGION']}"
    access_key_id "#{ENV['AWS_ES_USER_ACCESS_KEY']}"
    secret_access_key "#{ENV['AWS_ES_USER_SECRET_KEY']}"
  </endpoint>
</match>

The index is logstash- but i wanted it to be customname-

Expected Behavior or What you need to ask

... Index in aws-es to be customname-*

Using Fluentd and ES plugin versions

OS version - Amazon Linux 2 *Gem file 'fluent-plugin-aws-elasticsearch-service' version '2.4.0' 'fluentd' version '1.9.3'
AWS ES Service plugin version - 7.4

cosmo0920 commented 4 years ago

<match es.**>
 @type "aws-elasticsearch-service"
 type_name "access_log"
 logstash_format true
 include_tag_key true
 tag_key "@log_name"
 flush_interval 1s
 index_name customname
 <endpoint>
   url "#{ENV['AWS_ES_CLUSTER_ENDPOINT']}"
   region "#{ENV['AWS_ES_CLUSTER_REGION']}"
   access_key_id "#{ENV['AWS_ES_USER_ACCESS_KEY']}"
   secret_access_key "#{ENV['AWS_ES_USER_SECRET_KEY']}"
 </endpoint>
</match>

This is wrong usage of index_name. When specifying logstash_format as true, you should set up logstash_prefix customname for changing logstash style indices prefix name.

See: https://github.com/uken/fluent-plugin-elasticsearch#logstash_format

ranajit-jana commented 4 years ago

Pardon my ignorance. Suggested changes are working. Thank you for pointing to right direction

atomita / fluent-plugin-aws-elasticsearch-service