Unable to send logs to s3 using aws-elasticsearch-plugin - Help!

nareshnani529 commented 5 years ago

@type forward @id input_forward
<filter *.**>
@type record_transformer

container_id ${record["container_id"]}

<match *.**>
type aws-elasticsearch-service
logstash_format true
logstash_prefix ${tag[0]}
include_tag_key true
tag_key "Application_name"
flush_interval 5s

<buffer>
  flush_at_shutdown true
  flush_mode immediate
  flush_thread_count 8
  flush_thread_interval 1
  flush_thread_burst_interval 1
  retry_forever true
  retry_type exponential_backoff
</buffer>

    <endpoint>
            url https://vpc-xxxx.us-east-1.es.amazonaws.com
            region us-east-1
            access_key_id "xxxx"
            secret_access_key "xxxx"
    </endpoint>
<match *.**>
type copy

type s3
aws_key_id "xxxxx"
aws_sec_key "xxxx"
s3_bucket "logstream-testbucket"
s3_region "us-east-1"
path logs/
buffer_path /var/log/td-agent/buffer/s3
time_slice_format %Y-%m-%d/%H
time_slice_wait 10m

OS version - Ubuntu 18.04 LTS td-agent version : 2.5 AWS ES version - 6.8 fluent-plugin-aws-elasticsearch-service (2.1.0)

cosmo0920 commented 5 years ago

AWS ES Service plugin doesn't emit events. Try copy plugin: https://docs.fluentd.org/output/copy

nareshnani529 commented 5 years ago

@cosmo0920 Thanks for quick reply!

I tried with copy plugin if you see in the above config with store" parameter not seems working. Could you help me with any references or sample examples using copy" that will be helpful? Thank you!

cosmo0920 commented 4 years ago

Copy plugin copies pipeline stream into multiple store directives.

                     +--------------> S3 (Inside the first <store> directive)
Copy --------------- |
 (@type copy)        +--------------> AWS ES Service (Inside the second <store> directive)

Copy plugin should be used like the following:

<match *.**>
  @type copy
  <store>
    @type aws-elasticsearch-service
    logstash_format true
    logstash_prefix ${tag[0]}
    include_tag_key true
    tag_key "Application_name"
    flush_interval 5s

    <buffer>
      flush_at_shutdown true
      flush_mode immediate
      flush_thread_count 8
      flush_thread_interval 1
      flush_thread_burst_interval 1
      retry_forever true
      retry_type exponential_backoff
    </buffer>

    <endpoint>
            url https://vpc-xxxx.us-east-1.es.amazonaws.com
            region us-east-1
            access_key_id "xxxx"
            secret_access_key "xxxx"
    </endpoint>
  </store>
  <store>
    @type s3
    aws_key_id "xxxxx"
    aws_sec_key "xxxx"
    s3_bucket "logstream-testbucket"
    s3_region "us-east-1"
    path logs/
    buffer_path /var/log/td-agent/buffer/s3
    time_slice_format %Y-%m-%d/%H
    time_slice_wait 10m
  </store>
</match>

atomita / fluent-plugin-aws-elasticsearch-service

Unable to send logs to s3 using aws-elasticsearch-plugin - Help! #58