search

fluent / fluent-plugin-s3

Amazon S3 input and output plugin for Fluentd
https://docs.fluentd.org/output/s3
314 stars 218 forks source link

time_slice is not shown in s3_object_key_format if <buffer tag> is specified #295

Closed ksandrmatveyev closed 3 years ago

ksandrmatveyev commented 5 years ago

Hello,

Issue is that %{time_slice} is not shown if specified. According to https://github.com/fluent/fluent-plugin-s3#v10-style we have to specify if we want to use %Y%m%d%h%M keys. But what about %{time_slice}? does it require , too?

td-agent version: 3.5.0 OS: Windows Server 2016 Config example:

<match security>
  @type s3
  @id security-to-s3
  s3_bucket "logs"
  s3_region "eu-west-1"
  auto_create_bucket false
  check_bucket false
  check_object true
  time_slice_format %Y%m%d%H
  utc true
  s3_object_key_format "security-logs/%{hostname}/%{time_slice}/${tag}_%{index}.%{file_extension}"
  store_as json
  <buffer tag>
    @type file_single
    path C:/td-agent/security
    calc_num_records true
  </buffer>
  <format>
    @type json
  </format>
</match>
# result is the same if you specify buffer file
  <buffer>
    @type file
    path C:/td-agent/security
    flush_mode interval
    flush_interval 1m
    chunk_limit_size 1m
    flush_at_shutdown true
    flush_thread_count 3
  </buffer>

I assume that %{time_slice} should work anyway or other equivalent might be used. Could someone clarify this?

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

github-actions[bot] commented 3 years ago

This issue was automatically closed because of stale in 30 days

ehudon commented 2 years ago

I ran into the same issue with this plugin. I spent a lot of time figuring out what the issue was. I add this comment here in case someone else has the same issue.

The answer lies in how the time_slice is being assigned: https://github.com/fluent/fluent-plugin-s3/blob/89dc6abe4c2e91be4b8ac68bafc190aa0dc01b28/lib/fluent/plugin/out_s3.rb#L279-L287

It takes the timekey on the chunk and chunks are written from the buffer.

It started working the moment I added a time and timeKey to the buffer. More on this here: https://docs.fluentd.org/configuration/buffer-section#time

This solved it for me:

  <buffer time>
      @type memory
      timekey 1s
      timekey_wait 1s
  </buffer>

If you add the debug option you can see the metadata when the buffer is flushed:

2022-07-22 13:09:12.356500422 +0000 fluent.debug: {"message":"out_s3: write chunk 5e4648899493c2c299d5e733b10723c6 with metadata #<struct Fluent::Plugin::Buffer::Metadata timekey=1658495323, tag=nil, variables=nil, seq=0> to s3://[redacted]/logs/20220722-13/[redacted].gz"}