source cloudwatch_logs after end_time has passed

Problem

I read logs from 1 day from cwl and want to write it to s3. This is the only task my fluentd has.
That means there comes a time when all logs are read from cwl and are written to s3. but how do you recognize that?


<source>
  @type cloudwatch_logs
  tag cloudwatch.in
  log_group_name /ecs/my_log
  use_log_stream_name_prefix true
  state_file fluent/group_stream.in.state
  start_time "2021-03-17 00:00:00Z"
  end_time "2021-03-17 23:59:59Z"
  time_range_format "%Y-%m-%d %H:%M:%S%z"
  throttling_retry_seconds 100
</source>

<match cloudwatch.in>
  @type s3
  s3_bucket my_bucket
  s3_region "#{ENV['AWS_DEFAULT_REGION']}"
  path "mbtest/${tag}/%Y/%m/%d/%H/"
  s3_object_key_format %{path}${tag}_%{index}.%{file_extension}
  <format>
    @type single_value
    message_key log
  </format>
  <buffer tag,time>
    @type file
    path "/fluent/s3/"
    timekey 3600 # 1 hour partition
    timekey_wait 10m
    timekey_use_utc false
    chunk_limit_size 256m
  </buffer>
</match>

fluent-plugins-nursery / fluent-plugin-cloudwatch-logs

source cloudwatch_logs after end_time has passed #231

Problem