Open markuman opened 3 years ago
I read logs from 1 day from cwl and want to write it to s3. This is the only task my fluentd has. That means there comes a time when all logs are read from cwl and are written to s3. but how do you recognize that?
<source> @type cloudwatch_logs tag cloudwatch.in log_group_name /ecs/my_log use_log_stream_name_prefix true state_file fluent/group_stream.in.state start_time "2021-03-17 00:00:00Z" end_time "2021-03-17 23:59:59Z" time_range_format "%Y-%m-%d %H:%M:%S%z" throttling_retry_seconds 100 </source> <match cloudwatch.in> @type s3 s3_bucket my_bucket s3_region "#{ENV['AWS_DEFAULT_REGION']}" path "mbtest/${tag}/%Y/%m/%d/%H/" s3_object_key_format %{path}${tag}_%{index}.%{file_extension} <format> @type single_value message_key log </format> <buffer tag,time> @type file path "/fluent/s3/" timekey 3600 # 1 hour partition timekey_wait 10m timekey_use_utc false chunk_limit_size 256m </buffer> </match>
Problem
I read logs from 1 day from cwl and want to write it to s3. This is the only task my fluentd has.
That means there comes a time when all logs are read from cwl and are written to s3. but how do you recognize that?