search

aws / amazon-cloudwatch-logs-for-fluent-bit

A Fluent Bit output plugin for CloudWatch Logs
Apache License 2.0
174 stars 49 forks source link

Unknown configuration property 'log_retention_days' #234

Closed sunnygoel87 closed 2 years ago

sunnygoel87 commented 2 years ago

Hi,

We setup FluentBit agent on Amazon EKS cluster to offload Application logs on Amazon CloudWatch. By default, CloudWatch service retains logs forever. They don't expire. But we want CloudWatch to retain the logs only for 14 days.

Based on that requirement, We added "log_renetion_days" parameter in FluentBit ConfigMap resource and set value to 14 days. After deploying the Deamonset, we see the following error in the POD. Could you please help us resolve this issue ?

[2022/03/03 07:22:00] [error] [config] cloudwatch_logs: unknown configuration property 'log_retention_days'. The following properties are allowed: region, log_group_name, log_stream_name, log_stream_prefix, log_key, extra_user_agent, log_format, role_arn, auto_create_group, endpoint, sts_endpoint, metric_namespace, and metric_dimensions.

zhonghui12 commented 2 years ago

Hi @sunnygoel87,

log_retention_days should be supported by cloudwatch_logs plugin: https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch#configuration-parameters. Could I know which aws-for-fluent-bit version you are using?

sunnygoel87 commented 2 years ago

Hi @zhonghui12 , Thanks for your response. I used following image to deploy Fluent-bit as DaemonSet on EKS cluster. Please let me know if you need any other details. Thanks.

image: amazon/aws-for-fluent-bit:2.22.0

zhonghui12 commented 2 years ago

I see. That's our latest image. Could you please share your config if possible?

sunnygoel87 commented 2 years ago

@zhonghui12 , I was able to configure the retention policy on LogGroup successfully. After getting the latest container image - I created a custom IAM policy(as shown below) and attached that to worker node InstanceRole. And this setup resolved the issue. Thanks for your support.

Policy Name - CloudWatchAgentPutLogsRetention

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "logs:PutRetentionPolicy", "Resource": "*" } ] }