Retention in days seems has no effect

[OleksandrMykytenko]

Problem

I have specified retention_in_days 3 parameter for out_cloudwatch_logs. Log group within Cloudwatch was created successfully and I have bunch of logs in my streams. However, logs are not expiring (I see Never Expire in log group's metadata).

Steps to replicate

  <match **>
    @type cloudwatch_logs
    @id out_cloudwatch_logs_containers
    region "#{ENV.fetch('REGION')}"
    log_group_name "/eks/#{ENV.fetch('CLUSTER_NAME')}/containers"
    log_stream_name_key stream_name
    remove_log_stream_name_key true
    auto_create_stream true
    retention_in_days 3
    <buffer>
      flush_interval 5
      chunk_limit_size 2m
      queued_chunks_limit_size 32
      retry_forever true
    </buffer>
  </match>

Expected Behavior or What you need to ask

Log Group which being created should have retention policy with expiration in 3 days.

Using Fluentd and CloudWatchLogs plugin versions

• Amazon Linux 2
• Within Kubernetes (inside EKS)
• Docker image fluent/fluentd-kubernetes-daemonset:v1.3.3-debian-cloudwatch-1.2

[pharindoko]

Similar - checked permissions. Currently the used role has logs:* ....

Fluentd Image: v0.12-debian-cloudwatch

Received following exception in Kubernetes cluster when log groups are created with auto_create_stream true.

2019-04-08 14:33:58 +0000 [warn]: failed to set retention policy for Log group 'Kubernetes.Cluster.kube-system' with error ["/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.1/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/plugins/response_target.rb:23:in `call'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/request.rb:70:in `send_request'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/aws-sdk-cloudwatchlogs-1.4.0/lib/aws-sdk-cloudwatchlogs/client.rb:1490:in `put_retention_policy'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:381:in `put_retention_policy'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:371:in `create_log_group'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:177:in `block in write'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:144:in `each'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/fluent-plugin-cloudwatch-logs-0.4.5/lib/fluent/plugin/out_cloudwatch_logs.rb:144:in `write'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-0.12.43/lib/fluent/buffer.rb:354:in `write_chunk'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-0.12.43/lib/fluent/buffer.rb:333:in `pop'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-0.12.43/lib/fluent/output.rb:342:in `try_flush'", "/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-0.12.43/lib/fluent/output.rb:149:in `run'"]

[pharindoko]

If anyone else ever has the same exception: allowed retention values....

--retention-in-days (integer) The number of days to retain the log events in the specified log group. Possible values are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653.

[OleksandrMykytenko]

Seems it works now. I forgot to add IAM permission logs:PutRetentionPolicy for my log group. I suppose that was an issue. Tested with values 3 and 7 for retention_in_days parameter

[ghost]

I tried to install Fluentd as daemon set in my EKS cluster following this tutorial Set Up FluentD as a DaemonSet to Send Logs to CloudWatch Logs

You can find used fluend config via this link

It doesn't have retention_in_days field, so I added it myself.

But retention value works only upon creation. If I update config, it doesn't change retention policy for existing log group

[tontondematt]

I tried to install Fluentd as daemon set in my EKS cluster following this tutorial Set Up FluentD as a DaemonSet to Send Logs to CloudWatch Logs

You can find used fluend config via this link

It doesn't have retention_in_days field, so I added it myself.

But retention value works only upon creation. If I update config, it doesn't change retention policy for existing log group

Can you confirm this is working for you please ?

[ghost]

@tontondematt I will check tonight and let you know. Previously made workaround creating log group and their retention period explicitly

[mukundjalan]

@tontondematt did the update ever work for you?