fluent-plugins-nursery / fluent-plugin-cloudwatch-logs

CloudWatch Logs Plugin for Fluentd
MIT License
201 stars 141 forks source link

Unable to dynamically use namespace_name as AWS tags for cw log groups #170

Closed voltairevi closed 4 years ago

voltairevi commented 4 years ago

Problem

For billing use mostly, I am trying to add AWS tags based on namespaces to dynamically created log groups but unable to do so. ...

Steps to replicate

I've created a custom helm chart to apply fluentd to my eks cluster. Nonetheless, deployment to k8 is standard.

Here's a snippet of my fluentd.conf:

     <filter kubernetes.**>
       @type record_transformer
       enable_ruby true
       <record>
         pod_name ${record.fetch("kubernetes", Hash.new).fetch("pod_name", "unknown_pod")}
         container_name ${record.fetch("kubernetes", Hash.new).fetch("container_name", "unknown_container")}
         namespace_name ${record.fetch("kubernetes", Hash.new).fetch("namespace_name", "unknown_namespaces")}
         log_group_name ${record.fetch("kubernetes", Hash.new).fetch("namespace_name", "unknown_namespaces")}-{{ .Values.cluster_name }}
       </record>
     </filter>
    <match **>
      @type cloudwatch_logs
      log_group_name_key log_group_name
      log_stream_name_key pod_name
      auto_create_stream true
      log_group_aws_tags { "Group": "'${namespace_name}'", "Environment": "Staging" }
      retention_in_days 30
      put_log_events_retry_limit 20
    </match>  

I've tried different variations of passing namespace_name "'${namespace_name}'", "'#{namespace_name}'". etc but end up getting this error,

2020-01-16 01:27:58 +0000 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2020-01-16 01:27:59 +0000 chunk="59c37b86386fd06422dc116573b643be" error_class=Aws::CloudWatchLogs::Errors::InvalidParameterException error="1 validation error detected: Value '{Group=${namespace_name}, Environment=Staging}' at 'tags' failed to satisfy constraint: Map value must satisfy constraint: [Member must have length less than or equal to 256, Member must have length greater than or equal to 0, Member must satisfy regular expression pattern: ^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-@]*)$]"

Expected Behavior or What you need to ask

Basically, I was hoping to pass namespace_name as an aws tag, so when the log group gets created, it will be assigned a tag which will have the namespace.

{
    "tags": {
        "Environment": "Staging", 
        "Group": "kube-system"
    }
}

Am I correct to assume that the plugin log_group_aws_tags only accepts string in this format { "tag1": "value1", "tag2": "value2" } ? Would appreciate any guidance on achieving this goal. Thanks!

...

Using Fluentd and CloudWatchLogs plugin versions

cloudwatch_agent: amazon/cloudwatch-agent:1.230621.0 fluentd: fluent/fluentd-kubernetes-daemonset:v1.7.3-debian-cloudwatch-1.0 EKS running kubernetes v1.14

cosmo0920 commented 4 years ago

Am I correct to assume that the plugin log_group_aws_tags only accepts string in this format { "tag1": "value1", "tag2": "value2" } ?

Correct.

Would appreciate any guidance on achieving this goal. Thanks!

log_group_aws_tags handles Hash object. But extract_placeholders does not handle Hash object but handles String object.... This feature request should be difficult to achieve this....