Modify filter not working with Record Accessors

[kharenis]

Bug Report

When using the modify filter to copy nested fields using Record Accessors, the filter fails to find the fields.

To Reproduce

• Configuration:

  [FILTER]
  Name                modify
  Match               kube.*
  Copy                $kubernetes['labels']['app']    application
  Copy                $kubernetes['labels']['app.kubernetes.io/name'] application
  Copy                $kubernetes['container_name']   application
  Remove              $kubernetes['pod_id']
  Remove              $kubernetes['docker_id']
  Remove              $kubernetes['container_hash']

• Fluent Bit debug log

  [2022/01/21 14:54:40] [debug] [filter:modify:modify.4] Rule COPY $kubernetes['labels']['app'] TO application : No keys matching $kubernetes['labels']['app'] found, not applying rule
  [2022/01/21 14:54:40] [debug] [filter:modify:modify.4] Rule COPY $kubernetes['labels']['app.kubernetes.io/name'] TO application : No keys matching $kubernetes['labels']['app.kubernetes.io/name'] found, not applying rule
  [2022/01/21 14:54:40] [debug] [filter:modify:modify.4] Rule COPY $kubernetes['container_name'] TO application : No keys matching $kubernetes['container_name'] found, not applying rule

• Output (sent to aws Cloudwatch)

  {
  "log": "[14:55:24 INF]  REDACTED",
  "stream": "stdout",
  "time": "2022-01-21T14:55:24.532788648Z",
  "kubernetes": {
      "pod_name": "REDACTED",
      "namespace_name": "REDACTED",
      "pod_id": "723efe40-122a-4b93-960c-f79b09dc583f",
      "labels": {
          "app": "REDACTED",
          "controller-uid": "618de15b-8c23-47a2-8ac8-34cc1d59122d",
          "job-name": "REDACTED",
          "release": "REDACTED"
      },
      "annotations": {
          "kubernetes.io/psp": "eks.privileged"
      },
      "host": "REDACTED",
      "container_name": "REDACTED",
      "docker_id": "59a6665e0e27623e475a49a382e070c63c72b8673e6debcdc63444a455ecb6a9",
      "container_hash": "REDACTED",
      "container_image": "REDACTED"
  }
  }

• Steps to reproduce the problem:

Expected behavior 'application' field should be added to the log structure with the value taken from one of the 3 found fields. $kubernetes['pod_id'], $kubernetes['docker_id'], $kubernetes['container_hash'] should be removed from the log structure.

Your Environment

• Version used: Latest
• Configuration:

  
  [FILTER]
  Name                kubernetes
  Match               kube.*
  Merge_Log           On
  Keep_Log            Off
  K8S-Logging.Parser  On
  K8S-Logging.Exclude On

[FILTER] Name expect Match kube.* key_exists kubernetes key_exists log action exit

[FILTER] Name grep Match kube.* Exclude $kubernetes['namespace_name'] kube-system

[FILTER] Name stdout Match kube.*

[FILTER] Name modify Match kube.* Copy $kubernetes['labels']['app'] application Copy $kubernetes['labels']['app.kubernetes.io/name'] application Copy $kubernetes['container_name'] application Remove $kubernetes['pod_id'] Remove $kubernetes['docker_id'] Remove $kubernetes['container_hash']

* Environment name and version (e.g. Kubernetes? What version?): Kubernetes, Latest
* Operating System and version: AWS Linux
* Filters and plugins: kubernetes, expect, grep, stdout, modify, cloudwatch_logs

[nokute78]

https://github.com/fluent/fluent-bit/issues/2152 Currently, filter_modify doesn't support record accessor.

Only STRING:KEY of Conditions supports record accessor. https://docs.fluentbit.io/manual/pipeline/filters/modify#conditions You can set Record Accessor as STRING:KEY for nested key.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[andrewnazarov]

This would be a great help if one wants to work with nested keys. We are facing the same issue trying to modify the k8s metainformation.

[dgsardina]

Same use case and issue, ended up forwarding to fluentd instead.

[chrw]

I'm having the same issue trying to copy the Kubernetes namespace into a variable.

[ryan65]

Hi Anyone found a workaround. I need to remove the pod_id from the kubernetes metadata object, couldnt find a way either with modify or record_modifier No nested support.

[pranavmarla]

Hi Anyone found a workaround. I need to remove the pod_id from the kubernetes metadata object, couldnt find a way either with modify or record_modifier No nested support.

@ryan65 At this point, I suspect a Lua filter is your only option

[ryan65]

Thanks , will try.

[RalfWenzel]

hi, i have the same issue, pod_id and hash values should be removed . Is there any plan to add this as a feature?

[agup006]

@nokute78 for the PR does that help RA with modify?

[sabdalla80]

Just keeping this thread alive, I can also use this feature.

[maxmoshe]

If anybody needs an example for a solution using lua, this worked for me:

[FILTER]
    Name lua
    Match *
    script /fluent-bit/scripts/alter_fields.lua
    call alter_fields

function alter_fields(tag, timestamp, record)
          new_record = record
          new_record["applicationName"] = record["kubernetes"]["namespace_name"]
          new_record["subsystemName"] = record["kubernetes"]["container_name"]
          new_record["kubernetes"]["annotations"] = nil
          return 1, timestamp, new_record
end

Docs: https://docs.fluentbit.io/manual/pipeline/filters/lua

[blackluck]

Anyone have a solution to this kind of problem, without lua? We have lot of problems with lua and want to get rid of it, so that's not an option.

[homiakos]

I support this.

the same grep plugin can work with k8s metadata: https://docs.fluentbit.io/manual/pipeline/filters/grep

In general, it’s strange to change the metadadata format and not support plugins working with the new format

[adrianmace]

This behaviour is necessary. There are so many custom lua scripts floating around doing the same thing.

[brudman]

I agree it would be great if this was supported! Another option for doing this is to use the nest filter described by liyuntao in https://github.com/fluent/fluent-bit/issues/422

[FILTER] Name nest Match kube.* Operation lift Nested_under kubernetes Add_prefix kubernetes.

[FILTER] Name record_modifier Match kube. Remove_key kubernetes.container Remove_key kubernetes.pod_id Remove_key kubernetes.docker_id

Then if you need the kubernetes object again:

[FILTER] Name nest Match kube. Operation nest Wildcard kubernetes. Nest_under kubernetes Remove_prefix kubernetes.