emit transaction failed: error_class=IndexError error="undefined group name reference: prefix"

[jmervine]

Seeing the following

2022-01-03 21:14:32 +0000 [warn]: #0 emit transaction failed: error_class=IndexError error="undefined group name reference: prefix" location="/var/lib/gems/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.3/lib/fluent/plugin/filter_kubernetes_metadata.rb:347:in `[]'" tag="containers.var.log.containers.gravity-site-6vnwv_kube-system_gravity-site-bc73b0a91349641db32ae703ae1b19f25b71c83843e7157aea0abe00560251a4.log"
  2022-01-03 21:14:32 +0000 [warn]: #0 /var/lib/gems/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.3/lib/fluent/plugin/filter_kubernetes_metadata.rb:347:in `[]'
  2022-01-03 21:14:32 +0000 [warn]: #0 /var/lib/gems/2.7.0/gems/fluent-plugin-kubernetes_metadata_filter-2.9.3/lib/fluent/plugin/filter_kubernetes_metadata.rb:347:in `block in filter_stream'

It appears to be related to a (somewhat) hidden change in version 2.9.3, which oddly has no tag. If you look at: https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/compare/v2.9.2..798febb2c079ecce84348ee593ae6e77b30d1be0#diff-0b4b9044f59b6776e7937e8f714cb59db8858c728efb76dd18456f026f423f19R347

Note that sha comes from the PR marked for v2.9.3 release.

I'm not sure if this is intentional and I'm missing something on my end, but figured it was worth mentioning.

[tangzhankun]

@jmervine Thanks for pointing it out. I met this strange error today when doing testing. I'm now using the same v2.9.3 version plugin against a new Kubernetes v1.21.

Not pretty sure if this is caused by the K8s version or the plugin version. IIRC, the same plugin version with the same config file worked previously.

[jcantrill]

Please post your configuration. My expectation would this should only be an issue if you define a custom value for tag_to_kubernetes_name_regexp which could be better documented

[jcantrill]

Labeling a bug as we can resolve to not require a prefix in the regex

[tangzhankun]

@jcantrill @jmervine This is my configurations for your reference:

      <filter event.pod.**.*>
        @type kubernetes_metadata
        # Use a custom pattern to extract the pod metadata for Kubernetes API lookup.
        # The container_name and docker_id are used as a caching key for pod metadata
        # so we populate them with the pod UID. Note: pod name may contain '.' character
        tag_to_kubernetes_name_regexp 'event\.pod.[^\.]+\.(?<namespace>[^_]+)_(?<pod_name>[^_]+)_(?<container_name>[^_]+)_(?<docker_id>[^_]+)$'
        skip_container_metadata       true
        skip_namespace_metadata       true
        annotation_match [
          "execution_date"
        ]
      </filter>
# Events now have a new "kubernetes" field with the pod labels and annotations

I took a look at the PR but is not very clear of the root cause. Actually, I didn't quite understand how the "tag_to_kubernetes_name_regex" really works. Could you please share your suggestion to avoid such an IndexError error? Use the old version like v2.9.2 or change the value of tag_to_kubernetes_name_regexp?

[jcantrill]

https://rubygems.org/gems/fluent-plugin-kubernetes_metadata_filter/versions/2.9.4

[jcantrill]

@jcantrill @jmervine This is my configurations for your reference:

      <filter event.pod.**.*>
        @type kubernetes_metadata
        # Use a custom pattern to extract the pod metadata for Kubernetes API lookup.
        # The container_name and docker_id are used as a caching key for pod metadata
        # so we populate them with the pod UID. Note: pod name may contain '.' character
        tag_to_kubernetes_name_regexp 'event\.pod.[^\.]+\.(?<namespace>[^_]+)_(?<pod_name>[^_]+)_(?<container_name>[^_]+)_(?<docker_id>[^_]+)$'
        skip_container_metadata       true
        skip_namespace_metadata       true
        annotation_match [
          "execution_date"
        ]
      </filter>
# Events now have a new "kubernetes" field with the pod labels and annotations

I took a look at the PR but is not very clear of the root cause. Actually, I didn't quite understand how the "tag_to_kubernetes_name_regex" really works. Could you please share your suggestion to avoid such an IndexError error? Use the old version like v2.9.2 or change the value of tag_to_kubernetes_name_regexp?

This configuration is only required if the running collector is reading logs from a non-standard kubernetes location, something other then /var/log/containers or /var/log/pods. This configuration allows the plugin to query the API server to retrieve meta data which requires: namespace, podname. If you mount the default locations into your container then you can remove this config.