fabric8io / fluent-plugin-kubernetes_metadata_filter

Enrich your fluentd events with Kubernetes metadata
Apache License 2.0
350 stars 166 forks source link

feat: add kubernetes controller metadata #375

Closed aauren closed 11 months ago

aauren commented 11 months ago

This PR adds controller metadata to the list of available data that could be retrieved by the metadata filter. It is defaulted to false so that the default functionality of the plugin is not changed and people that do not want this information are not impacted by it.

When set to true, it produces an integer keyed map of the list of controllers that own / manage the pod. This information is available from the API server via the following:

kubectl get pod <foo> -o 'jsonpath={.metadata.ownerReferences}'

It will add something like the following to each record (flattened for readability):

kubernetes.controller.0.kind = ReplicaSet
kubernetes.controller.0.name = pong-672fb1239d

This information is useful for those wanting to get more direct references to the higher level objects in k8s like ReplicaSets, DaemonSets, Jobs, etc. from their logs without having to make / enforce strict labeling policies.

My Ruby is a bit rusty, so feel free to let me and I'll make any changes required. I have tested this and am running the plugin with this patch in a medium sized development cluster and it so far works fine.

alanconway commented 11 months ago

@aauren a couple thoughts that may or may not help.
The ES docs say "you cannot query each object independently of the other objects in the array."

Arrays are used in 2 cases:

  1. You have a list with a known order, you need to preserve that order.
  2. You don't care about order, but there is no unique key you can use to identify objects.

I think you are in case 2, and regardless of ES restrictions that means that: there is no meaningful way to query objects independently anyway. You cannot know the numbering in advance, so you can't formulate a query to get a desired value based on its number. You can only grab the whole array and then iterate to find the thing you want. Even after you have downloaded the array you can't use the indexes to make individual queries because the set of values, and therefore the numbering, may have changed since your last query.

So is it really a problem?

Cheers, Alan.

jcantrill commented 11 months ago

https://rubygems.org/gems/fluent-plugin-kubernetes_metadata_filter/versions/3.4.0

aauren commented 11 months ago

@jcantrill thanks for the prompt PR feedback and cutting the new release!