search

kiwigrid / helm-charts

Helm charts for Kubernetes curated by Kiwigrid
https://kiwigrid.github.io
MIT License
186 stars 209 forks source link

[fluentd-elasticsearch] Ability to set a cluster identifier #342

Closed rgarcia89 closed 4 years ago

rgarcia89 commented 4 years ago

FEATURE REQUEST:

Is it somehow possible currently to set a cluster ID? In my case we have multiple k8s clusters. I would like to set a label or similar which should be added to the log metadata. This way I could filter for log entries of a specific cluster in case the app is spread over multiple clusters.

monotek commented 4 years ago

how about using the nodename? don't know where else you want to get the cluster name from if you don't set it by some label manually.

rgarcia89 commented 4 years ago

I would definitely like to do it using a label. However, I have to admit that I have no clue currently how to do that. I mean I know how to add labels to the k8s nodes. But how will they make it to elastic? So that they are attached to every log entry

monotek commented 4 years ago

imho node labels are ignored by https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter.

so you would have to add that label to all of your pods at the moment.

rgarcia89 commented 4 years ago

oh I see...Is there some other way to inject it somehow. From what I can see fluentd is not passing all pod labels to elastic. Is there a special prefix that need to be in the label-name?

monotek commented 4 years ago

to be honest i can't try it out at the moment as we're using loki and stackdriver for all our clusters now.

as far as i remember there was no need for special label naming but short living pods may be a usecase where labels are missed out.

namesapce_labels should be logged too so storing the clustername there should be easier as addign it to all pods.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.