search

fabric8io / fluent-plugin-kubernetes_metadata_filter

Enrich your fluentd events with Kubernetes metadata
Apache License 2.0
350 stars 166 forks source link

Remove de_dot functionality from plugin #320

Closed jcantrill closed 2 years ago

jcantrill commented 2 years ago

This functionality is a hold over from the original implementation that did a log of tangential things. Remove this functionality and refer users to alternate plugins

aauren commented 2 years ago

@jcantrill It makes sense that de_dot functionality would not be part of this plugin by default.

In your message though you mentioned referring users to alternate plugins, do you happen to know of a replacement for this functionality? Did this already get documented somewhere?

jcantrill commented 2 years ago

In your message though you mentioned referring users to alternate plugins, do you happen to know of a replacement for this functionality? Did this already get documented somewhere?

I do not know of a specific plugin though one could use record_transformer https://github.com/repeatedly/fluent-plugin-record-modifier#ruby-code-trick-for-complex-logic

OlleLarsson commented 1 year ago

We recently upgraded to a version of this plugin were the de_dot feature has been removed and we encountered tons of mapping conflicts in elasticsearch/opensearch due to the removal.

I'm wondering how you have solved this, or you never encounter such mapping problems(I know it links to fluent-bit) for annotations or labels in your platform? We'll be downgrading until we know how to approach this problem of having dots in kubernetes metadata, as the plugin no longer handles it.

jcantrill commented 1 year ago

We recently upgraded to a version of this plugin were the de_dot feature has been removed and we encountered tons of mapping conflicts in elasticsearch/opensearch due to the removal.

You might either find a plugin, write a plugin, or use this technique https://github.com/repeatedly/fluent-plugin-record-modifier#ruby-code-trick-for-complex-logic

I'm wondering how you have solved this, or you never encounter such mapping problems(I know it links to fluent-bit) for annotations or labels in your platform? We'll be downgrading until we know how to approach this problem of having dots in kubernetes metadata, as the plugin no longer handles it.

Mapping collisions from my experience are either related to trying to push different data types to the same field which is not supported by Elasticsearch. There is a way around this by declaring fields not indexable but that may or may not be useful for you. We solved the type issue by modifying the es templates to be nested

The other issue described here is how Elasticsearch treats dotted keys: is it the field name or the path the the field. My recollection is there originally was a bug that was fixed (es2.x) and that later you could tell ES via a runtime switch how to treat dotted fields. It may have also been the nested template change also resolved this issue; I don't recall.

OlleLarsson commented 1 year ago

Hi, thanks for the reply. Looks like they have added a new feature, subobjects, in ES8 that allows for the behavior we are looking, unfortunately we are using Opensearch which does not have that feature. We'll have to look and see how we should proceed with this. I was just a bit surprised when we upgraded the plugin and discovered that the default behavior of replacing dots with _ had been removed ;) Have a nice weekend!