Open eli-gc opened 7 months ago
I think it's related to my kubernetes metadata labels. I have two labels: app and app.kubernetes.io/name. I believe it is being rejected because one is a text type and the other is a nested object so Elasticsearch doesn't know how to handle it. Was there a change in how types or dots are handled past 1.15.0? I didn't see anything in the change log. It works just by rolling back to 1.15.0 from 1.15.1+ so I know it isn't Elasticsearch version.
I think it's related to my kubernetes metadata labels. I have two labels: app and app.kubernetes.io/name. I believe it is being rejected because one is a text type and the other is a nested object so Elasticsearch doesn't know how to handle it. Was there a change in how types or dots are handled past 1.15.0?
This is what the root cause of this issue. For handling this, you need to install ES template to define the field type. see: https://www.elastic.co/guide/en/elasticsearch/reference/8.11/mapping.html see also: https://www.elastic.co/jp/blog/antidote-index-mapping-exceptions-ignore_malformed
Same Issue for me, it's prevent me to upgrade to latest version, it used to work and just stopped
@cosmo0920 Is ES template a plugin? Or are you saying I need to make a template myself?
You guys need to create and install Elasticsearch mappings by yourself. Auto mapping sometimes causes mistypes on their handled documents.
Thanks, I'll give it a shot and report back.
I wasn't able to get the mapping to work. It says app cannot be changed from text to ObjectMapper.
PUT /mapping-test-index
{
"mappings": {
"properties": {
"app": {
"type": "text"
},
"app.kubernetes.io/name": {
"type": "text"
}
}
}
}
@eli-gc Did you find a working configuration ? I've been experiencing the same conflict issue with an "app" string label
@xdubois I did not. We decided to move away from Fluentd, but you could try adding the de_dot filter manually or possibly use flattened. De_dot got removed from Fluentd which was the root of my issue. Check out these issues for more info: de_dot removal elasticsearch#63530
Not sure the one of the solution candidates but Fluentd has dedot filter plugin: https://github.com/lunardial/fluent-plugin-dedot_filter This should replace dot(.) with a specified character.
Thanks for responses guys Couldn't make it work with the dedot plugin We switched to filebeat for the ease of configuration
I have the same issue with the logging-operator
version 1.6.0
, which uses FluentD with this plugin. I get error below:
"reason"=>"[1:1018] failed to parse field [kubernetes.labels.app] of type [keyword] in document with id 'piLbx44BBp6m9YwiO00W'. Preview of field's value: '{kubernetes={io/component=controller}}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:988"}
Even with dedot
filter it doesn't seem to work.
<filter **>
@type dedot
@id clusterflow:logging:nginx:2
de_dot_separator _
</filter>
failed to parse field [kubernetes.labels.app] of type [keyword] in document
This is because the pointed field is not keyword i.e. just a within 256 length text. Depending on the automatic mapping caused this issue.
@cosmo0920 Well, filebeat, which was used before on our deployment, provides label field as kubernetes.labels.app_kubernetes_io/component
with its value controller
, however fluent-bit/fluentd based on that log above messed up and the result is kubernetes.labels.app
with its value kubernetes={io/component=controller}
.
The original format in Kubernetes looks like: app.kubernetes.io/component: controller
Of course, I can tweak mapping on the Elasticsearch side, but that won't solve weird parsing.
Hmm.., it's weird. Just for my curiosity, isn't it solved by using fluent-bit instead of Fluentd with this plugin?
@cosmo0920 I deleted mapping for my index to allow all data to come in.
And I noticed that the problematic label displayed in Kibana as kubernetes.labels.app.kubernetes.io/component
with its value controller
.
So, I guess we can work with that and rewrite our mapping.
However, I am still wondering about that log I posted, as the error message is quite confusing because, based on that, the value should look like kubernetes={io/component=controller}
.
The error message came from Elasticsearch itself. So, we couldn't display more clearly unfortunately.
Recently I also have same problem,but I solved it. I use the fluentd filter plugin,and the configuration information is as follows:
<filter kubernetes.**>
@type record_transformer
remove_keys $.docker.container_id,$.kubernetes.container_image_id,$.kubernetes.pod_id,$.kubernetes.namespace_id,$.kubernetes.master_url,$.kubernetes.labels.pod-template-hash,$['kubernetes']['labels']['app.kubernetes.io/instance'],$['kubernetes']['labels']['app.kubernetes.io/managed-by'],$['kubernetes']['labels']['app.kubernetes.io/version'],$['kubernetes']['labels']['app.kubernetes.io/name'],$['kubernetes']['labels']['app.kubernetes.io/component'],$['kubernetes']['labels']['app.kubernetes.io/part-of']
</filter>
Then,I upgrade fluentd helm package.I hope I can help you. fluentd filter plugin
@xiaojun90713 so you just deleted the keys? What if I need the data from them?
@xiaojun90713 so you just deleted the keys? What if I need the data from them? The logs data can still be collected normally, but I don’t need the labels for these logs. In one other hand, I have other label for these logs collect, it does not affect my collection of logs. As a k8s log collector, I already have enough labels to query logs. For me, these remove_keys are useless. Of course, if you need to keep these labels, you can replace them with others through the filters plugin. I don’t know if you understand what I mean.
@xiaojun90713 Thanks for the explanation. I suppose I could try to remove one of the labels that are colliding for me. That might work for me. However, your approach is removing the problem rather than a solution. The caveat is It only works if those labels are not required.
(check apply)
Problem
I cannot upgrade past 1.15.1 or else I get this error. There is no error in 1.15.0. I did not see any breaking changes in the release notes of fluentd 1.15.1
#0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: document_parsing_exception [reason]: '[1:660] failed to parse field [kubernetes.labels.app] of type [text] in document with id
Steps to replicate
Either clone and modify https://gist.github.com/pitr/9a518e840db58f435911
OR
Provide example config and message
Expected Behavior or What you need to ask
Using Fluentd and ES plugin versions