Open 123BLiN opened 3 years ago
Actually after some investigation, correct test should be:
test "lables_test" do
config = %[
format cluster.${namespace_name}.${labels.app.kubernetes.io/name}
]
record = {
"log" => "Example",
"kubernetes" => {
"pod_name" => "understood-butterfly-nginx-logging-demo-7dcdcfdcd7-h7p9n",
"namespace_name" => "default",
"labels" => {
"app.kubernetes.io/name" => "traefik",
"app.kubernetes.io/managed-by" => "helm"
}
}
}
d = create_driver(config)
d.run(default_tag: 'test') do
d.feed("tag1", event_time, record.dup)
d.feed("tag1", event_time, record.dup)
end
events = d.events
puts events
end
And it is failed - got unknown instead actual label value
{"log"=>"Example", "kubernetes"=>{"pod_name"=>"understood-butterfly-nginx-logging-demo-7dcdcfdcd7-h7p9n", "namespace_name"=>"default", "labels"=>{"app.kubernetes.io/name"=>"traefik", "app.kubernetes.io/managed-by"=>"helm"}}}
cluster.default.unknown
Any update on this issue ? I also need to access app.kubernetes.io/name
label from tag normalizer.
Is there any Update, I have the same issue
2021-12-14 11:08:47 +0000 [warn]: #0 send an error event to @ERROR: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'Could not dynamically add mapping for field [app.kubernetes.io/instance]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text].'" location=nil tag="kubernetes.var.log.containers.longhorn-manager-sth9b_longhorn-system_longhorn-manager-20de6ff1a3842b0757d557d9bef44183785eca967bbda35814005c0d763c6a1b.log" time=2021-12-14 11:05:20.007901338 +0000
experiencing the same problem here. Any news?
I also encountered this problem, but I finally adopted this solution
# ClusterFlow:
spec:
filters:
- record_modifier:
records:
- kube_app_name: ${record.dig('kubernetes', 'labels', 'app.kubernetes.io/name') || "unknown"}
# ClusterOutput
spec:
elasticsearch:
buffer:
tags: tag,time,kube_app_name,$.kubernetes.namespace_name
index_name: log.${$.kubernetes.namespace_name}.${kube_app_name}.%Y%m%d
This issue is resolved by https://github.com/kube-logging/fluent-plugin-tag-normaliser/pull/5
Is your feature request related to a problem? Please describe. I'm using logging operator with tag_normalizer and trying to route logs to different idexes with https://github.com/uken/fluent-plugin-elasticsearch#dynamic-configuration. Somehow I'm not able to use
${kubernetes.labels.app-kubernetes-io/name}
as a placeholder for index name pattern but${tag} works, so I've decided to try to adjust tag itself however same problem here - I'm not able to use
${labels.app-kubernetes-io/name}` as a value in tag - it is always empty (unknown)Describe the solution you'd like to see Maybe I need to use some special syntax I'm not aware about
Describe alternatives you've considered Tried to set index name with placeholders but it does not work either
Additional context When I'm trying to run tests from this repo but with another set of labels - it works, but it does not work in the real life fluentd :(
result: