Open midestefanis opened 3 years ago
Hi @midestefanis , you can enrich your log with kubernetes metadata using the kubernetes
filter and use a placeholder in your log_group and log_stream name. Did you try following this guideline in our README? Hope it will give some guidance.
Hi @hossain-rayhan thanks for your answer. Is the kubernetes filter working in this version of fluent-bit? And yes, i have looked into that readme but it doesn't say anything about eks, only ecs.
@midestefanis The tutorial shows you how to generically use keys in the logs to template the stream/group names. It works with any keys.
If you have a key in the logs like kubernetes.pod_name
you could make the log stream name template be:
log_stream_name from-fluent-bit-$(kubernetes['pod_name']
Hi @PettitWesley, yes, but to do that the application has to log in with that key. In a large company you cannot make all the applications that want to mount on your infra to have to refactor their logs. The correct approach would be to be able to have the kubernetes metadata (with the kubernetes filter) and add it to it.
@midestefanis I think you are not understanding my suggestion.
You can do the following:
This experience works. We have many customers using it.
@PettitWesley It doesn't work on eks-fargate. I have already tried it. I found this -> https://github.com/aws/containers-roadmap/issues/1197
Oh I did not realize you are an EKS Fargate customer. In that case, yes, you must wait till that roadmap issue is implemented. We know that it is a high priority request that many customers need.
@PettitWesley I am having trouble with this on EKS too. I can see the metadata from kubernetes but the templating does not work for some reason.
My config:
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
DB /var/log/flb_kube.db
Parser docker
Docker_Mode On
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Merge_Log On
Merge_Log_Key data
Keep_Log On
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name cloudwatch
Match *
region eu-central-1
log_group_name eks/workloads
log_stream_name from-fluent-bit-$(kubernetes['namespace_name'])
log_stream_prefix fluentbit-$(kubernetes['pod_name'])
log_format json
log_retention_days 7
{ "kubernetes": { "annotations": { "kubernetes.io/psp": "eks.privileged" }, "container_hash": "quay.io/calico/node@sha256:bc4aa22272ef814ed4c3ce02b78fab8ddf446fceeef370482c4ebc4edabc06a1", "container_image": "quay.io/calico/node:v3.19.1", "container_name": "calico-node", "docker_id": "ce0887fc17decdafabc392ff973db5e5057bce7c0e581edea00a59f96179f1c9", "host": xxxxxl", "labels": { "app.kubernetes.io/name": "calico-node", "controller-revision-hash": "58665c8cf4", "pod-template-generation": "1" }, "namespace_name": "networking", "pod_id": "d446a251-abfa-4382-86f2-9e0150e1ef6e", "pod_name": "calico-node-qtjpk" }, "log": "2021-07-12 17:11:18.015 [INFO][39] felix/config_params.go 432: Parsing value for FelixHostname: xxxxxx (from environment variable)\n", "stream": "stdout", "time": "2021-07-12T17:11:18.015137585Z" }
@vinay-ux The problem might be here:
log_stream_name from-fluent-bit-$(kubernetes['namespace_name'])
log_stream_prefix fluentbit-$(kubernetes['pod_name'])
The templating works only with log_group_name
and log_stream_name
. Don't also use the prefix option, I think it takes precedence.
What is the final log stream name in CW?
@hossain-rayhan Can we add a validation so that both prefix and name are an invalid config.
@PettitWesley I think I understand what you meant. Just to be sure, log_stream_name
and log_stream_prefix
are either / or option. They cannot be put together and we will add a validation to stop such case. I am taking an action item.
@hossain-rayhan yeah they plugin should fail if they are used together IMO.
Right, this makes a lot of sense @PettitWesley , thank you very much :)
What is the final log stream name in CW? $(kubernetes['namespace_name'])-var.log.containers......
Dear @PettitWesley , I have a question about this issue.
I want to create log group for each namespace. But it's not working. I checked logs from fluentbit pod, InvalidParameterException occured.
Is there something I have configured incorrectly?
My config.
[INPUT]
Name tail
Path /var/log/containers/*.log
DB /var/log/flb_kube.db
Parser docker
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name cloudwatch_logs
Match kube.*
region ap-northeast-2
log_group_name /eks/dev/$(kubernetes['namespace_name'])
log_stream_name $(kubernetes['pod_name'])
auto_create_group true
Exception message
[2024/08/22 06:27:16] [error] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogStream API responded with error='InvalidParameterException', message='1 validation error detected: Value '/eks/dev/$(kubernetes['namespace_name'])' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\\.\\-_/#A-Za-z0-9]+'
Even though I have configured it in the form of $(kubernetes['namespace_name']) as referenced on the https://github.com/aws/amazon-cloudwatch-logs-for-fluent-bit#templating-log-group-and-stream-names page, it seems that the value is not being recognized. Is there anything else I can check?
¿Are you planning to add Kubernetes metadata to the logs?, eg:
[FILTER] Name kubernetes Match kube.* Merge_Log On Keep_Log Off K8S-Logging.Parser On K8S-Logging.Exclude On
Or how can I collect all the logs of a namespace and send to a specific log_group. Because with the default configuration all the logs go to the same log_group. Default configuration:
[OUTPUT] Name cloudwatch_logs Match * region us-east-1 log_group_name fluent-bit-cloudwatch log_stream_prefix from-fluent-bit- auto_create_group true
Without the k8s metadata in the logs and without being able to send the logs of a namespace to a particular log_group I cannot distinguish the different applications.