[EKS/Fargate] [Logging]: No logs with log group/stream templates

[kncesarini]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request This is my current (and working) EKS fargate logging CM:

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  flb_log_cw: "true"
  filters.conf: |
    [FILTER]
        Name parser
        Match *
        Key_name log
        Parser crio
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        Labels Off
        Annotations Off
        Buffer_Size 0
        Kube_Meta_Cache_TTL 300s
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *
        region eu-north-1
        log_group_name my-logs
        log_stream_prefix from-fluent-bit-
        log_retention_days 60
        auto_create_group true
  parsers.conf: |
    [PARSER]
        Name crio
        Format Regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

Following the docs at https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch I tried adding the the following configuration to the output in order to separate the logs by namespace:

log_group_template eks-logs-$kubernetes['namespace_name']
log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']

However when I do, no logs appear anymore in CloudWatch (adding either of these configs on their own causes the issue). The "default" log group no longer gets any logs, and no new log group is created. Additionally I don't get any logs from the flb agent. The describe pod command however says they are enabled after restarting a deployment:

Events:
  Type     Reason          Age                From               Message
  ----     ------          ----               ----               -------
  Normal   LoggingEnabled  96s                fargate-scheduler  Successfully enabled logging for pod

Which service(s) is this request for? EKS Fargate Logging

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? See above

Are you currently working around this issue? Currently forced to send all logs in the cluster to the same log group, and with less readable log stream names.

Additional context Kubernetes v1.23

Attachments n/a

[sergeisantoyo]

I'm facing this issue in my cluster as well. I'm using Kubernetes v1.21.

Changing the log_group_name works, but as soon as I add log_group_template or log_stream_template to my config, I stop getting logs.

[tri-bao]

I need this feature too because the default generated log stream is way too long If a pod has several containers, it will be very hard to know which log stream is for what container unless open the stream detail.

I want the C plugin (cloudwatch_logs) in stead of the GoLang (cloudwatch) one to achieve a high throughput. However, if seems to me that EKS platform eks.2 with kubernets 1.24 still uses an old version of the C plugin (version 1.5) instead of the current version which support log_stream_template https://github.com/fluent/fluent-bit/blob/df73b2200969006a36e116279312e1142612801d/plugins/out_cloudwatch_logs/cloudwatch_logs.c#L130

Is there any way for us to specify the plugin version via the config map aws-logging?

[liampetti]

Any update on this issue?

Using log_group_template doesn't seem to have any effect for me with logs just defaulting to the log_group_name using Kubernetes 1.24

it is very hard to track all logs in a single group when running lots of pods..

[Kartoshnikov]

Any updates here? It seems like the EKS on Fargate built-in log router runs Fluent Bit with the version below 1.4.0 because it doesn't support templating in log_stream_name, e.g. fargate-$(tag). It's a shame, really(