search

aws / aws-for-fluent-bit

The source of the amazon/aws-for-fluent-bit container image
Apache License 2.0
456 stars 135 forks source link

After deploying via Helm using EKS blueprints, I'm unable to see the '/dataplane', '/application', and '/host' log streams #724

Open alexo1088 opened 1 year ago

alexo1088 commented 1 year ago
### Describe the question/issue Hi folks. I've been deploying the 'aws-for-fluentbit' chart using the [EKS Blueprints for add-ons](https://aws-ia.github.io/terraform-aws-eks-blueprints/v4.32.1/add-ons/aws-for-fluent-bit/). Unfortunately, I'm noticing discrepancies between what the helm chart deploys, and what the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html) has you deploy. The biggest issue is that the helm chart method does not create the '/dataplane', '/application', and '/host' log groups. Instead, it creates individual log streams per workload. This seems like a fine replacement for the '/application' stream, but with no '/dataplane', we're losing the ability to obtain kubelet logs. I'm wondering how I can obtain kubelet logs using the helm deployment method. While I've created a ticket within the github repo for the EKS blueprints, I have yet to get a response. I'm hoping this is a simple case of just needing to add additional config parameters into IaC in order to get the desired results. Any suggestions or guidance would be appreciated! ### Configuration My configuration for the deployment of the helm chart is basic. Ideally, i'd like to incorporate whatever is needed into Terraform to get this working: ``` enable_aws_for_fluentbit = true aws_for_fluentbit_helm_config = { version = '0.1.28' ### Visit https://artifacthub.io/packages/helm/aws/aws-for-fluent-bit to see latest chart version available when deploying } ``` The helm chart itself is using this image: `│ spec: │ │ containers: │ │ - image: public.ecr.aws/aws-observability/aws-for-fluent-bit:2.31.11 ` ### Fluent Bit Log Output I am seeing some errors in the logs, but nothing that specifically indicates that it's failing to generate the log streams, so this seems more like something missing from the configuration than anything else: ``` │ -nmtrg [2023/08/22 17:02:05] [ info] [engine] flush chunk '1-1692723718.513621628.flb' succeeded at retry 1: task_id=0, input=tail.0 > output= │oudwatch_logs.0 (out │d=0) │ │ -nmtrg [2023/08/22 17:02:58] [error] [http_client] broken connection to logs.us-east-1.amazonaws.com:443 ? │ │ │ │ -nmtrg [2023/08/22 17:02:58] [error] [http_client] broken connection to logs.us-east-1.amazonaws.com:443 ? │ │ │ │ -nmtrg [2023/08/22 17:02:58] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send log events │ │ │ │ -nmtrg [2023/08/22 17:02:58] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send log events │ │ │ │ -nmtrg [2023/08/22 17:02:58] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events │ │ │ │ -nmtrg [2023/08/22 17:02:58] [ warn] [engine] failed to flush chunk '1-1692723778.286784402.flb', retry in 8 seconds: task_id=0, input=tail.0 │output=cloudwatch_lo │.0 (out_id=0) │ │ -tv9ft [2023/08/22 17:03:01] [error] [http_client] broken connection to logs.us-east-1.amazonaws.com:443 ? │ │ │ │ -tv9ft [2023/08/22 17:03:01] [error] [http_client] broken connection to logs.us-east-1.amazonaws.com:443 ? │ │ │ │ -tv9ft [2023/08/22 17:03:01] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send log events │ │ │ │ -tv9ft [2023/08/22 17:03:01] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send log events │ │ │ │ -tv9ft [2023/08/22 17:03:01] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events │ │ │ │ -tv9ft [2023/08/22 17:03:01] [ warn] [engine] failed to flush chunk '1-1692723781.147657826.flb', retry in 8 seconds: task_id=0, input=tail.0 │output=cloudwatch_lo │.0 (out_id=0) │ │ ``` ### Fluent Bit Version Info I've tried different chart versions, as well as the last three versions of the image ### Cluster Details

This is for EKS running EC2 EKS nodes. This is a private cluster with nodes running on private EC2 subnets

Application Details

Steps to reproduce issue

To reproduce, you can attempt to deploy the addon via the chart, but I don't think this is an issue with the EKS blueprint addons, because this seems to be deploying a standard aws-for-fluentbit image. What i'm looking for is guidance on how to add the mentioned log streams into IaC so that they can be deployed upon initial deployment.

Related Issues

PettitWesley commented 1 year ago

Hello!

I think there is a confusion here between the helm chart and the CloudWatch Container Insights Daemonset.

Helm Chart currently only supports pod logs

The helm chart by default only collects pod logs, and it doesn't have config options to enable anything else: https://github.com/aws/eks-charts/tree/master/stable/aws-for-fluent-bit

Container Insights Daemonset collects dataplane, application (pod), and host logs