Failed to use EKS Fargate Logging

[andrewkwok2022]

Describe the question/issue

I tried to enable logging for my EKS Fargate but in vain. I would like to know what I have done wrong for the configuration:

I have followed the these 2 guides (https://aws.amazon.com/blogs/containers/fluent-bit-for-amazon-eks-on-aws-fargate-is-here/ https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html ) to set up the namespace, configmap and iam policy. When I “kubectl describe” the pod, I can see “Logging: LoggingEnabled” in Annotations and “Successfully enabled logging for pod” from “fargate-scheduler” in Events. However, after applying all the required config and yaml files, I have not seen anything related to my EKS cluster in CloudWatch Logs.

Here is my logging-config.yaml

kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-logging
  namespace: aws-observability
data:
  output.conf: |
    [OUTPUT]
      Name cloudwatch_logs
      Match *
      region us-west-2
      log_group_name fluent-bit-cloudwatch
      log_stream_prefix from-fluent-bit-
      auto_create_group true
      log_key log

Apart from that, as the server logs of my Jboss application are stored in the directory /usr/local/jboss/standalone/log, and I figure out that if nodegroup is in used, Fluent Bit provides input config to specify the location to “save” the log (ref: https://aws.amazon.com/blogs/containers/capturing-logs-at-scale-with-fluent-bit-and-amazon-eks/ )

Example:

       [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        DB                /var/log/flb_kube.db
        Parser            docker
        Docker_Mode       On
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10

How to do the same thing in EKS Fargate, so that I can collect all the log messages from my application inside a specific folder?

Thank you!

Cluster Details

EKS Fargate

[andrewkwok2022]

Finally got the CloudWatch Log Events after creating VPC Endpoints for com.amazonaws.us-west-2.logs, but still want to know how to configure "INPUT" in Fargate.

[bayyuaji]

We can not create section INPUT on AWS EKS Fargate. AWS only ACCEPT FILTER, PARSER, & OUTPUT sections as mentioned in this https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html