aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.18k stars 313 forks source link

Does EKS Fargate built-in logging support splunk? #1318

Open vaibhavgupta3007 opened 3 years ago

vaibhavgupta3007 commented 3 years ago

I understand that Fargate can push logs to cloudwatch, ES and Kinesis, but i am looking for solution if we can use same configuration to push logs to splunk using Fragate built-in Support.

mhausenblas commented 3 years ago

Thanks for your question and the status is as we described it in https://aws.amazon.com/blogs/containers/fluent-bit-for-amazon-eks-on-aws-fargate-is-here/ that is it's possible and we're working on a direct integration.

vaibhavgupta3007 commented 2 years ago

@mhausenblas , @kiranmeduri , @mndoci, @joebowbeer , @mogren : Do you have any update on splunk direct integration? I am trying to set up fluent bit using link https://aws.amazon.com/blogs/containers/fluent-bit-for-amazon-eks-on-aws-fargate-is-here/ to push logs on cloudwatch for now, but seems like it is also not working and does not support kubernetes filter.

here is my configmap file:

kind: ConfigMap apiVersion: v1 metadata: name: aws-logging namespace: aws-observability data: output.conf: | [OUTPUT] Name cloudwatch_logs Match * region us-west-2 log_group_name eks/fluent-bit-cloudwatch log_stream_prefix from-fluent-bit- auto_create_group true log_key log

filters.conf: | [FILTER] Name kubernetes Match kube.* Merge_Log On Buffer_Size 0 Kube_Meta_Cache_TTL 300s K8S-Logging.Parser On ~

somehow logs are not getting kubernetes meta data. and its printing in text format instead json.

ex:

2022-02-15T23:21:44.793733091Z stdout F 2022-02-15 15:21:44.792 WARN 1 --- [ main] d.s.r.o.OperationImplicitParameterReader : Unable to interpret the implicit parameter configuration with dataType: string, dataTypeClass: class java.lang.Void 2022-02-15T23:21:44.793733091Z stdout F 2022-02-15 15:21:44.792 WARN 1 --- [ main] d.s.r.o.OperationImplicitParameterReader : Unable to interpret the implicit parameter configuration with dataType: string, dataTypeClass: class java.lang.Void

please let me know if i have missed something.