[EKS] [request]: EKS Fargate Support for more Fluent Bit output plugins

[Byeong-jun-Kim]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request I want to use logging to fluentd(forward output) or logstash(http output)

Which service(s) is this request for? EKS on Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Only CloudWatch, ES, Kinesis are available now as basic EKS Fargate fluent bit functions.

Are you currently working around this issue? I added a fluent-bit container in every pods.

Additional context We hope that the general http & forward output function will be implemented as soon as possible in order for EKS Fargate to become a more practical service.

Attachments

[mhausenblas]

Thanks for raising this issue @Byeong-jun-Kim. Could you please expand a bit on what you'd like to see beyond what we described in Fluent Bit for Amazon EKS on AWS Fargate is here?

[Byeong-jun-Kim]

Thank you for your reply. The document says: 'You can choose between CloudWatch, ElasticSearch, Kinesis Firehose and Kinesis Streams as outputs.' I feel that the output support above is insufficient. Many Kubernetes users collect and process logs from fluent bit to fluentd, and send them to various places. Besides that, it would be great to have Forward Output which sent to the same stack, fluentd, and also http Output that can be used in general. Is my request a little clearer? @mhausenblas

[pmbeaumier]

@mhausenblas Adding another example here. In our case the HTTP output mentioned by @Byeong-jun-Kim would be great to have as we're using SumoLogic for our log system and this is the recommended way to send logs to SumoLogic. The Fluentbit documentation actually provides an example. Currently we have to go through a cumbersome configuration with Cloudwatch in-between. Thanks

[Dametrain]

To add to this I would like to see all native Fluentbit outputs as documented here: https://docs.fluentbit.io/manual/v/1.5/pipeline/outputs/. To start Datadog and Splunk would be my top two. As a work around you can use Kinesis Firehose, however that just adds cost and complexity.

[prashilgupta]

+1

https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html Built-in EKS Fargate logging places some restrictions on the configurations you can set. The output plugins are validated and only these: firehose, kinesis_firehose, cloudwatch, cloudwatch_logs, and kinesis plugins are supported (others are rejected as the configmap is applied. We want the HTTP OUTPUT plugin to be enabled at least so we can provide another EKS pod to consume the same. Hopefully, this ticket will help prioritize the same!

[amitkatyal]

+1

[lennartt]

It would be ideal if we can send the logs to a HTTP output to Logstash.

[philomory]

Yeah, this limitation basically prevents us from using Fargate's logging functionality at all, until the GELF output is supported.

I'm curious, @Byeong-jun-Kim, how did adding a fluent-bit container to every pod work out for you? It seems to me like you wouldn't be able to have that pod access the "node"'s filesystem to read the logs. Or are you just have your pods log to a file on a e.g. emptyDir volume so that it can be read by the fluent-bit pod?

[Byeong-jun-Kim]

@philomory yes you're right We are processing the log with a fluent-bit by redirecting all the logs as a file. Since the storage space is limited, we designated a specific lifecycle by saving it by date. But it's very tricky and inefficient. That's why I made this request.

[fishpen0]

Grafana shops would be ecstatic for loki support.

[alexku7]

please, add support the kubernetes parser which is intended to enrich the logs with useful information like the namespace, container names, pod name and etc ... https://docs.fluentbit.io/manual/pipeline/filters/kubernetes

[sburns]

Would like to see the DataDog output supported as well. Currently can work around by Cloudwatch -> DataDog forwarder -> DataDog, but that's money I'd just rather not spend.

[FelipeLujan]

Greetings.

At our company, we use Coralogix for log storage and analysis.

Our logs get centralized in a Kafka cluster (confluent) and then consumed by an app that formats and forwards them to Coralogix.

The following config

    [OUTPUT]
        Name                                  kafka
        Match                                 *
        Brokers                               {{ .Values.logging.kafkaEndpoint }}:{{ .Values.logging.kafkaPort }}
        Topics                                test-topic
        Topic_Key                             topic
        Format                                json
        rdkafka.security.protocol             SASL_SSL
        rdkafka.sasl.mechanism                PLAIN
        rdkafka.sasl.username                 xxxx
        rdkafka.sasl.password                 xxxx
        rdkafka.queue.buffering.max.messages  500000

causes the following error "0500-amazon-eks-fargate-configmaps-admission.amazonaws.com" denied the request: kafka is not a supported output plugin. Please fix the logging configmap

Please add support for Kafka outputs

[zquintana]

We too would really appreciate the Datadog and Http plugins, is that on the roadmap?

[pitstop-sirish-bajpai]

+1

loki plugin support on EKS fargate fluentbit

[tloriato]

~I'm looking forward to this issue but never participated on this forum before. Is there anywhere with the governance rules and possible timelines on how improvements are chosen? Is there anything I can do to help push this more?~

~Especially to Loki as a tail-end!~

Update: We moved everything out of Fargate into Immutable OS

[marcusdb]

+1

loki plugin support on EKS fargate

[kintoandar]

Could we please have Loki support for Fargate logs?

Please 🙏

[DevSusu]

@mhausenblas there's quite a reaction in this thread, would you be willing to take this issue further? Any kind of response will be appreciated, it's been over an year after you commented.

[rielas]

S3 please 🙏

[eyalle]

Splunk please! 🙏

[teochenglim]

Oh dear, we are using ec2 node and considering switching to Fargate, however the current support of fluent-bit is limited and almost useless.

1. It doesn't support s3, so the log is not long term and cheap? Or we need to manually export from Cloudwatch to s3? That means we can only use Fargate as toy and not production ready?

2. For almost all of us have our existing log system, that means we need additional label or minimum mutation label on our log in a multiple environment settings.

3. Since everyone would likely need the Kubernetes metadata parser, wouldn't it be nice if default already parse? yet customer has minimum control?

4. What choice do we have now? Maybe i should plugin a fluentd side-car (500M binary) on my golang microservice (20M binary) just to support this limitation.

[ianrodrigues]

No more updates on this one? HTTP output would be great.

[singhnix]

Hello,

One of my customer have below requirements as well. Kindly add this to the feature request as well:

Currently , Fargate provides a hook for logging with fluentbit: https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html .

Problem: AWS only exposes very specific plugins. https://github.com/aws/containers-roadmap/issues/1242 .

We need the fluent forward output plugin in order to get fluentbit fargate logs to Grafana Enterprise Logs (GEL).Fargate validates against the following supported output: es, firehose, kinesis_firehose, cloudwatch, cloudwatch_logs, and kinesis.

Request to AWS is to add “Fluent forward output plugin” to the above list.

Also, kindly advise if any workaround is there for the same.

[junowong0114]

Kudos to @Byeong-jun-Kim. Adding the "http" or "forward" output can tremendously improve the logging experience on EKS Fargate.

My organization has an existing ECS/lambda -> Fluentd on ECS -> on-premise Elasticsearch architecture for some reasons.

1. Fluentd can congregate logs into batch requests before sending them to Elasticsearch, which greatly reduces network/CPU loading.
2. We can enforce centralized data enrichment or manipulation rules. We can also do that by Elasticsearch ingest pipeline, but then we would lose the other advantages.
3. Fluentd provides an extra layer of buffer between microservices and Elasticsearch so that in the unfortunate event of a network/Elasticsearch outage, the application logs are properly handled by Fluentd instead of drowning the microservices' resources.
4. There is of course the cost-saving factor as we managed our on-premise Elasticsearch cluster.

All of these are not possible because the "forward" and "http" output filters are disabled. Now I change the code to log to both stdout and to a log file, mount that file to Fluent Bit sidecar, and add another sidecar to perform log rotation... It is a nightmare and is seriously hindering the adoption of AWS EKS on Fargate within my organization.

Please fix this, or at least let us know if "forward" and "http" output filter will be considered to be added.

[rafilkmp3]

Guys do you think will be possible to configure my promtail to scrap logs from fargate namespace?

[jordanbcooper]

Still looking for Loki ouput support!

[sameeraksc]

datadog please

[VeresMarcell]

+1

Loki support would be great

[teimor]

any HTTP support would be great

[Rudya93]

+1

Loki support would be great

[gespi1]

What is the status of this? There has not been any word from the official maintainers in awhile, almost abandoned.

Looking for Loki plugin support and any kind of update. Ty

[Rudya93]

+1

[trylix]

+1

Loki support would be great

[codesenju]

• 1

[prein]

At our company, we use Coralogix for log storage and analysis.

Did you try using elasticsearch output to ship directly to Coralogix? Did it work? If yes, would you mind sharing how to? Thanks!

[sonykphilip]

+1 for Loki

[smitb]

+1 for Loki

[asychev]

+1 for Loki

[chas0amx]

I attempted to forward logs from Karpenter, running on an AWS Fargate node, to fargate-eks-sidecar-injector or any other destination and encountered the following error:

panic: open sink "https://aos-xxx.us-east-1.es.amazonaws.com/": no sink found for scheme "https" goroutine 1 [running]: github.com/samber/lo.must({0x25adc00, 0xc00090c160}, {0x0, 0x0, 0x0})

I observed that the logger seems to have functionality related to parsing URLs.

Does anyone know the correct solution in this case or how to correctly forward logs to an external endpoint (rather than to the filesystem)?

[billnbell2]

Sematext would be good.

Otis Gospodnetic otis@sematext.com

[hjweddie]

+1 for Loki

[ojohn]

Another one for Loki

[jeffreydlcheung]

+1 for supporting "Forward" and also other output listed fluentbit manual

[sharraj]

Many users asked for this "HTTP support would be great". Appreciate if this can be prioritized as it will open up ways to connect to way more end points.

[Mkori1]

Forward output plugin would be great.