Support Amazon Linux?

[blopker]

Hey, I opened a issue here: https://github.com/fluent/fluent-bit-packaging/issues/1 but I guess it's not being monitored, so I'll ask here!

Are there any plans to support an Amazon Linux package?

Thanks!

[edsiper]

@blopker do you know if this Docker container replicate the same file system environment ?

https://hub.docker.com/_/amazonlinux/

[lebenitza]

See https://github.com/fluent/fluent-bit-packaging/issues/1

[jujugrrr]

@blopker do you know if this Docker container replicate the same file system environment ?

https://hub.docker.com/_/amazonlinux/

Hi @edsiper , yes it does, however it's a pruned version. If you want a container that matches a "real" AWS EC2 Amazon Linux (or amazon linux 2 more specifically) you can use:

# Dockerfile
FROM amazonlinux:2

RUN set -eux \
    && yum -y groupinstall AMI \
    && yum -y clean all \
    && rm -rf /var/cache/yum

CMD ["/usr/sbin/init"]

Amazon seems to push for fluent bit(https://aws.amazon.com/blogs/opensource/centralized-container-logging-fluent-bit/), so it would make sense to have non-container packaging support as well.

[PettitWesley]

@jujugrrr Right now, this is our recommendation for running on EC2 outside of a container: https://github.com/aws/aws-for-fluent-bit/tree/master/examples/fluent-bit/systems-manager-ec2

I've successful used those steps to run Fluent Bit on Amazon Linux 2. If you don't want to use SSM, you can run the commands shown in the SSM document; installing td-agent-bit via the cent OS packages worked for me.

[jujugrrr]

I've successful used those steps to run Fluent Bit on Amazon Linux 2. If you don't want to use SSM, you can run the commands shown in the SSM document; installing td-agent-bit via the cent OS packages worked for me.

Thanks @PettitWesley. I'll use this 😉

[shivshankarb]

@PettitWesley , I just added comment https://github.com/fluent/fluent-bit/issues/1427 I would like to check if the ES OUTPUT block will expected to work with tf-agent-bit? Or does it need an additional configuration?

[PettitWesley]

@shivshankarb Looks like your issue is resolved? :) https://github.com/fluent/fluent-bit/issues/1427#issuecomment-589731740

[shivshankarb]

yes thank you. td-agent-bit installation via the cent OS packages worked well for Amazon Linux 2.

[shivshankarb]

Hi @PettitWesley, While using HTTP authentication for elasticsearch on EC2 the td-agent-bit.conf file requires to set the HTTP_Passwd password string. Is there any way to encrypt the passwords? Can td-agent-bit retrieve this HTTP_Passwd variable value from the ssm parameter store?

[OUTPUT]
Name es
Match *
Host vpcXXXXX.es.amazonaws.com
Port 443
index ec2-test-index
Retry_limit 1
Type _doc
HTTP_user es-access-user
HTTP_Passwd ******

[PettitWesley]

Hey @shivshankarb unfortunately that's not possible today.

Can td-agent-bit retrieve this HTTP_Passwd variable value from the ssm parameter store

That's a good feature request for us :)

We (AWS) work in this repo. Please open an issue to track the feature request (if not, I will).

[shivshankarb]

Thanks for quick reply @PettitWesley. Can you please raise it for us? it will be great help if you manage to prioritize it as well. We are quite happy with it but can not push to test until we do something for the raw password. Regards, Shiv

[PettitWesley]

it will be great help if you manage to prioritize it as well

@shivshankarb Myself and my teammates are working hard to improve Fluent Bit, and in my opinion this feature request is a very good idea. Unfortunately there's a lot we have to work on, so it might be a few months. You'll have to find a workaround or alternate solution until then.

[shivshankarb]

Hi @PettitWesley, what option 'fluentd' for ec2 follow to retrieve secret on AMZ linux? I am just considering if its worth a try using 'fluentd' for EC2 instead of fluent-bit(via td-agent-bit)? Reagrds, Shiv

[PettitWesley]

@shivshankarb I don't think that Fluentd supports this feature either, as far as I know.

But I wonder- could you modify the SSM script to pull parameter values and set them as env vars on td-agent-bit?

https://github.com/aws/aws-for-fluent-bit/tree/master/examples/fluent-bit/systems-manager-ec2

[patrick-stephens]

I think the original request is satisfied here, can we close?