How to use "process _credentials"?

[jryberg]

Hi,

Would it be possible to add support for process_credentials just as in aws-fluent-plugin-kinesis?

Ref: https://github.com/awslabs/aws-fluent-plugin-kinesis

[jryberg]

I tried to use the generic method of using a config file such as this

user@a96f1908bf7b:~/.aws# cat config
[profile logs]
credential_process = /usr/local/bin/get_credentials
region = eu-central-1

and then export AWS_PROFILE=logs

This is what I get

user@a96f1908bf7b:/# /opt/td-agent-bit/bin/td-agent-bit --verbose -e /fluent-bit/firehose.so -e /fluent-bit/cloudwatch.so -e /fluent-bit/kinesis.so -c /opt/td-agent-bit/etc/fluent-bit.conf
Fluent Bit v1.5.7
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/09/29 08:13:54] [Warning] [config] I cannot open /opt/td-agent-bit/etc/parsers.conf file
[2020/09/29 08:13:54] [Warning] [config] I cannot open /opt/td-agent-bit/etc/plugins.conf file
[2020/09/29 08:13:54] [ info] [engine] started (pid=114)
[2020/09/29 08:13:54] [ info] [storage] version=1.0.5, initializing...
[2020/09/29 08:13:54] [ info] [storage] in-memory
[2020/09/29 08:13:54] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
INFO[0000] [kinesis 0] plugin parameter stream = '<redacted>'
INFO[0000] [kinesis 0] plugin parameter region = 'eu-central-1'
INFO[0000] [kinesis 0] plugin parameter data_keys = ''
INFO[0000] [kinesis 0] plugin parameter partition_key = 'container_id'
INFO[0000] [kinesis 0] plugin parameter role_arn = ''
INFO[0000] [kinesis 0] plugin parameter endpoint = ''
INFO[0000] [kinesis 0] plugin parameter sts_endpoint = ''
INFO[0000] [kinesis 0] plugin parameter append_newline = true
INFO[0000] [kinesis 0] plugin parameter time_key = ''
INFO[0000] [kinesis 0] plugin parameter time_key_format = ''
INFO[0000] [kinesis 0] plugin parameter experimental_concurrency = ''
INFO[0000] [kinesis 0] plugin parameter experimental_concurrency_retries = ''
INFO[0000] [kinesis 0] plugin parameter log_key = ''
INFO[0000] [kinesis 0] plugin parameter aggregation = ''
INFO[0000] [kinesis 0] plugin parameter compression = ''
[2020/09/29 08:13:54] [ info] [sp] stream processor started
ERRO[0010] [kinesis 0] PutRecords failed with NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, logs.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed

As you can see, it did fail to load profile "logs".

/usr/local/bin/get_credentials does work, it will return credentials and it works for other kind of implementations.

I can also use aws cli with --profile logs and the cli does work as expected.

Best regards Johan Ryberg

[jryberg]

SOLVED!

It seems I had to export AWS_CONFIG_FILE=/absolute/path/to/config to make it work

So, it's possible to use process _credentials / credential_process but it was not very easy. Can it be more accessible or is this the way to do it?

[PettitWesley]

The core plugins will soon have credential process support as well. https://github.com/fluent/fluent-bit-docs/pull/573

[atreyd]

@jryberg was it working for you ? I have been trying the same thing from a non AWS machine ( from on-premise ), but this seems not to be working. Even if I have tried configuring profile as well as exported the AWS credentials. I have set all the variables like AWS_CONFIG_FILE=~/.aws/config, AWS_PROFILE=default and ,AWS_SHARED_CREDENTIALS_FILE=~/.aws/credentials and restarted the fluentbit DAEMON on the machine, but it's still giving the same error as td-agent-bit[63771]: time="2022-01-23T23:02:57-08:00" level=error msg="[kinesis 0] PutRecords failed with NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, .\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n"

[mbastawisy-careem]

Facing the same issue. Trying to connect from non AWS machine through the credentials process and no luck.

The core plugins will soon have credential process support as well. https://github.com/fluent/fluent-bit-docs/pull/573

So any plans to support that?

[PettitWesley]

@atreyd @mbastawisy-careem credential process should work, I think you just can't use this go plugin in this repo, you just use the C version: https://docs.fluentbit.io/manual/pipeline/outputs/kinesis

https://github.com/fluent/fluent-bit-docs/pull/573/files

[mbastawisy-careem]

I got it working yesterday after setting this magical environment variable

AWS_SDK_LOAD_CONFIG = 1

Which seems to do the trick to favor the config file over the credentials file.

But thanks @PettitWesley, for pointing out that there is a new kinesis plugin that is based on c. We will consider migrating to that if necessary .