search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.63k stars 5.58k forks source link

Add support for refreshing AWS Credentials when using temporary session tokens #13268

Open wave2 opened 1 year ago

wave2 commented 1 year ago

Use Case

When using temporary security credentials to connect to AWS, the Telegraf agent should detect changes to the shared_credential_file to prevent errors and the need to restart the agent.

Expected behavior

The Telegraf agent should refresh the credentials when the shared credential file is updated, periodically or in response to an ExpiredToken error.

It looks like this behaviour was added to the amazon-cloudwatch-agent in the form of a refresh frequency on the credentials provider:

https://github.com/aws/amazon-cloudwatch-agent/blob/main/cfg/aws/refreshable_shared_credentials_provider.go

Actual behavior

When another process updates the shared_credential_file, the Telegraf agent fails to PutMetricData and instead throws the following error:

2023-05-11T15:26:09Z E! [outputs.cloudwatch] Unable to write to CloudWatch : operation error CloudWatch: PutMetricData, https response error StatusCode: 403, RequestID: 4d5fbc11-8aba-489a-90b4-0966ab111ae5, api error ExpiredToken: The security token included in the request is expired

You can reproduce this by configuring the cloudwatch output plugin to use the SSM Managed credential file (e.g. /var/lib/amazon/ssm/credentials) on a non-EC2 host running the SSM Agent:

https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent-technical-details.html#credentials-file https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-managed-linux.html

Additional info

No response

powersj commented 1 year ago

Hi,

https://github.com/aws/amazon-cloudwatch-agent/blob/main/cfg/aws/refreshable_shared_credentials_provider.go

This example is using github.com/aws/aws-sdk-go/aws/credentials while we use github.com/aws/aws-sdk-go-v2/credentials

In the v2 repo I see comments related to refreshing the SSOTokenProvider, but not a SharedCredentialsFiles. Do you see the same? If so, would you be willing to file an upstream issue with the request to support the same as v1 and let us know the #?

Thanks!

wave2 commented 1 year ago

Thanks!

I added a comment to a similar closed issue on that repo that was not not resolved:

https://github.com/aws/aws-sdk-go-v2/issues/1449

If it's not possible to re-open that issue I will raise a new one in a day or two.

powersj commented 1 year ago

Awesome thanks! Let us know what you find out.

wave2 commented 1 year ago

I raised a fresh ticket with the aws-sdk-go-v2 project here:

https://github.com/aws/aws-sdk-go-v2/issues/2135

wave2 commented 1 year ago

The upstream issue was closed stating there are currently no plans to add the ability to refresh shared credentials in aws-sdk-go-v2. Load once is intended behaviour it seems.

A cross-sdk request was raised over a year ago requesting this functionality (https://github.com/aws/aws-cli/issues/9034) and would welcome upvotes :)

Is there any way to attempt to reload the credentials in response to an ExpiredToken error?

powersj commented 1 year ago

Is there any way to attempt to reload the credentials in response to an ExpiredToken error?

While I would prefer to see this land in the upstream library, we would also be happy to see a PR for this behavior. It would need to be a bit careful as data in cloudwatch is sent in batches in different go routines, and we wouldn't want all of them at the same time to try to reload the file or reinitialize the aws client.