search

awslabs / amazon-kinesis-agent

Continuously monitors a set of log files and sends new data to the Amazon Kinesis Stream and Amazon Kinesis Firehose in near-real-time.
Other
355 stars 222 forks source link

Kinesis Agent 1.1.6 fails to send records when IMDSv2 Tokens are required #246

Open absa-rsuarez opened 2 years ago

absa-rsuarez commented 2 years ago

Logging this here for the sake of helping others, when running the Kinesis agent referenced in the documentation for Red Hat: https://docs.aws.amazon.com/streams/latest/dev/writing-with-agents.html#download-install

You are directed to download v1 (1.1.6) of the agent. When IMDSv2 tokens are set to required this version is unable to locate the EC2 Instance credentials from the metadata endpoint.

This results in the following error being logged in the Kinesis Agent logs:

2022-08-03 09:38:27.582+0200  (cw-metrics-publisher) com.amazon.kinesis.streaming.agent.metrics.CWPublisherRunnable [WARN] Could not publish 2 datums to CloudWatch
com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
    at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)
    at com.amazonaws.http.AmazonHttpClient.getCredentialsFromContext(AmazonHttpClient.java:796)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:822)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:714)
    at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:465)
    at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:427)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:376)
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:1068)
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:1038)
    at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData(AmazonCloudWatchClient.java:937)
    at com.amazon.kinesis.streaming.agent.metrics.DefaultCWMetricsPublisher.publishMetrics(DefaultCWMetricsPublisher.java:65)
    at com.amazon.kinesis.streaming.agent.metrics.CWPublisherRunnable.runOnce(CWPublisherRunnable.java:137)
    at com.amazon.kinesis.streaming.agent.metrics.CWPublisherRunnable.run(CWPublisherRunnable.java:89)
    at java.lang.Thread.run(Thread.java:750)

When instead using v2.x (2.0.6 in our case)

https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-latest.amzn2.noarch.rpm

the agent is again able to locate credentials even with tokens required. We have requested an update of the documentation from our account team as well.