search

awslabs / amazon-kinesis-agent

Continuously monitors a set of log files and sends new data to the Amazon Kinesis Stream and Amazon Kinesis Firehose in near-real-time.
Other
352 stars 218 forks source link

Kinesis Agent 2.0.8 fails to send records using IAM role when IMDSv2 is enabled #264

Open sparya opened 1 year ago

sparya commented 1 year ago

I am using Kinesis Agent to send records to Firehose Stream. All the accesses are provided on a Role which is attached to ECS container where agent is running. Agent is able to send records when IMDSv2 is disabled on EC2 machine on which container is running. On Enabling IMDSv2, agent fails to pick IAM role and choose to access delivery stream via UserDefinedCredentialsProvider. Kineis Agent Logs throws below error -

com.amazon.kinesis.streaming.agent.tailing.AsyncPublisher [ERROR] AsyncPublisher[fh:ad_relevance_score_logs:/tmp/ad_relevance/ad_scores.log*]:RecordBuffer(id=2,records=2,bytes=450) Retriable send error (com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [UserDefinedCredentialsProvider: Unable to create credentials from user defined credentials provider, AgentAWSCredentialsProvider: Unable to load credentials from agent config. Missing entries: awsAccessKeyId awsSecretAccessKey, EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.ContainerCredentialsProvider@43c77db2: The environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is empty, com.amazonaws.auth.profile.ProfileCredentialsProvider@5a8d3e11: profile file cannot be null, com.amazonaws.auth.InstanceProfileCredentialsProvider@3cfd7e98: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null; Proxy: null)]). Will retry.

Kinesis Agent Version - 2.0.8 aws-java-sdk.version - 1.12.503 and 1.12.390 (tried with both) Also Added assumeRoleARN in agent.json config and explicitly specified IAM Role having all access. Is it a limitation of Kinesis Agent?

buddhike commented 10 months ago

Hi @sparya, Thanks for reporting this issue. Could you please share the ECS agent version you are running in your EC2 instance? 🙏