Telegraf not connecting to AWS Cloudwatch using rolearn.

[pckeyan]

Hi All,

I have a req to post application metris to AWS Cloudwatch. We have restriction to use hard coded AWS credentials, hence would like to explore the application to use roleArn. But I am getting the following error:

cat /var/log/telegraf/telegraf.log

017/03/24 15:10:50 E! cloudwatch: Error in ListMetrics API call : NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2017/03/24 15:10:50 E! Failed to connect to output cloudwatch, retrying in 15s, error was 'NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors'

Config snippet

[[outputs.cloudwatch]]
  region = "us-east-1"
  **role_arn = "arn:aws:iam::ACC_NUM:role/ROLE_NAME"**
  namespace = "myspace/jvmstats"

Thanks and appreciate any help here Karthik

[feelobot]

Have you tried specifying that in the config? https://github.com/influxdata/telegraf/tree/master/plugins/outputs/kinesis has support and the output plugin for cloudwatch looks like it has support https://github.com/influxdata/telegraf/blob/master/plugins/outputs/cloudwatch/cloudwatch.go#L22

[rorysavage77]

I can confirm this bug still persists. I have confirmed that my machine's role has complete access to AWS Cloudwatch using Python Boto scripts, however when using Telegraf and specifying this configuration it produces the same error results.

For verbose messaging see aws.Config.CredentialsChainVerboseErrors' 

2017-06-21T12:11:39Z E! cloudwatch: Error in ListMetrics API call : NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors 2017-06-21T12:11:39Z E! NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors 2017-06-21T12:11:40Z D! Attempting connection to output: cloudwatch 2017-06-21T12:11:57Z E! cloudwatch: Error in ListMetrics API call : NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors 2017-06-21T12:11:57Z E! Failed to connect to output cloudwatch, retrying in 15s, error was 'NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors'

[lkashl]

Did you get this working? The symptoms are the same as the issue I had on my end.

If so can you validate whether you can do the following as your normal user: curl http://169.254.169.254/user-data/

If that succeeds can you then execute telegraf as the same user and see whether you get the same errors? In my case it was caused by proxy vars making the local Amazon metadata unqueryable

[sunjay-jeffrish]

Use the method of sts role_arn : For that Let me explain the use case of getting aws cloudwatch metrics. Step ! : Create a role aws_iam_role like cloudwatchmetrics: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-sharing-logs-create-role.html Step 2: Attach a policy which will give its permission to read cloudwatchmetrics: https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-attach-managed-policy/ Step 3: find the role which your ec2 instance is using : Step 4 : give the power of assuming cloudwatchmetrics role to your ec2 . Step 5 : if you are using kubernetes in podAnnotaions add iam.amazonaws.com/role: arn:aws:iam::ACCOUNTID:role/cloudwatchmetricsrole

It will work maybe you will get x509 certificate error . But lets leave that for an other day 💯

[tmiroslav]

I am using inputs.cloudwatch plugin, and can confirm that, after following @sunjay-jeffrish procedure to set-up IAM role and policy, it does not work. It only works if I attach Admin policy to telegraf ec2 role. Otherwise, ec2 where telegraf is running is failing to assume a role. This is the error:

E! [inputs.cloudwatch] Error in plugin: failed to get metric data: AccessDenied: User: arn:aws:sts::xxxx:assumed-role/FrameTick/instanceid is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::xxxxxx:role/FrameTick

When using other credentials, like access_key/secret_key, I see that

"cloudwatch:GetMetricData",
 "cloudwatch:ListMetrics"

permission is enough. Why role_arn needs Admin policy , so telegraf can properly assume role in this case?

[srebhan]

@pckeyan, @tmiroslav, @sunjay-jeffrish, @lkashl, @rorysavage77, @feelobot is this still the case with latest Telegraf? If so, is there a simple way to reproduce the issue on my side?

[telegraf-tiger[bot]]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Forums or provide additional details in this issue and reqeust that it be re-opened. Thank you!