Permission denied when requesting metadata

[Fodoj]

I keep getting this unpleasant error:

Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: [AmazonCloudWatchPlugin][cloudwatch.modules.configuration.credentialsreader] Cannot read AWS credentials from file. Defaulting to use IAM Role.
Nov 18 16:41:55 ip-172-21-82-61 systemd[1]: Started Collectd statistics daemon.
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: [AmazonCloudWatchPlugin][cloudwatch_writer] Cannot initialize plugin. Cause: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/meta-data/iam/security-credentials/ (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa384341cd0>: Failed to establish a new connection: [Errno 13] Permission denied',))
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: Traceback (most recent call last):
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: File "/opt/collectd-plugins/cloudwatch_writer.py", line 22, in aws_init
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: config = ConfigHelper()
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: File "/opt/collectd-plugins/cloudwatch/modules/configuration/confighelper.py", line 41, in __init__
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: self._load_configuration()
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: File "/opt/collectd-plugins/cloudwatch/modules/configuration/confighelper.py", line 66, in _load_configuration
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: self._load_credentials()
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: File "/opt/collectd-plugins/cloudwatch/modules/configuration/confighelper.py", line 89, in _load_credentials
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: self.credentials = self._get_credentials_from_iam_role()
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: File "/opt/collectd-plugins/cloudwatch/modules/con
Nov 18 16:41:55 ip-172-21-82-61 collectd[13613]: Initialization complete, entering read-loop.

If I try to curl metadata server I get correct result without any errors.

[tchia04]

Perhaps the instance doesn't have a IAM role that lets you PutMetricData?

Do you have something like this on your instance's policy document?

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1480026642000",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

[Fodoj]

The problem was SELinux. This plugin doesn't work with rhel7 epel collectd package:

1. /opt/collectd-plugins needs to be re-labeled with collectd_var_lib_t something (sorry I don't remember except)
2. Boolean collectd_tcp_something should be enabled.

I ended up throwing away the whole horrible 500+ python lines of code of setup.py and replaced it with 20 lines Chef recipe that installs plugin to /var/lib/collectd and enables the SE Boolean.

[sturdytroyready]

@Fodoj would you mind gisting that recipe? :)