search

awslabs / collectd-cloudwatch

A collectd plugin for sending data to Amazon CloudWatch
MIT License
200 stars 132 forks source link

Could not put metric data using the following endpoint - Exception: 403 Client Error: Forbidden for url #55

Open ratneshamarnath opened 7 years ago

ratneshamarnath commented 7 years ago

Hi All,

I am trying to run collectd-cloudwatch on AWS instance.

I am seeing following error in while I run "sudo /etc/init.d/collectd restart" -

_=============================================== [2017-10-02 06:38:11] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Could not put metric data using the following endpoint: 'https://monitoring.us-east-2.amazonaws.com/'. [Exception: 403 Client Error: Forbidden for url: https://monitoring.us-east-2.amazonaws.com/?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=dev-shm&MetricData.member.1.MetricName=df.percent_bytes.used&MetricData.member.1.StatisticValues.Maximum=0.0&MetricData.member.1.StatisticValues.Minimum=0.0&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=0.0&MetricData.member.1.Timestamp=20171002T063711Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=de [2017-10-02 06:38:11] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Request details: 'Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=dev-shm&MetricData.member.1.MetricName=df.percent_bytes.used&MetricData.member.1.StatisticValues.Maximum=0.0&MetricData.member.1.StatisticValues.Minimum=0.0&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=0.0&MetricData.member.1.Timestamp=20171002T063711Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=dev&MetricData.member.2.MetricName=df.percentbytes.used&MetricData.member.2.StatisticValues.Maximum=0.0112053789198&MetricData.member.2.StatisticValues.Minimum=0.0112053789198&Met ============================================

Could someone provide me pointer to proceed further on it.

Thanks Ratnesh

yimuniao commented 7 years ago

Normally, it was caused by the invalid key, how do you setup your EC2 instance? do you attach IAM role to the EC2 instance? or do you put the credentials to /root/.aws/credentials?

ratneshamarnath commented 7 years ago

Hi,

I have tried both using IAM role and credential steps.

===================================================================

[ec2-user@ip-172-31-30-191 src]$ [ec2-user@ip-172-31-30-191 src]$ sudo ./setup.py Installing dependencies ... OK Installing python dependencies ... OK Downloading plugin ... OK Extracting plugin ... OK Moving to collectd plugins directory ... OK Copying CloudWatch plugin include file ... OK DEBUG:urllib3.util.retry:Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254 DEBUG:urllib3.connectionpool:http://169.254.169.254:80 "GET /latest/meta-data/placement/availability-zone/ HTTP/1.1" 200 10

Choose AWS region for published metrics:

  1. Automatic [us-east-2]
  2. Custom Enter choice [1]: 1 DEBUG:urllib3.util.retry:Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254 DEBUG:urllib3.connectionpool:http://169.254.169.254:80 "GET /latest/meta-data/instance-id/ HTTP/1.1" 200 19

Choose hostname for published metrics:

  1. EC2 instance id [i-0a780d545c7ad981c]
  2. Custom Enter choice [1]: 1 DEBUG:urllib3.util.retry:Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254 DEBUG:urllib3.connectionpool:http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 10

Choose authentication method:

  1. IAM Role [cloudwatch]
  2. IAM User Enter choice [1]: 1

Enter proxy server name:

  1. None
  2. Custom Enter choice [1]: 1

Enter proxy server port:

  1. None
  2. Custom Enter choice [1]: 1

Include the Auto-Scaling Group name as a metric dimension:

  1. No
  2. Yes Enter choice [1]: 1

Include the FixedDimension as a metric dimension:

  1. No
  2. Yes Enter choice [1]: 1

Enable high resolution:

  1. Yes
  2. No Enter choice [2]: 2

Enter flush internal:

  1. Default 60s
  2. Custom Enter choice [1]: 1

Choose how to install CloudWatch plugin in collectd:

  1. Do not modify existing collectd configuration
  2. Add plugin to the existing configuration

  3. Use CloudWatch recommended configuration (4 metrics) Enter choice [3]: 3 Plugin configuration written successfully. Creating backup of the original configuration ... OK Replacing collectd configuration ... OK Replacing whitelist configuration ... OK Stopping collectd process ... OK Starting collectd process ... OK [ec2-user@ip-172-31-30-191 src]$ [ec2-user@ip-172-31-30-191 src]$ [ec2-user@ip-172-31-30-191 src]$ [ec2-user@ip-172-31-30-191 src]$ sudo /etc/init.d/collectd restart Stopping collectd: [FAILED] Starting collectd: [ OK ] [ec2-user@ip-172-31-30-191 src]$ [ec2-user@ip-172-31-30-191 src]$ tail -f /var/log/collectd.log [2017-10-02 17:41:38] [info] plugin_load: plugin "df" successfully loaded. [2017-10-02 17:41:38] [info] plugin_load: plugin "memory" successfully loaded. [2017-10-02 17:41:38] [info] plugin_load: plugin "swap" successfully loaded. [2017-10-02 17:41:38] [info] plugin_load: plugin "python" successfully loaded. [2017-10-02 17:41:38] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.ec2getclient] Could not get the autoscalig group name using the following endpoint: 'https://ec2.us-east-2.amazonaws.com/'. [Exception: 401 Client Error: Unauthorized for url: https://ec2.us-east-2.amazonaws.com/?Action=DescribeTags&Filter.1.Name=key&Filter.1.Value.1=aws%3Aautoscaling%3AgroupName&Filter.2.Name=resource-id&Filter.2.Value.1=i-0a780d545c7ad981c&Version=2016-11-15&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJYIZE2JHMZVBEDTA%2F20171002%2Fus-east-2%2Fec2%2Faws4_request&X-Amz-Date=20171002T174137Z&X-Amz-SignedHeaders=host&X-Amz-Signature=a48034203dfcacb8565c3aca234be4ba1ea230104512e36e2b14625d7eee6bec] [2017-10-02 17:41:38] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.ec2getclient] Request details: 'Action=DescribeTags&Filter.1.Name=key&Filter.1.Value.1=aws%3Aautoscaling%3AgroupName&Filter.2.Name=resource-id&Filter.2.Value.1=i-0a780d545c7ad981c&Version=2016-11-15&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJYIZE2JHMZVBEDTA%2F20171002%2Fus-east-2%2Fec2%2Faws4_request&X-Amz-Date=20171002T174137Z&X-Amz-SignedHeaders=host&X-Amz-Signature=a48034203dfcacb8565c3aca234be4ba1ea230104512e36e2b14625d7eee6bec' [2017-10-02 17:41:38] [info] [AmazonCloudWatchPlugin][cloudwatch.modules.configuration.confighelper] Fetched asg name as NONE [2017-10-02 17:41:38] [info] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] No proxy server is in use [2017-10-02 17:41:38] [info] [AmazonCloudWatchPlugin][cloudwatch_writer] Initialization finished successfully. [2017-10-02 17:36:59] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.flusher] Adding Metric value is not numerical, key: swap--percent-used value: [nan] [2017-10-02 17:36:59] [info] Initialization complete, entering read-loop. [2017-10-02 17:37:53] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Could not put metric data using the following endpoint: 'https://monitoring.us-east-2.amazonaws.com/'. [Exception: HTTPSConnectionPool(host='monitoring.us-east-2.amazonaws.com', port=443): Max retries exceeded with url: /?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=7.59109311741&MetricData.member.1.StatisticValues.Minimum=0.299102691924&MetricData.member.1.StatisticValues.SampleCount=5&MetricData.member.1.StatisticValues.Sum=9.28461990044&MetricData.member.1.Timestamp=20171002T173702Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.2.Dimensions.member.2.Name=PluginInstan [2017-10-02 17:37:53] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Request details: 'Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=7.59109311741&MetricData.member.1.StatisticValues.Minimum=0.299102691924&MetricData.member.1.StatisticValues.SampleCount=5&MetricData.member.1.StatisticValues.Sum=9.28461990044&MetricData.member.1.Timestamp=20171002T173702Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-0a780d545c7ad981c&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=memory.percent.used&MetricData.member.2.StatisticValues.Maximum=5.21932980267&MetricData.member.2.StatisticValues.Minim

From the error description given on url (http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html), it seems that's problem is related to access -

1) AccessDenied -- Access Denied -- 403 Forbidden Client 2) AccountProblem -- There is a problem with your AWS account that prevents the operation from completing successfully. Please use Contact Us. -- 403 Forbidden Client

Please let me know if you need anything else to proceed further.

Thanks in Advance !! Ratnesh

yimuniao commented 7 years ago

Could you enable "debug" flag in /opt/collectd-plugins/cloudwatch/config//plugin.conf? Then restart collectd, then you will see there are http post trace in a temporary file like /tmp/collectd_plugin_request_trace_log

copies one of the commands, then run in terminal. it will give you more failure information. Like

[ec2-user@ip-172-31-34-234 config]$ curl -i -v -connect-timeout 1 -m 3 -w %{http_code}:%{http_connect}:%{content_type}:%{time_namelookup}:%{time_redirect}:%{time_pretransfer}:%{time_connect}:%{time_starttransfer}:%{time_total}:%{speed_download} -A "collectd/1.0" 'https://monitoring.eu-west-1.amazonaws.com/?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=0.5&MetricData.member.1.StatisticValues.Minimum=0.0&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=0.700300500902&MetricData.member.1.Timestamp=20171004T070624Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=memory.percent.used&MetricData.member.2.StatisticValues.Maximum=4.5757107528&MetricData.member.2.StatisticValues.Minimum=4.3323386136&MetricData.member.2.StatisticValues.SampleCount=6&MetricData.member.2.StatisticValues.Sum=26.7001647382&MetricData.member.2.Timestamp=20171004T070624Z&MetricData.member.3.Dimensions.member.1.Name=Host&MetricData.member.3.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.3.Dimensions.member.2.Name=PluginInstance&MetricData.member.3.Dimensions.member.2.Value=root&MetricData.member.3.MetricName=df.percent_bytes.used&MetricData.member.3.StatisticValues.Maximum=14.0054941177&MetricData.member.3.StatisticValues.Minimum=14.0054941177&MetricData.member.3.StatisticValues.SampleCount=6&MetricData.member.3.StatisticValues.Sum=84.0329647064&MetricData.member.3.Timestamp=20171004T070624Z&Namespace=collectd&Version=2010-08-01&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=accesskey%2F20171004%2Feu-west-1%2Fmonitoring%2Faws4_request&X-Amz-Date=20171004T070724Z&X-Amz-SignedHeaders=host&X-Amz-Signature=f4200b40a340830c887f2b32fd6cf880e8e3f33e20696ff3c805f72ba5f9d9fe'
* Rebuilt URL to: 1/
*   Trying 0.0.0.1...
* TCP_NODELAY set
* Immediate connect fail for 0.0.0.1: Invalid argument
* Closing connection 0
curl: (7) Couldn't connect to server
000:000::0.004:0.000:0.000:0.000:0.000:0.000:0.000*   Trying 52.94.219.139...
* TCP_NODELAY set
* Connected to monitoring.eu-west-1.amazonaws.com (52.94.219.139) port 443 (#1)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* ALPN/NPN, server did not agree to a protocol
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
*   subject: CN=monitoring.eu-west-1.amazonaws.com,O="Amazon.com, Inc.",L=Seattle,ST=Washington,C=US
*   start date: Sep 26 00:00:00 2017 GMT
*   expire date: Jun 26 23:59:59 2018 GMT
*   common name: monitoring.eu-west-1.amazonaws.com
*   issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET /?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=0.5&MetricData.member.1.StatisticValues.Minimum=0.0&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=0.700300500902&MetricData.member.1.Timestamp=20171004T070624Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=memory.percent.used&MetricData.member.2.StatisticValues.Maximum=4.5757107528&MetricData.member.2.StatisticValues.Minimum=4.3323386136&MetricData.member.2.StatisticValues.SampleCount=6&MetricData.member.2.StatisticValues.Sum=26.7001647382&MetricData.member.2.Timestamp=20171004T070624Z&MetricData.member.3.Dimensions.member.1.Name=Host&MetricData.member.3.Dimensions.member.1.Value=i-081dcbd2a7b7f0852&MetricData.member.3.Dimensions.member.2.Name=PluginInstance&MetricData.member.3.Dimensions.member.2.Value=root&MetricData.member.3.MetricName=df.percent_bytes.used&MetricData.member.3.StatisticValues.Maximum=14.0054941177&MetricData.member.3.StatisticValues.Minimum=14.0054941177&MetricData.member.3.StatisticValues.SampleCount=6&MetricData.member.3.StatisticValues.Sum=84.0329647064&MetricData.member.3.Timestamp=20171004T070624Z&Namespace=collectd&Version=2010-08-01&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=accesskey%2F20171004%2Feu-west-1%2Fmonitoring%2Faws4_request&X-Amz-Date=20171004T070724Z&X-Amz-SignedHeaders=host&X-Amz-Signature=f4200b40a340830c887f2b32fd6cf880e8e3f33e20696ff3c805f72ba5f9d9fe HTTP/1.1
> Host: monitoring.eu-west-1.amazonaws.com
> User-Agent: collectd/1.0
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
HTTP/1.1 403 Forbidden
< x-amzn-RequestId: dd2d4791-a8d2-11e7-a9a4-17012e56af42
x-amzn-RequestId: dd2d4791-a8d2-11e7-a9a4-17012e56af42
< Content-Type: text/xml
Content-Type: text/xml
< Content-Length: 312
Content-Length: 312
< Date: Wed, 04 Oct 2017 07:08:46 GMT
Date: Wed, 04 Oct 2017 07:08:46 GMT

< 
<ErrorResponse xmlns="http://monitoring.amazonaws.com/doc/2010-08-01/">
  <Error>
    <Type>Sender</Type>
    <Code>InvalidClientTokenId</Code>
    <Message>The security token included in the request is invalid.</Message>
  </Error>
  <RequestId>dd2d4791-a8d2-11e7-a9a4-17012e56af42</RequestId>
</ErrorResponse>
* Curl_http_done: called premature == 0
* Connection #1 to host monitoring.eu-west-1.amazonaws.com left intact
ratneshamarnath commented 7 years ago

Hey it worked when I created new AWS instance with same Role.

so why its not working if I assign role to existing AWS instance ?