search

awslabs / collectd-cloudwatch

A collectd plugin for sending data to Amazon CloudWatch
MIT License
200 stars 132 forks source link

Could not put metric data using the following endpoint #19

Closed den-is closed 7 years ago

den-is commented 7 years ago

Getting a lot of similar warning messages in /var/log/collectd.log

[2016-11-01 15:04:49] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Could not put metric data using the following endpoint: 'https://monitoring.us-east-1.amazonaws.com/'. [Exception: 400 Client Error: Bad Request for url: https://monitoring.us-east-1.amazonaws.com/?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-429dd254&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=0.676183320811&MetricData.member.1.StatisticValues.Minimum=0.475713570356&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=3.33067865993&MetricData.member.1.Timestamp=20161101T190348Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-429dd254&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.
[2016-11-01 15:04:49] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Request details: 'Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-429dd254&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=0.676183320811&MetricData.member.1.StatisticValues.Minimum=0.475713570356&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=3.33067865993&MetricData.member.1.Timestamp=20161101T190348Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-429dd254&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=swap.percent.used&MetricData.member.2.StatisticValues.Maximum=nan&MetricData.member.2.StatisticValues.Minimum=nan&MetricData.member.2.St
[2016-11-01 15:05:00] [info] [AmazonCloudWatchPlugin][cloudwatch.modules.flusher] [debug] flushing metrics df-root-percent_bytes-used[6]

OS: CentOS 7 EC2 instance, w/ IAM Role attached. us-east-1 collectd.conf has only one plugin enabled df only one metric is whitelisted in collectd-cloudwatch plugin - df-root-percent_bytes-used. Getting this metric in CloudWatch normally.

Can't see what's wrong with setup and how to fix this warnings. How to make it quiet finally. Looks like it's trying to put too much in one request or generated POST data is invalid - can't see whole request sent as you can see messages are omitted.

Update: getting this messages even when collectd service is stopped :/

sebasrp commented 7 years ago

Hi Denis, This is a duplicate of https://github.com/awslabs/collectd-cloudwatch/issues/13 The reason behind is that collectd returns 'nan' for the swap metric when its unable to gather that metric.

Few options that will prevent this from happening while we fix the above issue:

Closing as duplicate of the issue 13

edit: the collectd.log message above does try to publis the metric "MetricName=swap.percent.used" so it seems that your change "only one metric is whitelisted in collectd-cloudwatch plugin - df-root-percent_bytes-used." has not been taken into account.

Have you tried restarting collectd?

den-is commented 7 years ago

Thanks for suggestion, but it doesn't work

  1. swap plugin is disabled in collectd.conf (commented)
  2. whitelist.conf after initial setup was absolutely empty - it has only one thing i put there df-root-percent_bytes-used
[2016-11-01 16:53:44] plugin_load: plugin "logfile" successfully loaded.
[2016-11-01 16:53:44] [info] plugin_load: plugin "cpu" successfully loaded.
[2016-11-01 16:53:44] [info] plugin_load: plugin "df" successfully loaded.
[2016-11-01 16:53:44] [info] plugin_load: plugin "memory" successfully loaded.
[2016-11-01 16:53:44] [info] plugin_load: plugin "python" successfully loaded.
[2016-11-01 16:53:45] [notice] Systemd detected, trying to signal readyness.
[2016-11-01 16:53:45] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.configuration.credentialsreader] Cannot read AWS credentials from file. Defaulting to use IAM Role.
[2016-11-01 16:53:45] [info] [AmazonCloudWatchPlugin][cloudwatch_writer] Initialization finished successfully.
[2016-11-01 16:53:45] [info] Initialization complete, entering read-loop.
[2016-11-01 16:53:49] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Could not put metric data using the following endpoint: 'https://monitoring.us-east-1.amazonaws.com/'. [Exception: 400 Client Error: Bad Request for url: https://monitoring.us-east-1.amazonaws.com/?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-429dd254&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=1.17735470942&MetricData.member.1.StatisticValues.Minimum=0.351141208929&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=5.00953169442&MetricData.member.1.Timestamp=20161101T205248Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-429dd254&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2
[2016-11-01 16:53:49] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Request details: 'Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-429dd254&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=1.17735470942&MetricData.member.1.StatisticValues.Minimum=0.351141208929&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=5.00953169442&MetricData.member.1.Timestamp=20161101T205248Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-429dd254&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=swap.percent.used&MetricData.member.2.StatisticValues.Maximum=nan&MetricData.member.2.StatisticValues.Minimum=nan&MetricData.member.2.Sta
[2016-11-01 16:53:53] [info] [AmazonCloudWatchPlugin][cloudwatch.modules.flusher] [debug] flushing metrics df-root-percent_bytes-used[6]
[2016-11-01 16:54:33] [info] [AmazonCloudWatchPlugin][cloudwatch.modules.flusher] [debug] flushing metrics df-root-percent_bytes-used[6]
[2016-11-01 16:54:44] [info] [AmazonCloudWatchPlugin][cloudwatch.modules.flusher] [debug] flushing metrics df-root-percent_bytes-used[6]
[2016-11-01 16:54:49] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Could not put metric data using the following endpoint: 'https://monitoring.us-east-1.amazonaws.com/'. [Exception: 400 Client Error: Bad Request for url: https://monitoring.us-east-1.amazonaws.com/?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-429dd254&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=1.35304434979&MetricData.member.1.StatisticValues.Minimum=0.600600600601&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=5.81010092148&MetricData.member.1.Timestamp=20161101T205348Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-429dd254&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2
[2016-11-01 16:54:49] [warning] [AmazonCloudWatchPlugin][cloudwatch.modules.client.putclient] Request details: 'Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=Host&MetricData.member.1.Dimensions.member.1.Value=i-429dd254&MetricData.member.1.Dimensions.member.2.Name=PluginInstance&MetricData.member.1.Dimensions.member.2.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=1.35304434979&MetricData.member.1.StatisticValues.Minimum=0.600600600601&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=5.81010092148&MetricData.member.1.Timestamp=20161101T205348Z&MetricData.member.2.Dimensions.member.1.Name=Host&MetricData.member.2.Dimensions.member.1.Value=i-429dd254&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=swap.percent.used&MetricData.member.2.StatisticValues.Maximum=nan&MetricData.member.2.StatisticValues.Minimum=nan&MetricData.member.2.Sta
[2016-11-01 16:54:53] [info] [AmazonCloudWatchPlugin][cloudwatch.modules.flusher] [debug] flushing metrics df-root-percent_bytes-used[6]
sebasrp commented 7 years ago

Hi Denis, The log excerpts below shows you are trying to publish 2 metrics:

The second one has the 'nan' values which is the root cause of your problem. Are you modifying the file /opt/collectd-plugins/cloudwatch/config/whitelist.conf ? You will also need to restart collectd after making changes to that file.

den-is commented 7 years ago

Restarting instance helped... I have been stopping restarting collectd service many times. Log file has been written with this warning messages even when collectd was stopped and process list was showing nothing. Probably some zombie process or something like this. It's quiet now.

Thanks!