search

awslabs / collectd-cloudwatch

A collectd plugin for sending data to Amazon CloudWatch
MIT License
200 stars 132 forks source link

Flexible Dimension #46

Open linshu opened 7 years ago

linshu commented 7 years ago

This is a framework implementing a way to easily add any dimension. (Issue #43) The dimension coming in as a plugin. You can add it in dimensionplugin directory. As long as you initialize the dimension in Dimensions class init function, you can enable the dimension in the runtime by adding its name in dimensions.conf. I have include a fix for adding Unit (Issue #42) in this request since that original request comes from a branch we can't open access to the public.

Fodoj commented 7 years ago

Docs need to be updated to make people aware of how to use this feature

yimuniao commented 7 years ago

This pull request has conflict with master branch, could you merge the latest codes?

linshu commented 7 years ago

I have added the explanation how to use this feature in README.md.

Thanks.

yimuniao commented 7 years ago

Please confirm this contribution is under the terms of the MIT license. Thanks!

linshu commented 7 years ago

Yes, I confirm that this contribution is under the MIT license. Thanks!

Clecompt commented 7 years ago

+1 for this feature, anything this PR is waiting on?

yimuniao commented 7 years ago

Colleague in Open Source IP department is checking the license.

yimuniao commented 7 years ago

Is this contribution being made on behalf of the individual or on behalf of the company? looks like the you is employed at Palo Alto Networks

linshu commented 7 years ago

It is on behalf of the company, Palo Alto Networks. Thanks.

linshu commented 6 years ago

What are we waiting for to merge this PR?

yimuniao commented 6 years ago

Sorry. It is a little complicated. The latest reply from our security team:

Liang: Do you anticipate this being a one time contribution? Or do you expect to receive repeated contributions from this person? Thanks,

I replied: I expect to receive repeated contributions from this person.
I guess security guy may has some concern about the license as it is on behalf of your company.

Anyway, I need to wait for the reply from security team and then reserve some time to test it.

linshu commented 6 years ago

Thanks for the update. Yes, I might contribute more on this project if they are useful.

I can get our IP attorney to answer some further questions if needed.

Thanks.

bidesh commented 6 years ago

@yimuniao any update on the issue here? We are also waiting for this feature. :)

yimuniao commented 6 years ago

This contribution was approved by Security team. I will test it and if everything is ok, I will merge it soon.

yimuniao commented 6 years ago

I met following error, some metric has "none" for dimension "PluginInstance" :

[root@ip-172-31-33-105 config]# curl -i -v -connect-timeout 1 -m 3 -w %{http_code}:%{http_connect}:%{content_type}:%{time_namelookup}:%{time_redirect}:%{time_pretransfer}:%{time_connect}:%{time_starttransfer}:%{time_total}:%{speed_download} -A "collectd/1.0" 'https://monitoring.eu-west-1.amazonaws.com/?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=InstanceId&MetricData.member.1.Dimensions.member.1.Value=i-064072d9df139837d&MetricData.member.1.Dimensions.member.2.Name=FixedDimension&MetricData.member.1.Dimensions.member.2.Value=ALL&MetricData.member.1.Dimensions.member.3.Name=PluginInstance&MetricData.member.1.Dimensions.member.3.Value=&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=0.300601202405&MetricData.member.1.StatisticValues.Minimum=0.1001001001&MetricData.member.1.StatisticValues.SampleCount=5&MetricData.member.1.StatisticValues.Sum=1.00414778939&MetricData.member.1.Timestamp=20171208T045358Z&MetricData.member.1.Unit=active&MetricData.member.2.Dimensions.member.1.Name=FixedDimension&MetricData.member.2.Dimensions.member.1.Value=ALL&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=cpu.percent.active&MetricData.member.2.StatisticValues.Maximum=0.300601202405&MetricData.member.2.StatisticValues.Minimum=0.1001001001&MetricData.member.2.StatisticValues.SampleCount=5&MetricData.member.2.StatisticValues.Sum=1.00414778939&MetricData.member.2.Timestamp=20171208T045358Z&MetricData.member.2.Unit=active&MetricData.member.3.Dimensions.member.1.Name=InstanceId&MetricData.member.3.Dimensions.member.1.Value=i-064072d9df139837d&MetricData.member.3.Dimensions.member.2.Name=FixedDimension&MetricData.member.3.Dimensions.member.2.Value=ALL&MetricData.member.3.Dimensions.member.3.Name=PluginInstance&MetricData.member.3.Dimensions.member.3.Value=&MetricData.member.3.MetricName=memory.percent.used&MetricData.member.3.StatisticValues.Maximum=6.73906803553&MetricData.member.3.StatisticValues.Minimum=6.68952473823&MetricData.member.3.StatisticValues.SampleCount=6&MetricData.member.3.StatisticValues.Sum=40.3368944217&MetricData.member.3.Timestamp=20171208T045348Z&MetricData.member.3.Unit=used&MetricData.member.4.Dimensions.member.1.Name=FixedDimension&MetricData.member.4.Dimensions.member.1.Value=ALL&MetricData.member.4.Dimensions.member.2.Name=PluginInstance&MetricData.member.4.Dimensions.member.2.Value=NONE&MetricData.member.4.MetricName=memory.percent.used&MetricData.member.4.StatisticValues.Maximum=6.73906803553&MetricData.member.4.StatisticValues.Minimum=6.68952473823&MetricData.member.4.StatisticValues.SampleCount=6&MetricData.member.4.StatisticValues.Sum=40.3368944217&MetricData.member.4.Timestamp=20171208T045348Z&MetricData.member.4.Unit=used&MetricData.member.5.Dimensions.member.1.Name=InstanceId&MetricData.member.5.Dimensions.member.1.Value=i-064072d9df139837d&MetricData.member.5.Dimensions.member.2.Name=FixedDimension&MetricData.member.5.Dimensions.member.2.Value=ALL&MetricData.member.5.Dimensions.member.3.Name=PluginInstance&MetricData.member.5.Dimensions.member.3.Value=root&MetricData.member.5.MetricName=df.percent_bytes.used&MetricData.member.5.StatisticValues.Maximum=13.4382724762&MetricData.member.5.StatisticValues.Minimum=13.4382724762&MetricData.member.5.StatisticValues.SampleCount=6&MetricData.member.5.StatisticValues.Sum=80.6296348572&MetricData.member.5.Timestamp=20171208T045348Z&MetricData.member.5.Unit=used&MetricData.member.6.Dimensions.member.1.Name=FixedDimension&MetricData.member.6.Dimensions.member.1.Value=ALL&MetricData.member.6.Dimensions.member.2.Name=PluginInstance&MetricData.member.6.Dimensions.member.2.Value=root&MetricData.member.6.MetricName=df.percent_bytes.used&MetricData.member.6.StatisticValues.Maximum=13.4382724762&MetricData.member.6.StatisticValues.Minimum=13.4382724762&MetricData.member.6.StatisticValues.SampleCount=6&MetricData.member.6.StatisticValues.Sum=80.6296348572&MetricData.member.6.Timestamp=20171208T045348Z&MetricData.member.6.Unit=used&Namespace=collectd&Version=2010-08-01&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAI3SXMZHF2VA2C3ZQ%2F20171208%2Feu-west-1%2Fmonitoring%2Faws4_request&X-Amz-Date=20171208T045448Z&X-Amz-Security-Token=FQoDYXdzEJ7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDDkcP%2FfrV5eskgmoBCK3A1Gcje6AkVMDVxvfs%2B8X1DVDKgls9XHgBDEiHMWwUFX2JL%2BYoZ3yy89VEA3U1wpxZUQHlzyyEKySG64qeH%2F5v2LqmKbJcGJGzPr%2BA1FIIhJHTKxYzkBYLeMgk8%2FLVKIMGVyGyId0szWRuiwtTjVI%2BC%2BxhHaVTColwmWUMyFCC7%2BQ%2FmRI5IlwW5HfnUCB7xhUaRFh7dmM3XRCHnV0a6XuCI8Wz0vKnhOwQd%2F1gzqiKnObKJsM%2BEdlVzyfgvo92QfJj1y8e6IKD6kzNwa6GwQRQV8BN%2FBaS6ZkLPsRRe5QegpC%2B3VFKmSLf7nh1O9mLjvHHk%2B3KG6ldZpom1tB9BBLXA3xQyeNyARpYi3JgQRkjhwmG4x33KVOVRxfLncVJ7A7C%2BBrAoDuDWPNcHissPQ1na7d9UUjzix2Jp%2F48uNRInTF8GVOFwdL5xz5lG6YRzdvLRqCwHx8P538EHWCMfNcofcI5GZW1A%2BoKHg8OZnQNEUPVy1MImGKa%2B%2BQPMUbpQ0j4sKDZxy0pzwwYDNbXc6QG14Ogrv5qSlEEtv2eJHauamIEGV5LoLSmATWfyE%2FNS%2F1nfDJGToSWSYo17Ko0QU%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=279543f249bad79601d436262752a18620b9dc0901ad118123e8e98de34b8344'

<

Sender MissingParameter The parameter MetricData.member.1.Dimensions.member.3.Value is required. The parameter MetricData.member.3.Dimensions.member.3.Value is required. 0785107f-dbd4-11e7-9d1e-45079be837f6
yimuniao commented 6 years ago

After change some codes like: def dimension_get_plugin_instance(dimension, args): dimension[args['name']] = args['value'] if args['value'] else "NONE"

Following error occurred, Could you remove the changes related to "unit". We can use another pull request to address "unit" issue.

[root@ip-172-31-33-105 dimensionplugins]# curl -i -v -connect-timeout 1 -m 3 -w %{http_code}:%{http_connect}:%{content_type}:%{time_namelookup}:%{time_redirect}:%{time_pretransfer}:%{time_connect}:%{time_starttransfer}:%{time_total}:%{speed_download} -A "collectd/1.0" 'https://monitoring.eu-west-1.amazonaws.com/?Action=PutMetricData&MetricData.member.1.Dimensions.member.1.Name=InstanceId&MetricData.member.1.Dimensions.member.1.Value=i-064072d9df139837d&MetricData.member.1.Dimensions.member.2.Name=FixedDimension&MetricData.member.1.Dimensions.member.2.Value=ALL&MetricData.member.1.Dimensions.member.3.Name=PluginInstance&MetricData.member.1.Dimensions.member.3.Value=NONE&MetricData.member.1.MetricName=cpu.percent.active&MetricData.member.1.StatisticValues.Maximum=0.4&MetricData.member.1.StatisticValues.Minimum=0.0&MetricData.member.1.StatisticValues.SampleCount=6&MetricData.member.1.StatisticValues.Sum=0.6001001001&MetricData.member.1.Timestamp=20171208T062404Z&MetricData.member.1.Unit=active&MetricData.member.2.Dimensions.member.1.Name=FixedDimension&MetricData.member.2.Dimensions.member.1.Value=ALL&MetricData.member.2.Dimensions.member.2.Name=PluginInstance&MetricData.member.2.Dimensions.member.2.Value=NONE&MetricData.member.2.MetricName=cpu.percent.active&MetricData.member.2.StatisticValues.Maximum=0.4&MetricData.member.2.StatisticValues.Minimum=0.0&MetricData.member.2.StatisticValues.SampleCount=6&MetricData.member.2.StatisticValues.Sum=0.6001001001&MetricData.member.2.Timestamp=20171208T062404Z&MetricData.member.2.Unit=active&MetricData.member.3.Dimensions.member.1.Name=InstanceId&MetricData.member.3.Dimensions.member.1.Value=i-064072d9df139837d&MetricData.member.3.Dimensions.member.2.Name=FixedDimension&MetricData.member.3.Dimensions.member.2.Value=ALL&MetricData.member.3.Dimensions.member.3.Name=PluginInstance&MetricData.member.3.Dimensions.member.3.Value=NONE&MetricData.member.3.MetricName=memory.percent.used&MetricData.member.3.StatisticValues.Maximum=7.55928484644&MetricData.member.3.StatisticValues.Minimum=7.54591602018&MetricData.member.3.StatisticValues.SampleCount=6&MetricData.member.3.StatisticValues.Sum=45.3329034338&MetricData.member.3.Timestamp=20171208T062404Z&MetricData.member.3.Unit=used&MetricData.member.4.Dimensions.member.1.Name=FixedDimension&MetricData.member.4.Dimensions.member.1.Value=ALL&MetricData.member.4.Dimensions.member.2.Name=PluginInstance&MetricData.member.4.Dimensions.member.2.Value=NONE&MetricData.member.4.MetricName=memory.percent.used&MetricData.member.4.StatisticValues.Maximum=7.55928484644&MetricData.member.4.StatisticValues.Minimum=7.54591602018&MetricData.member.4.StatisticValues.SampleCount=6&MetricData.member.4.StatisticValues.Sum=45.3329034338&MetricData.member.4.Timestamp=20171208T062404Z&MetricData.member.4.Unit=used&MetricData.member.5.Dimensions.member.1.Name=InstanceId&MetricData.member.5.Dimensions.member.1.Value=i-064072d9df139837d&MetricData.member.5.Dimensions.member.2.Name=FixedDimension&MetricData.member.5.Dimensions.member.2.Value=ALL&MetricData.member.5.Dimensions.member.3.Name=PluginInstance&MetricData.member.5.Dimensions.member.3.Value=root&MetricData.member.5.MetricName=df.percent_bytes.used&MetricData.member.5.StatisticValues.Maximum=13.4494009018&MetricData.member.5.StatisticValues.Minimum=13.4492530823&MetricData.member.5.StatisticValues.SampleCount=6&MetricData.member.5.StatisticValues.Sum=80.6961593628&MetricData.member.5.Timestamp=20171208T062404Z&MetricData.member.5.Unit=used&MetricData.member.6.Dimensions.member.1.Name=FixedDimension&MetricData.member.6.Dimensions.member.1.Value=ALL&MetricData.member.6.Dimensions.member.2.Name=PluginInstance&MetricData.member.6.Dimensions.member.2.Value=root&MetricData.member.6.MetricName=df.percent_bytes.used&MetricData.member.6.StatisticValues.Maximum=13.4494009018&MetricData.member.6.StatisticValues.Minimum=13.4492530823&MetricData.member.6.StatisticValues.SampleCount=6&MetricData.member.6.StatisticValues.Sum=80.6961593628&MetricData.member.6.Timestamp=20171208T062404Z&MetricData.member.6.Unit=used&Namespace=collectd&Version=2010-08-01&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJOFEYYE5BXR5KEAA%2F20171208%2Feu-west-1%2Fmonitoring%2Faws4_request&X-Amz-Date=20171208T062504Z&X-Amz-Security-Token=FQoDYXdzEJ%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDKI4S78sY4GhxnK6LCK3AxIVULxUO4T%2B97BZZiFbp8BVxQbDAzip96hpYNREGJe4VnN%2FFvqZ8nNCJBqiJnXy9KXVr4uIQVg%2BtmPs5tzHxChVNzJ%2FI%2BmArKXJvfOIe2K%2FzQiPKz52Zi4EKMjgiPOXhp%2B9oyNELcrZlMMWYwQRVxFoxd86fb6QqilVgV%2BovDau9g6E3ys7LCjhPvE1wAZMO4skweJWITn9ctvMka9p8CQZjD2851vfV3SqKokrQH6f9C3SogB86srTc8PYnHRbehqzm2CPHekZjU5TufC%2FmHA39qYqbwgg5Ly09%2FwIp1%2Fah7Gxt69Feh%2FsBgt2j4Rqbk8YSMbzn7OWC5zTaUt051rlRPQdgHjXPflNML5DEtkZqUtmaNJiBdcZs2pE349RM9GIM9C0xOkLxI0pCzg6%2BsPsCs4x8qZ0p1M%2BesmFbHQ8DLvmR1p0w4Rg7cWvLtkTv4SiwKSzH9T6VUl1MDOXHpJDwLjiQY5SIygufTzQF26ZIcX4mvi6zPurPYA5vwPYFXI7LOIUMKYpySbXz4hnWy3y42%2FOpHi8fCJx%2FUMzfIqu4S12wtraaA5mYPrkd7So07Z9fhP81cIo%2Fdyo0QU%3D&X-Amz-SignedHeaders=host&X-Amz-Signature=f927c3933b868f1a4b53eaf8b93af96b3b508db55ac7e2672dc86fb8118b42c2'

<

Sender InvalidParameterValue The parameter MetricData.member.1.Unit must be a value in the set [ Seconds, Microseconds, Milliseconds, Bytes, Kilobytes, Megabytes, Gigabytes, Terabytes, Bits, Kilobits, Megabits, Gigabits, Terabits, Percent, Count, Bytes/Second, Kilobytes/Second, Megabytes/Second, Gigabytes/Second, Terabytes/Second, Bits/Second, Kilobits/Second, Megabits/Second, Gigabits/Second, Terabits/Second, Count/Second, None ]. The parameter MetricData.member.2.Unit must be a value in the set [ Seconds, Microseconds, Milliseconds, Bytes, Kilobytes, Megabytes, Gigabytes, Terabytes, Bits, Kilobits, Megabits, Gigabits, Terabits, Percent, Count, Bytes/Second, Kilobytes/Second, Megabytes/Second, Gigabytes/Second, Terabytes/Second, Bits/Second, Kilobits/Second, Megabits/Second, Gigabits/Second, Terabits/Second, Count/Second, None ]. The parameter MetricData.member.3.Unit must be a value in the set [ Seconds, Microseconds, Milliseconds, Bytes, Kilobytes, Megabytes, Gigabytes, Terabytes, Bits, Kilobits, Megabits, Gigabits, Terabits, Percent, Count, Bytes/Second, Kilobytes/Second, Megabytes/Second, Gigabytes/Second, Terabytes/Second, Bits/Second, Kilobits/Second, Megabits/Second, Gigabits/Second, Terabits/Second, Count/Second, None ]. The parameter MetricData.member.4.Unit must be a value in the set [ Seconds, Microseconds, Milliseconds, Bytes, Kilobytes, Megabytes, Gigabytes, Terabytes, Bits, Kilobits, Megabits, Gigabits, Terabits, Percent, Count, Bytes/Second, Kilobytes/Second, Megabytes/Second, Gigabytes/Second, Terabytes/Second, Bits/Second, Kilobits/Second, Megabits/Second, Gigabits/Second, Terabits/Second, Count/Second, None ]. The parameter MetricData.member.5.Unit must be a value in the set [ Seconds, Microseconds, Milliseconds, Bytes, Kilobytes, Megabytes, Gigabytes, Terabytes, Bits, Kilobits, Megabits, Gigabits, Terabits, Percent, Count, Bytes/Second, Kilobytes/Second, Megabytes/Second, Gigabytes/Second, Terabytes/Second, Bits/Second, Kilobits/Second, Megabits/Second, Gigabits/Second, Terabits/Second, Count/Second, None ]. The parameter MetricData.member.6.Unit must be a value in the set [ Seconds, Microseconds, Milliseconds, Bytes, Kilobytes, Megabytes, Gigabytes, Terabytes, Bits, Kilobits, Megabits, Gigabits, Terabits, Percent, Count, Bytes/Second, Kilobytes/Second, Megabytes/Second, Gigabytes/Second, Terabytes/Second, Bits/Second, Kilobits/Second, Megabits/Second, Gigabits/Second, Terabits/Second, Count/Second, None ]. 8f7fe4ca-dbe0-11e7-9db4-35f988bffdf0
linshu commented 6 years ago

@yimuniao I have fixed all the issues. Please review it.

Thanks.

garrettwilkin commented 4 years ago

I guess that this issue is pretty old now, but I'd love to have this feature.