Open thesmith opened 9 years ago
Clearly I need to write a way to replicate this.
According to Netflix, this could be because of a really small value: https://github.com/Netflix/servo/blob/master/servo-aws/src/test/java/com/netflix/servo/publish/cloudwatch/CloudWatchValueTest.java
0 values are allowed by CloudWatch since recording a 0 (versus recording no data point at all) can be significant.
I tried to tease out what this could be with the aws cli (aws cloudwatch put-metric-data ...), but since 0 values are allowed, I'm not sure what the actual problem is.
Having said all that this made me realize that if you wanted to record 0s (for counting behavior), you currently cannot, because the CloudWatchReporter omits 0 values to save money on API requests. So I'm opening an issue to make that configurable. #14
For reference, 0 values are allowed (via aws cli)
as a single data point
aws cloudwatch put-metric-data --namespace test --metric-data '[{"MetricName":"small_value","Value": 0}]'
as a statistic set
aws cloudwatch put-metric-data --namespace test --metric-data '[{"MetricName":"small_value","StatisticValues": {"SampleCount": 1, "Sum": 0, "Minimum": 0, "Maximum": 0}}]'
Data samples of 0 is not allowed, which is expected. That case is already handled by current CloudWatchReporter
aws cloudwatch put-metric-data --namespace test --metric-data '[{"MetricName":"small_value","StatisticValues": {"SampleCount": 0, "Sum": 1, "Minimum": 1, "Maximum": 1}}]'
Ok, using the Netflix link above, I was able to replicate the error once the positive small numbers got close to 0:
aws cloudwatch put-metric-data --namespace test --metric-data '[{"MetricName":"small_value","Value": 2.1289799200040754E-109}]'
You get that value by multiplying 1 by 0.5, keep multiplying current result by 0.5 another 360 times.
Evidently some of my metrics emit close enough to 0 numbers that AWS doesn't count them as valid.
I guess what we should do in this situation is just force the number to 0 if the number is sufficiently close to 0?
I'm seeing this as well.
I'm using the 0.3.3 through the dropwizard - cloudwatch integration, but it looks like the code to filter out empty data is the same across versions.
I'm recieving the following in my logs:
I'm assuming that the filter should also be checking that
input.getValue() > 0
as well as sample count?