HPA (v1beta1) does not Display CloudWatch Custom Metric Data

[vijayansarathy]

I am trying to scale a service in EKS using a custom metric named http_requests_total in the ContainerInsights/Prometheus namespace in CloudWatch. The custom metric was collected in CloudWatch using the CloudWatch Agent for Prometheus. The metric represents a Prometheus Counter that tracks the number of HTTP requests.

I have verified that I can query CloudWatch for this custom metric using AWS CLI. I can see data for the custom metric using the command below:

aws cloudwatch get-metric-data \ --metric-data-queries file://json-data-query-http.json \ --start-time 2020-12-13T15:00:00Z \ --end-time 2020-12-13T16:00:00Z > output.txt

The JSON query and output files are found in the attached ZIP archive.

Next, I created an ExternalMetric resource with the following command exactly like the JSON query that I used with the above AWS CLI. kubectl apply -f metric-http-rate.yaml

Then, I setup a HorizontalPodAutoscaler (v1beta1) based on this ExternalMetric. kubectl apply -f hpa-http-v2beta1.yaml

However, the HPA does not seem to show the correct metric data at all. It always displays the value as unknown as shown in the output below:

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE http-rate-consumer-scaler Deployment/sqs-consumer /5 (avg) 1 5 1 7m32s http-rate-consumer-scaler Deployment/sqs-consumer /5 (avg) 1 5 1 7m46s http-rate-consumer-scaler Deployment/sqs-consumer /5 (avg) 1 5 1 8m1s http-rate-consumer-scaler Deployment/sqs-consumer /5 (avg) 1 5 1 8m17s

Any idea what is wrong with what I am doing? Thanks.

This ZIP file has all the artifacts that I have referenced in the command above: YAML-JSON-TXT-Files.zip

[vijayansarathy]

The issue was the missing unit attribute. The documentation says that "If you omit unit then all data that was collected with any unit is returned, along with the corresponding units that were specified when the data was reported to CloudWatch".

However, it looks like it is a required attribute and should be set to None. Otherwise, you get the following validation error:

client.go:79] err: ValidationError: 1 validation error detected: Value '' at 'metricDataQueries.1.member.metricStat.unit' failed to satisfy constraint: Member must satisfy enum value set: [Gigabits/Second, Gigabits, Percent, Terabits, Bits, Kilobits, Megabytes/Second, Kilobytes, Megabits/Second, Terabytes, Bytes, Count, Gigabytes, Milliseconds, Count/Second, Gigabytes/Second, None, Terabytes/Second, Bytes/Second, Megabits, Kilobytes/Second, Megabytes, Microseconds, Kilobits/Second, Terabits/Second, Bits/Second, Seconds]