search

amazon-archives / k8s-cloudwatch-adapter

An implementation of Kubernetes Custom Metrics API for Amazon CloudWatch
Apache License 2.0
158 stars 98 forks source link

FailedGetExternalMetric when following "Sample Amazon SQS Guide" #83

Closed chay2199 closed 3 years ago

chay2199 commented 3 years ago

This is the error that I am getting when I use cmd kubectl describe hpa --namespace my-namespace

Name:                                                        my-scaler
Namespace:                                                my-namespace
Labels:                                                         <none>
Annotations:                                               <none>
CreationTimestamp:                                   Fri, 14 May 2021 16:03:13 +0530
Reference:                                                   Deployment/my-app
Metrics:                                                       ( current / target )
  "ccp-link-queue-length" (target average value):  <unknown> / 1
Min replicas:                                      1
Max replicas:                                      10
Deployment pods:                                   1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message

  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric my-namespace/my-queue-length/nil: unable to fetch metrics from external metrics API: no metric query found
Events:
  Type        Reason                             Age                    From                       Message

  Warning  FailedGetExternalMetric  11s (x898 over 3h50m)  horizontal-pod-autoscaler  unable to get external metric my-namespace/my-queue-length/nil: unable to fetch metrics from external metrics API: no metric query found

I am using my own sqs queue deployed on AWS and my own code to generate load. Tail of my CloudWatch-adapter logs are as follows:

I0514 13:55:32.604046 1 handler.go:67] externalMetricInfo: &{{ } {my-queue-length default /apis/metrics.aws/v1alpha1/namespaces/default/externalmetrics/my-queue-length 6d3e389a-096f-40af-a5d1-a35bb6b8ccf3 95202935 2 2021-05-13 20:54:29 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"metrics.aws/v1alpha1","kind":"ExternalMetric","metadata":{"annotations":{},"name":"my-queue-length","namespace":"default"},"spec":{"name":"my-queue-length","queries":[{"id":"sqs_ccplink","metricStat":{"metric":{"dimensions":[{"name":"QueueName","value":"my-request-queue-beta"}],"metricName":"NumberOfMessagesReceived","namespace":"AWS/SQS"},"period":60,"stat":"Average","unit":"Count"},"returnData":true}],"resource":{"resource":"deployment"}}} ] [] [] [{kubectl-client-side-apply Update metrics.aws/v1alpha1 2021-05-14 13:51:37 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 34 46 34 58 123 125 44 34 102 58 107 117 98 101 99 116 108 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 108 97 115 116 45 97 112 112 108 105 101 100 45 99 111 110 102 105 103 117 114 97 116 105 111 110 34 58 123 125 125 125 44 34 102 58 115 112 101 99 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 113 117 101 114 105 101 115 34 58 123 125 44 34 102 58 114 101 115 111 117 114 99 101 34 58 123 34 46 34 58 123 125 44 34 102 58 114 101 115 111 117 114 99 101 34 58 123 125 125 125 125],}}]} {my-queue-length <nil> <nil> [{ sqs_ccplink {{[{QueueName my-request-queue-beta}] NumberOfMessagesReceived AWS/SQS} 60 Average Count} 0xc00058bb2c}]}} I0514 13:55:32.604190 1 handler.go:68] adding to cache item 'my-queue-length' in namespace 'default' I0514 13:55:32.604200 1 controller.go:122] successfully processed item '{default/my-queue-length ExternalMetric}' I0514 13:55:32.604209 1 controller.go:79] processing next item I0514 13:55:32.604214 1 controller.go:86] processing item I0514 13:55:37.643111 1 provider_external.go:18] Received request for namespace: my-namespace, metric name: my-queue-length, metric selectors: I0514 13:55:37.643135 1 metric_cache.go:44] metric not found ExternalMetric/my-namespace/my-queue-length I0514 13:55:53.458856 1 provider_external.go:18] Received request for namespace: my-namespace, metric name: my-queue-length, metric selectors: I0514 13:55:53.458876 1 metric_cache.go:44] metric not found ExternalMetric/my-namespace/my-queue-length I0514 13:56:02.599194 1 controller.go:137] adding item to queue for 'default/my-queue-length' with kind 'ExternalMetric' I0514 13:56:02.604488 1 handler.go:54] processing item 'my-queue-length' in namespace 'default' I0514 13:56:02.604515 1 handler.go:67] externalMetricInfo: &{{ } {my-queue-length default /apis/metrics.aws/v1alpha1/namespaces/default/externalmetrics/my-queue-length 6d3e389a-096f-40af-a5d1-a35bb6b8ccf3 95202935 2 2021-05-13 20:54:29 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"metrics.aws/v1alpha1","kind":"ExternalMetric","metadata":{"annotations":{},"name":"my-queue-length","namespace":"default"},"spec":{"name":"my-queue-length","queries":[{"id":"sqs_ccplink","metricStat":{"metric":{"dimensions":[{"name":"QueueName","value":"my-request-queue-beta"}],"metricName":"NumberOfMessagesReceived","namespace":"AWS/SQS"},"period":60,"stat":"Average","unit":"Count"},"returnData":true}],"resource":{"resource":"deployment"}}} ] [] [] [{kubectl-client-side-apply Update metrics.aws/v1alpha1 2021-05-14 13:51:37 +0000 UTC FieldsV1 &FieldsV1{Raw:*[123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 34 46 34 58 123 125 44 34 102 58 107 117 98 101 99 116 108 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 108 97 115 116 45 97 112 112 108 105 101 100 45 99 111 110 102 105 103 117 114 97 116 105 111 110 34 58 123 125 125 125 44 34 102 58 115 112 101 99 34 58 123 34 46 34 58 123 125 44 34 102 58 110 97 109 101 34 58 123 125 44 34 102 58 113 117 101 114 105 101 115 34 58 123 125 44 34 102 58 114 101 115 111 117 114 99 101 34 58 123 34 46 34 58 123 125 44 34 102 58 114 101 115 111 117 114 99 101 34 58 123 125 125 125 125],}}]} {my-queue-length <nil> <nil> [{ sqs_ccplink {{[{QueueName my-request-queue-beta}] NumberOfMessagesReceived AWS/SQS} 60 Average Count} 0xc00058bb2c}]}} I0514 13:56:02.604626 1 handler.go:68] adding to cache item 'my-queue-length' in namespace 'default' I0514 13:56:02.604636 1 controller.go:122] successfully processed item '{default/my-queue-length ExternalMetric}' I0514 13:56:02.604646 1 controller.go:79] processing next item I0514 13:56:02.604651 1 controller.go:86] processing item I0514 13:56:08.471537 1 provider_external.go:18] Received request for namespace: my-namespace, metric name: my-queue-length, metric selectors: I0514 13:56:08.471559 1 metric_cache.go:44] metric not found ExternalMetric/my-namespace/my-queue-length

Output for external metrics

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/" {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"external.metrics.k8s.io/v1beta1","resources":[{"name":"my-queue-length","singularName":"","namespaced":true,"kind":"ExternalMetricValueList","verbs":["get"]}]}

I have given SQS and GetMetricsData access by adding a role to the SA of CloudWatch deployment. How can I know if the adapter is working properly or not? And why HPA is not able to find the external metrics?