SQS ExternalMetric HPA error

In my environment, I hav AWS EKS - 1.16 with metric server v0.5.0 and hpa. I have installed this v0.10.0 Cloudwatch adapter to the cluster under namespace custom-metrics

kubectl get --raw "/apis/external.metrics.k8s.io" | jq .

{
  "kind": "APIGroup",
  "apiVersion": "v1",
  "name": "external.metrics.k8s.io",
  "versions": [
    {
      "groupVersion": "external.metrics.k8s.io/v1beta1",
      "version": "v1beta1"
    }
  ],
  "preferredVersion": {
    "groupVersion": "external.metrics.k8s.io/v1beta1",
    "version": "v1beta1"
  }
}

I have attached IAM Role to the service-account as below:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: k8s-cloudwatch-adapter
  namespace: custom-metrics
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT>:role/myrole

Here myrole has AssumeRoleWithWebIdentityfor the OIDC with sub as system:serviceaccount:custom-metrics:k8s-cloudwatch-adapter

While i try to follow the steps as on scaling-kubernetes-deployments-with-amazon-cloudwatch-metrics, the HPA shows the below error and scaling does not happen.

ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric perf/hello-queue-length/nil: unable to fetch metrics from external metrics API: no metric query found

I see the below message on adapter logs

I0530 06:01:29.044363       1 handler.go:67] externalMetricInfo: &{{ } {sqs-helloworld-length  perf /apis/metrics.aws/v1alpha1/namespaces/perf/externalmetrics/sqs-helloworld-length a33356b9-11e9-49f9-8fd2-9666e63ae5fc 2426191 1 2021-05-30 05:34:04 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"metrics.aws/v1alpha1","kind":"ExternalMetric","kubectl get crd  queries":[{"id":"sqs_helloworld_length","metricStat":{"metric":{"dimensions":[{"name":"QueueName","value":"helloworld"}],"metricName":"ApproximateNumberOfMessagesVisible","namespace":"AWS/SQS"},"period":60,"stat":"Average","unit":"Count"},"returnData":true}],"metadata":{"annotations":{},"name":"sqs-helloworld-length","namespace":"perf"},"spec":{"name":"sqs-helloworld-length"}}
] [] []  []} {sqs-helloworld-length <nil> <nil> []}}
I0530 06:01:29.044430       1 handler.go:68] adding to cache item 'sqs-helloworld-length' in namespace 'perf'
I0530 06:01:29.044439       1 controller.go:122] successfully processed item '{perf/sqs-helloworld-length ExternalMetric}'
I0530 06:01:29.044448       1 controller.go:79] processing next item
I0530 06:01:29.044454       1 controller.go:86] processing item

1 metric_cache.go:44] metric not found ExternalMetric/perf/hello-queue-length 
httplog.go:90] GET /apis/external.metrics.k8s.io/v1beta1/namespaces/perf/hello-queue-length: (2.511725ms) 400 [kube-controller-manager/v1.16.15 (linux/amd64) kubernetes/cf75c96/system:serviceaccount:kube-system:horizontal-pod-autoscaler 10.240.146.186:60812]

Both HPA (sqs-consumer-scaler) and ExternalMetric (sqs-helloworld-length) are installed on the same namespace as app. Here in this case perf

What would be the reason for this failure?

amazon-archives / k8s-cloudwatch-adapter

SQS ExternalMetric HPA error #91