Open rbrigden opened 4 years ago
@rbrigden -- I had same problem, but fixed by adding configmaps
to k8s-cloudwatch-adapter-resource-reader
cluster role:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: k8s-cloudwatch-adapter-resource-reader
rules:
- apiGroups:
- ""
resources:
- namespaces
- pods
- services
- configmaps
verbs:
- get
- list
Obviously, this still needs to be fixed in the manifest.
the usage of configmap had been removed, which version are you using?
@chankh -- I'm using:
image: chankh/k8s-cloudwatch-adapter:v0.9.0
which matches: https://github.com/awslabs/k8s-cloudwatch-adapter/blob/master/deploy/adapter.yaml
@lanecm do you mind checking if your k8s-cloudwatch-adapter
deployment has reference to any configmaps? The reference to configmap was removed since 93feeca8ad3fc287ddcbb77edb32d782645d839d
@chankh -- Oh, good catch... Yes, I added a kustomize
patch to set AWS_REGION
via a configmap we have in the cluster, e.g.,
- op: add
path: /spec/template/spec/containers/0/env
value:
- name: AWS_REGION
valueFrom:
configMapKeyRef:
key: AWS_REGION
name: env-region-config
Is AWS_REGION
required?
By default the adapter will try to get the region from EC2 metadata API, unless you are getting metrics from a different region, you should not need to specify AWS_REGION
@chankh -- I still get the following error:
F0813 11:32:12.206953 1 adapter.go:104] unable to run CloudWatch metrics adapter: unable to load configmap based request-header-client-ca-file: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:custom-metrics:k8s-cloudwatch-adapter" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
@lanecm seems like the adapter is trying to get configmap extension-apiserver-authentication
, where was this defined?
@chankh -- Searching for extension-apiserver-authentication
returns two items in our repo:
k8s-cloudwatch-adapter
:apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: k8s-cloudwatch-adapter-auth-reader
namespace: custom-metrics
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: k8s-cloudwatch-adapter
namespace: custom-metrics
metrics-server
:apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
labels:
app: metrics-server
chart: metrics-server-2.8.8
heritage: Tiller
release: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
Might be a problem with the identical named rolebindings?
@lanecm I've check the configs from other adapters as well, they all seems to be sharing the extension-apiserver-authentication-reader
. Also checked in my cluster, the role extension-apiserver-authentication-reader
has the policy rules to get/list/watch the extension-apiserver-authentication
configmap.
Name: extension-apiserver-authentication-reader
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
configmaps [] [extension-apiserver-authentication] [get list watch]
looks like the permissions should be granted there
@chankh I am also getting same error with eks 1.14
E0828 20:08:20.342746 1 reflector.go:153] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:custom-metrics:k8s-cloudwatch-adapter" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
If I try to add configmaps permission to resources as @lanecm suggested to k8s-cloudwatch-adapter-resource-reader
I get following error
E0828 22:22:19.502214 1 reflector.go:309] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to watch *v1.ConfigMap: unknown (get configmaps)
Any ideas?
@patelronak I fixed it error message after grant extension-apiserver-authentication-reader permissions.
command: kubectl edit roles.rbac.authorization.k8s.io -n kube-system extension-apiserver-authentication-reader
rules:
- apiGroups:
- ""
resourceNames:
- extension-apiserver-authentication
resources:
- configmaps
verbs:
- get
+ - list
+ - watch
@acgs771126 thank you very much for that insight. I was able to get over that error but pods are still not scaling and I do not see any errors in any of the logs. I am seeing unknown
for targets when I look at hpa (sqs-consumer-scaler)
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
sqs-consumer-scaler Deployment/sqs-consumer <unknown>/30 (avg) 1 10 0 2d19h
kubectl -n custom-metrics logs k8s-cloudwatch-adapter-59bf47d556-ntsj5
is showing that sqs_helloworld_length values are computed correctly
I0831 16:36:46.328608 1 handler.go:69] externalMetricInfo: &{{ } {sqs-helloworld-length default /apis/metrics.aws/v1alpha1/namespaces/default/externalmetrics/sqs-helloworld-length 87f56031-e893-11ea-8a2f-0e0c91594855 183392849 1 2020-08-27 18:31:32 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"metrics.aws/v1alpha1","kind":"ExternalMetric","metadata":{"annotations":{},"name":"sqs-helloworld-length","namespace":"default"},"spec":{"name":"sqs-helloworld-length","queries":[{"id":"sqs_helloworld_length","metricStat":{"metric":{"dimensions":[{"name":"QueueName","value":"helloworld"}],"metricName":"ApproximateNumberOfMessagesVisible","namespace":"AWS/SQS"},"period":60,"stat":"Average","unit":"Count"},"returnData":true}],"resource":{"resource":"deployment"}}}
] [] [] []} {sqs-helloworld-length [{ sqs_helloworld_length {{[{QueueName helloworld}] ApproximateNumberOfMessagesVisible AWS/SQS} 60 Average Count} true}]}}
I0831 16:36:46.328717 1 handler.go:114] adding to cache item 'sqs-helloworld-length' in namespace 'default'
I0831 16:36:46.328726 1 controller.go:122] successfully processed item '{default/sqs-helloworld-length ExternalMetric}'
I0831 16:36:46.328735 1 controller.go:79] processing next item
I0831 16:36:46.328744 1 controller.go:86] processing itemI0831 16:36:46.328608 1 handler.go:69] externalMetricInfo: &{{ } {sqs-helloworld-length default /apis/metrics.aws/v1alpha1/namespaces/default/externalmetrics/sqs-helloworld-length 87f56031-e893-11ea-8a2f-0e0c91594855 183392849 1 2020-08-27 18:31:32 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"metrics.aws/v1alpha1","kind":"ExternalMetric","metadata":{"annotations":{},"name":"sqs-helloworld-length","namespace":"default"},"spec":{"name":"sqs-helloworld-length","queries":[{"id":"sqs_helloworld_length","metricStat":{"metric":{"dimensions":[{"name":"QueueName","value":"helloworld"}],"metricName":"ApproximateNumberOfMessagesVisible","namespace":"AWS/SQS"},"period":60,"stat":"Average","unit":"Count"},"returnData":true}],"resource":{"resource":"deployment"}}}
] [] [] []} {sqs-helloworld-length [{ sqs_helloworld_length {{[{QueueName helloworld}] ApproximateNumberOfMessagesVisible AWS/SQS} 60 Average Count} true}]}}
I0831 16:36:46.328717 1 handler.go:114] adding to cache item 'sqs-helloworld-length' in namespace 'default'
I0831 16:36:46.328726 1 controller.go:122] successfully processed item '{default/sqs-helloworld-length ExternalMetric}'
I0831 16:36:46.328735 1 controller.go:79] processing next item
I0831 16:36:46.328744 1 controller.go:86] processing item
NAME READY UP-TO-DATE AVAILABLE AGE
sqs-consumer 1/1 1 1 14m
The hpa config looks correct to me
apiVersion: v1
items:
- apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
annotations:
autoscaling.alpha.kubernetes.io/metrics: '[{"type":"External","external":{"metricName":"sqs-helloworld-length","targetAverageValue":"30"}}]'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"sqs-consumer-scaler","namespace":"default"},"spec":{"maxReplicas":10,"metrics":[{"external":{"metricName":"sqs-helloworld-length","targetAverageValue":30},"type":"External"}],"minReplicas":1,"scaleTargetRef":{"apiVersion":"apps/v1beta1","kind":"Deployment","name":"sqs-consumer"}}}
creationTimestamp: "2020-08-28T21:23:41Z"
name: sqs-consumer-scaler
namespace: default
resourceVersion: "183746041"
selfLink: /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/sqs-consumer-scaler
uid: beef9d00-e974-11ea-8a2f-0e0c91594855
spec:
maxReplicas: 10
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: sqs-consumer
status:
currentReplicas: 0
desiredReplicas: 0
kind: List
metadata:
resourceVersion: ""
selfLink: ""
I was able to resolve the above error by using helm deployment instead of the adapter.yaml file deployment for the adapter.
what is your external metric setting? and also you can kubectl describe hpa to checking why hpa get unknown status.
When I check the logs with
kubectl -n custom-metrics logs -l=app=k8s-cloudwatch-adapter
, the following error message is firing off repeatedly