search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.19k stars 4.87k forks source link

Horizontal pod autoscaler not able to get metrics in minikube deployment #9370

Closed adr-arroyo closed 3 years ago

adr-arroyo commented 3 years ago

Steps to reproduce the issue:

  1. $ minikube start — extra-config=controller-manager.horizontal-pod-autoscaler-upscale-delay=1m — extra-config=controller-manager.horizontal-pod-autoscaler-downscale-delay=1m — extra-config=controller-manager.horizontal-pod-autoscaler-sync-period=10s — extra-config=controller-manager.horizontal-pod-autoscaler-downscale-stabilization=1m
  2. $ minikube add-ons enable metrics-server
  3. Create .yaml with resource requests and limits:
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: orion
  name: orion
spec:
  replicas: 1
  selector:
    matchLabels:
      app: orion
  template:
    metadata:
      labels:
        app: orion
    spec:
      containers:
      - args:
        - -dbhost
        - mongo-db
        - -logLevel
        - DEBUG
        - -noCache
        name: fiware-orion
        image: fiware/orion:2.3.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 1026
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 200m
            memory: 0.5Gi
      restartPolicy: Always
  1. $ kubectl -n test-1 autoscale deployment orion --min=1 --max=5 --cpu-percent=50

Full output of failed command:

Command $ kubectl -n test-1 describe hpa orion returns: 

Name:                                                  orion
Namespace:                                             udp-test-1
Labels:                                                <none>
Annotations:                                           CreationTimestamp:  Thu, 01 Oct 2020 14:00:46 +0000
Reference:                                             Deployment/orion
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (0) / 20%
Min replicas:                                          1
Max replicas:                                          5
Deployment pods:                                       1 current / 1 desired
Conditions:
  Type            Status  Reason                   Message
  ----            ------  ------                   -------
  AbleToScale     True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive   False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  ScalingLimited  False   DesiredWithinRange       the desired count is within the acceptable range
Events:
  Type     Reason                        Age                   From                       Message
  ----     ------                        ----                  ----                       -------
  Warning  FailedComputeMetricsReplicas  39s (x12 over 4m27s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedGetResourceMetric       24s (x13 over 4m27s)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API

Command $ minikube addons list returns:

|-----------------------------|----------|--------------|
|         ADDON NAME          | PROFILE  |    STATUS    |
|-----------------------------|----------|--------------|
| ambassador                  | minikube | disabled     |
| dashboard                   | minikube | enabled ✅   |
| default-storageclass        | minikube | enabled ✅   |
| efk                         | minikube | disabled     |
| freshpod                    | minikube | disabled     |
| gvisor                      | minikube | disabled     |
| helm-tiller                 | minikube | disabled     |
| ingress                     | minikube | enabled ✅   |
| ingress-dns                 | minikube | disabled     |
| istio                       | minikube | disabled     |
| istio-provisioner           | minikube | disabled     |
| kubevirt                    | minikube | disabled     |
| logviewer                   | minikube | disabled     |
| metallb                     | minikube | disabled     |
| metrics-server              | minikube | enabled ✅   |
| nvidia-driver-installer     | minikube | disabled     |
| nvidia-gpu-device-plugin    | minikube | disabled     |
| olm                         | minikube | disabled     |
| pod-security-policy         | minikube | disabled     |
| registry                    | minikube | disabled     |
| registry-aliases            | minikube | disabled     |
| registry-creds              | minikube | disabled     |
| storage-provisioner         | minikube | enabled ✅   |
| storage-provisioner-gluster | minikube | disabled     |
|-----------------------------|----------|--------------|

As you may see in the commands output, even though it seems that the metrics server is working properly (metrics in hpa Orion say: resource cpu on pods (as a percentage of request): 0%), when it comes to the events produced by the Orion hpa there is an error regarding the computation of the metrics:

horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API

What is the reason for this horizontal pod autoscaler not working properly?

adr-arroyo commented 3 years ago

Isn't there anyone that can help me on this?

tstromberg commented 3 years ago

It seems quite likely that the answer here will be general to Kubernetes rather than specific to minikube.

That said, it seems unlikely, but is it possible that the metrics-server needs to be started first? I'm not quite sure how this is supposed to work. More likely that this is related to one of these issues, and may be indicative of a missing flag in either the controller or metrics server. There are some hints in these issues:

Please let me know what you discover!

marcphilipp commented 3 years ago

I have the same issue. @adr-arroyo Did you find a solution?

adr-arroyo commented 3 years ago

Unluckily I haven't @marcphilipp

If you happen to find a solution, please post it here :)

eddytruyen commented 3 years ago

You have to add the following clusterrolebinding 

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:heapster
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-system

This is clearly a minikube bug

And the existing clusterrole system:heapster is outdated so that no stats of statefulsets or nodes are possible. So execute

kubectl delete clusterrole system:heapster

and instead add the following clusterrole 

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:heapster
rules:
- apiGroups:
  - ""
  resources:
  - events
  - namespaces
  - nodes
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - deployments
  - statefulsets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
adr-arroyo commented 3 years ago

Thank you for your contribution @eddytruyen

I will try to test it on my environment

ishankhare07 commented 3 years ago

Hi @adr-arroyo , did you setup metrics server in the cluster before running this test? Try this https://github.com/kubernetes-sigs/metrics-server#installation

I've tried a test on kind before and can confirm that it does not include metrics server, and has to be installed separately. I think this will also be the case with minikube as well

medyagh commented 3 years ago

It seems quite likely that the answer here will be general to Kubernetes rather than specific to minikube.

That said, it seems unlikely, but is it possible that the metrics-server needs to be started first? I'm not quite sure how this is supposed to work. More likely that this is related to one of these issues, and may be indicative of a missing flag in either the controller or metrics server. There are some hints in these issues:

Please let me know what you discover!

I agree with thomas this seems to be a general kuberentes issue, If anyone find a root cause for this, please update and re-open this issue, but I think we can close this on minikube side for now.

allan-kg commented 1 year ago

Confirmed that it only works when you enable the module right after starting Minikube.

It happened to me today, even when attempting to use very basic stuff from Kubernetes' guide :

It kept showing the output of kubectl get hpaas :

NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/50%   1         10        0          14

Since the logs were empty, I had to look into the describe option, that gave me :

Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Fri, 20 Jan 2023 12:26:10 -0300
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: did not receive metrics for any ready pods
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetResourceMetric       60s                horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedComputeMetricsReplicas  60s                horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedGetResourceMetric       15s (x3 over 45s)  horizontal-pod-autoscaler  failed to get cpu utilization: did not receive metrics for any ready pods
  Warning  FailedComputeMetricsReplicas  15s (x3 over 45s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for any ready pods

If you wanna reproduce it, I have tried some other yaml files (not related to hpa) before trying to use it. Something like 20-30 creations and deletions. After that I tried to enable the addon after finding a reference to it on the internet (there was nothing about it on the guide itself).

Okay :/ , just after sending this comment, I tried to create some objects and now nothing works at all, not even the example from the guide after restarting minikube.