Closed dominch closed 1 year ago
Do you see cpu metrics in kubectl top pod -A
?
What do the metrics-server pod logs say?
yes, metric servers seems to work correctly for all pods (-A) as well as for owncloud:
kubectl top pod -n owncloud
NAME CPU(cores) MEMORY(bytes)
my-cloud-owncloud-5cdb678c48-8bgbm 7m 193Mi
also whole prometheus works with no problem
Here is definition of this hpa:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
annotations:
meta.helm.sh/release-name: my-cloud
meta.helm.sh/release-namespace: owncloud
creationTimestamp: "2023-08-23T12:45:11Z"
labels:
app.kubernetes.io/instance: my-cloud
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: owncloud
app.kubernetes.io/version: 10.12.2
helm.sh/chart: owncloud-0.5.3
name: my-cloud-owncloud
namespace: owncloud
resourceVersion: "308826302"
uid: 703af1af-6927-413f-bcab-9ca4eab12a1d
spec:
maxReplicas: 3
metrics:
- resource:
name: cpu
target:
averageUtilization: 80
type: Utilization
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-cloud-owncloud
status:
conditions:
- lastTransitionTime: "2023-08-23T12:45:26Z"
message: the HPA controller was able to get the target's current scale
reason: SucceededGetScale
status: "True"
type: AbleToScale
- lastTransitionTime: "2023-08-23T12:45:26Z"
message: 'the HPA was unable to compute the replica count: failed to get cpu utilization:
missing request for cpu in container owncloud of Pod my-cloud-owncloud-5cdb678c48-8bgbm'
reason: FailedGetResourceMetric
status: "False"
type: ScalingActive
currentMetrics: null
currentReplicas: 1
desiredReplicas: 0
The status would appear to suggest that you haven't set a CPU request on the pod in question? You have to actually set resource requests and limits on your pods for the HPA to work...
message: 'the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu in container owncloud of Pod my-cloud-owncloud-5cdb678c48-8bgbm'
Thanks for that tip, I briefly checked that before and helm chart suggested that some limits should be in default configuration, but it was only example. I added those to deployment and HPA successfully got his value. Thanks again for pointing that out
Environmental Info: K3s Version:
Node(s) CPU architecture, OS, and Version: Mostly:
Linux kv92rs 5.10.0-24-amd64 #1 SMP Debian 5.10.179-5 (2023-08-08) x86_64 GNU/Linux
also bookworm and on some nodesCluster Configuration: 6 nodes, 3 masters
Describe the bug: Added owncloud via it's helm package, which added following hpa to cluster:
tagets reporting unknown cpu usage, so whole thing is not working at all, following events were recorded for this:
Metrics server seems to work correctly and everything os ok with it also with whole prometheus stack. The problem exists only here in hpa which cannot get those needed information. I also tried to add --insecure-tls flag to check if this is an issue, but it's not.
Steps To Reproduce: Probably to this will be easily reproduced with any hpa, the one I have is from helm repo:
and values (probably only last section is important here)
Expected behavior: HPA should work, should be able to retrieve cpu usage from metric server.
Actual behavior: HPA cannot get cpu usage from metric server.
Additional context / logs: Usually problem is caused by the lack of metrics server, here it's configured and works perfectly, so there have to be something that unables to communicate from hpa to it.