HPA & VPA are not compatible by design

[mossuchida]

Objective:

• Autoscale both vertically & horizontally to achieve the best resource utilization for kubernetes cluster by using HPA & VPA

Issue:

• Default Scheduler uses "resource request" to determine which pod to be deployed on which node, however, hardcoded "resource request" value might be wrong. VPA is capable of changing the "resource request" dynamically so that each pods will be routed to a node which has more resources. However, HPA also uses "resource request" value to determine the scale up/down & it needs fixed value, there is a conflict between HPA v.s. VPA & Default Scheduler combination.

Proposed Solution

• Current HPA users need "resource request" for HPA, by default, HPA should use "resource request", however, to support HPA + VPA, add an option to change the HPA base value from "resource request" to "resource limit"

Benefit:

• User is able to reschedule over-utilized pods to other nodes using VPA & Default scheduler. When resource utilization is reached to maximum performance of the pod & there is no more performance gained by the single pod, HPA will deploy additional pod. Default scheduler uses current pod resource request (real time resource request by VPA) to accurately schedule additional pod to unutilized node, thus, we are able to achieve the maximum resource utilization automatically. Scheduler-v2.pptx

[mossuchida]

I found another solution, closing this issue.

[sidheshdivekar29]

@mossuchida Do you mind sharing the solution.

[aviramha]

I found another solution, closing this issue.

I'd love to know too!

[mossuchida]

I used safe-scheduler instead : https://github.com/IBM/kube-safe-scheduler. It will monitor current resource util & create pod to proper node. I am not using VPA any more as it evicts pods every time & causes app downtime.

[kipusoep]

Looking for the same thing as well, but the mentioned kube-safe-scheduler seems abandoned 😔 Any other alternatives?

[mossuchida]

@kipusoep, This is the best solution, just that community is taking forever to review. What it does is to change resource of container without restarting the container. Which would enable HPA & VPA to work together : https://github.com/kubernetes/kubernetes/pull/102884

[mohammadkhavari]

Are there any updates or new solutions to use multi-dimensional auto-scaling? Would you please share? @mossuchida @kipusoep @aviramha

[aviramha]

hey @mohammadkhavari, I haven't found a solution but it's not relevant for me anymore. From a bit of Googling I did right now (as it's still interesting) I've found GKE is offering such feature: https://cloud.google.com/kubernetes-engine/docs/how-to/multidimensional-pod-autoscaling maybe they'll open source it later on :)