search

Azure-Samples / virtual-node-autoscale

A sample application to demonstrate Autoscale with AKS Virtual Nodes
MIT License
73 stars 40 forks source link

[Question] Prefer AKS agents over ACI #39

Open cnsgithub opened 5 years ago

cnsgithub commented 5 years ago

Hey guys,

I just stumbled upon virtual kubelet support in AKS and had the very same intention of first preferring regular AKS agent nodes over Azure Container Instances before I've found this sample project where readme.md says:

In order to control the Kubernetes scheduler to "favor" VM backed Kubernetes nodes BEFORE scaling out to the virtual node we can use a Kubernetes Webhook Admission Controller to add pod affinity and toleration key/values to all pods in a correctly labeled namespace.

I haven't used your admission controller yet, however specifying anti-affinity and tolerations in the pod specs manually should have the same effects.

Unfortunately, what I am experiencing when creating a deployment that tolerates the virtual-kubelet taint but prefers not getting scheduled to virtual-kubelet, is, that it's not carried out accordingly by the scheduler, i.e. it's being scheduled straight to virtual-kubelet and deploying a ACI pod instead of taking an AKS agent.

Somehow, despite using a weight of 100 in the pod spec for the anti-affinity, the priority for the AKS agents does not seem to be high enough to be scheduled there. If I understand correctly, the scheduler's priority calculation is very complex and there are many different algorithms besides node affinity that apply and will be combined: https://github.com/kubernetes/kubernetes/tree/master/pkg/scheduler/algorithm/priorities

Can you confirm that your example is (still) running as expected or am I doing something wrong?

Thanks.

rbitia commented 5 years ago

We actually took out the affinity controller because it was behaving strangely. Were you still deploying the controller or not?

cnsgithub commented 5 years ago

No, I just did some manual experiments... Did you encounter the same strange behavior or something else was wrong with it?

iMacX commented 3 years ago

Hi,

any update on this point? Is it possible to scale out pods on VM's first until they are full and then schedule pods on virtual node ?

thx for the feedback.

hasandogu commented 3 years ago

Is there any update on this issue? We would like to use the available VMs first before scheduling pods to the ACI as well. Is the admission-controller still the preferred way to do that?

Thank you

hasandogu commented 3 years ago

I actually created a discussion over at KEDA regarding an interesting autoscaling issue this might also be related with: https://github.com/kedacore/keda/discussions/1571