search

zilliztech / milvus-operator

The Kubernetes Operator of Milvus.
https://milvus.io
Apache License 2.0
33 stars 20 forks source link

How to config rollingUpdate strategy #112

Open douglarek opened 1 month ago

douglarek commented 1 month ago

Is there a way to configure the rollingUpdate strategy for the querynode deployment separately in the milvus-operator? In cases where there are a large number of queryNode pods with high memory usage, a maxSurge of 25% causes significant resource fluctuations in the cluster.

douglarek commented 1 month ago

Is there a way to configure the rollingUpdate strategy for the querynode deployment separately in the milvus-operator? In cases where there are a large number of queryNode pods with high memory usage, a maxSurge of 25% causes significant resource fluctuations in the cluster.

default settings in queryNode deployment:

strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
    type: RollingUpdate
douglarek commented 1 month ago

I noticed that in commit 242f4d49cce2d30593d6bbe8467648bc0b67c266 of version 0.9, it was changed to a more conservative 1. Can this parameter be exposed later?

haorenfsa commented 1 month ago

@douglarek No for now, but maybe yes in Milvus v2.4 or v2.5. That's mainly because the k8s deployment controller won't respect the terminating pods when rolling https://github.com/kubernetes/kubernetes/issues/107920, which is crucial for the rolling progress of Milvus. Please note that milvus-operator now uses 2 deployment for rolling. So the normal strategy for deployment won't work. The @zilliztech/milvus team insists on keep rolling 1 at a time for the time being in order to ensure the availability.