Lots of Helm release-related Secrets being created despite low release history setting in code

zilliztech / milvus-operator

The Kubernetes Operator of Milvus.

https://milvus.io

Apache License 2.0

33 stars 20 forks source link

Lots of Helm release-related Secrets being created despite low release history setting in code #145

Open bcbrockway opened 3 days ago

bcbrockway commented 3 days ago

I can see that the Helm release history is set to 2 here but we are still seeing an enormous number of Secrets being created in our Milvus namespaces which is triggering our quota alerts:

Any idea what might be going on here?

haorenfsa commented 2 days ago

Thank you for the feedback @bcbrockway. Could you provide your milvus CR & the logs of milvus-operator during that period?

bcbrockway commented 2 days ago

Hi @haorenfsa, as you can see from the logs, we had an immutable field issue with some StatefulSets at the time which we eventually resolved by orphan-deleting them and letting the milvus-operator reapply the releases, but I would still expect it to only keep 2 sh.helm.release.v1 secrets per StatefulSet.

Explore-logs-2024-07-03 08_54_03.txt

haorenfsa commented 1 day ago

@bcbrockway Looks like the helm pkg overwrite the MaxHistory with the value in Upgrade.MaxHistory struct when doing upgrade

I check the helm command's code, HelmSetting.MaxHistory is only useful when using helm command. When using it as a pkg, we should directly set Upgrade.HaxHistory. Otherwise it's considered as 0 (which means unlimited)