[Bug]: An milvus-operator error with updating milvus cluster：You must provide your current passwords when upgrading the release

[zuoyg]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:2.4.0 to 2.4.13
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):   pulsar 
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): centos
- CPU/Memory: 32C128G * 6
- GPU: NO
- Others:

Current Behavior

when i update milvuscluster search-milvus-hba from 2.4.0 to 2.4.13, milvus-operator problem occurred：“2024-10-16T02:37:51Z INFO controller.milvus reconcileAll {"err": "reconcile milvus: run group failed: groups error: execution error at (etcd/templates/NOTES.txt:114:4): \nPASSWORDS ERROR: You must provide your current passwords when upgrading the release.\n”

Expected Behavior

finish update

Steps To Reproduce

1. edit milvus-operator deployment :image to v1.0.1, complete upgrade。
2. edit milvuscluster image from 2.4.0 to 2.4.13, the cluster has been not upgrade.

Milvus Log

2024-10-16T03:03:07Z INFO controller.milvus reconcileAll {"err": "reconcile milvus: run group failed: groups error: execution error at (etcd/templates/NOTES.txt:114:4): \nPASSWORDS ERROR: You must provide your current passwords when upgrading the release.\n Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.\n Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases\n\n 'auth.rbac.rootPassword' must not be empty, please add '--set auth.rbac.rootPassword=$ETCD_ROOT_PASSWORD' to the command. To get the current value:\n\n export ETCD_ROOT_PASSWORD=$(kubectl get secret --namespace \"search-milvus-hba\" search-hba-etcd -o jsonpath=\"{.data.etcd-root-password}\" | base64 --decode)\n"} 2024-10-16T03:03:07Z ERROR Reconciler error {"controller": "milvus", "controllerGroup": "milvus.io", "controllerKind": "Milvus", "Milvus": {"name":"search-hba","namespace":"search-milvus-hba"}, "namespace": "search-milvus-hba", "name": "search-hba", "reconcileID": "b8ee0780-2499-4cb6-8d6f-84736a86121b", "error": "reconcile milvus: run group failed: groups error: execution error at (etcd/templates/NOTES.txt:114:4): \nPASSWORDS ERROR: You must provide your current passwords when upgrading the release.\n Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.\n Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases\n\n 'auth.rbac.rootPassword' must not be empty, please add '--set auth.rbac.rootPassword=$ETCD_ROOT_PASSWORD' to the command. To get the current value:\n\n export ETCD_ROOT_PASSWORD=$(kubectl get secret --namespace \"search-milvus-hba\" search-hba-etcd -o jsonpath=\"{.data.etcd-root-password}\" | base64 --decode)\n", "errorVerbose": "groups error: execution error at (etcd/templates/NOTES.txt:114:4): \nPASSWORDS ERROR: You must provide your current passwords when upgrading the release.\n Note that even after reinstallation, old credentials may be needed as they may be kept in persistent volume claims.\n Further information can be obtained at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases\n\n 'auth.rbac.rootPassword' must not be empty, please add '--set auth.rbac.rootPassword=$ETCD_ROOT_PASSWORD' to the command. To get the current value:\n\n export ETCD_ROOT_PASSWORD=$(kubectl get secret --namespace \"search-milvus-hba\" search-hba-etcd -o jsonpath=\"{.data.etcd-root-password}\" | base64 --decode)\n\ngithub.com/milvus-io/milvus-operator/pkg/controllers.(Group).Wait\n\t/workspace/pkg/controllers/groups.go:88\ngithub.com/milvus-io/milvus-operator/pkg/controllers.ParallelGroupRunner.Run\n\t/workspace/pkg/controllers/group_runner.go:59\ngithub.com/milvus-io/milvus-operator/pkg/controllers.(MilvusReconciler).ReconcileAll\n\t/workspace/pkg/controllers/milvus.go:43\ngithub.com milvus-operator.log

Anything else?

we have two milvuscluster : search-milvus-hba and search-milvus-hbb. This issue did not occur during previous upgrades, and this time the HBB cluster has also been upgraded normally.

[yanliang567]

/assign @haorenfsa please help to take a look /unassign

[haorenfsa]

@zuoyg Looks like somehow your etcd's deployment value field .auth.rbac.enabled is set to true, while milvus-operator haven't yet support this feature.

For walk around, you can try set the field .spec.dependencies.etcd.inCluster.Values.auth.rbac.enabled to false in your Milvus CR manifest, and apply that to your kubernetes.