minio / operator

Simple Kubernetes Operator for MinIO clusters :computer:
https://min.io/docs/minio/kubernetes/upstream/index.html
GNU Affero General Public License v3.0
1.2k stars 450 forks source link

Regression: PR 2253 causes all-at-once tenant update #2332

Open mmulvanny opened 2 days ago

mmulvanny commented 2 days ago

Expected Behavior

Updating a tenant should cause pods to update in a rolling fashion, and the MinIO service should remain available at all times.

Current Behavior

Updating to 6.0.3 causes the operator to delete all tenant pods at once, causing a MinIO outage.

Possible Solution

Steps to Reproduce (for bugs)

  1. Deploy MinIO tenant with multiple pods managed by operator version 5.0.15
  2. Upgrade operator to 6.0.3

Context

We upgraded the MinIO operator from 5.0.15 to 6.0.3.

Regression

This was caused by the combination of PR 2221 (which moved environment configuration to a sidecar) and PR 2253 (which deleted pods on configuration changes). Was PR 2253 intended to remove rolling updates?

Your Environment

harshavardhana commented 2 days ago

There is no such thing as rolling updates @mmulvanny in our operator. We always employ in-place updates of the container binary and then subsequently statefulset rolls the changes.

However the cluster itself must be online way before this, please share the operator logs and let us make the right assessment on what happened.

Thanks

cesnietor commented 2 days ago

For upgrading to version 6 we have documentation since there are breaking changes: please see: https://min.io/docs/minio/kubernetes/eks/operations/install-deploy-manage/upgrade-minio-operator.html#upgrade-minio-operator-5-0-15-to-operator-version-stable