search

minio / operator

Simple Kubernetes Operator for MinIO clusters :computer:
https://min.io/docs/minio/kubernetes/upstream/index.html
GNU Affero General Public License v3.0
1.18k stars 448 forks source link

Provisioned Tenant from Operator Istio sidecar issues #1995

Open joelcomp1 opened 6 months ago

joelcomp1 commented 6 months ago

If I have istio sidecars on for both Minio Operator and Tenant objects, whent he Tenant pods get created the validate-arguments init container fails because the mTLS tunnel isn't up yet. This maybe more of a feature request then a bug (or just documentation) can the validate-arguments init container be disabled or re-configured when istio sidecars are injected?

Expected Behavior

Tenant pods should come up with Istio enabled

Current Behavior

validate-arguments crash loop because they can't connect to the pod

Possible Solution

Disable init containers on Tenant deployments when istio sidecars are injected (manually via Tenant or Operator config)

Steps to Reproduce (for bugs)

  1. Install minio operator with namespace labeled as istio-injection=enabled
  2. Install a tenant CRD in namespace with istio-injection=enabled

Context

Trying to run Minio with Istio mTLS

Regression

No

Your Environment

fouadsemaan commented 6 months ago

Since you are running operator and tenant on Istio, do you still have pre-packaged minio tls enabled? Did you turn it off? Also do you have authorization policy set to allow operator namespace to link to tenant namespace?

joelcomp1 commented 6 months ago

I do still have it enabled but per all these old issues that fixed service labels I assumed that was OK: https://github.com/minio/operator/issues/749

The issue really is the way the init containers work with the service mesh due to the the proxy not being up to deal with the traffic. I can't tell from the tenant if the init container is required or of it could optionally be disabled. Maybe I just have to deal with this until SidecarContainers feature gate is GA in K8s

DanSalt commented 1 month ago

We're also observing the same thing as we try to upgrade to the latest major version of Minio.

I guess fundamentally the question is - is Operator-based Minio designed to work with a Service Mesh like Istio?

Our initial attempts were around using the "special" Istio user for the InitContainer. But that causes issues in that the configuration file is then owned by the wrong user, which causes issues later when running the sidecar.

Just curious how other Service Mesh users are working around this.

ramondeklein commented 1 month ago

What is preventing MinIO from running inside the service mesh? From what I read in this topic, I understand that the initialization pods are scheduled before the Istio sidecar is injected and that causes the pod to fail. If you can share logging that shows what is actually failing, then that may help.

This is the first topic I've seen in months where a service mesh is used (and doesn't work), so demand for it is low. Therefor, we probably won't officially support it. If we can implement a simple fix to make it work, then please submit a PR. It's not that we don't want to support it, but resources are limited.