search

minio / operator

Simple Kubernetes Operator for MinIO clusters :computer:
https://min.io/docs/minio/kubernetes/upstream/index.html
GNU Affero General Public License v3.0
1.22k stars 455 forks source link

Operator canot talk to tenant because of: The Access Key Id you provided does not exist in our records #2320

Open sebhoss opened 1 month ago

sebhoss commented 1 month ago

Expected Behavior

The operator should be able to talk to its tenants using the secret provides in the spec.users array of a tenant.

Current Behavior

The operator logs the following error and fails to connect:

I0917 12:33:34.584792       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"f15c42e3-701a-4a48-95bb-fa753a346aca", APIVersion:"minio.min.io/v2", ResourceVersion:"2096771256", FieldPath:""}): type: 'Warning' reason: 'BucketsCreatedFailed' Buckets creation failed: The Access Key Id you provided does not exist in our records.

At first, I thought this was another case of https://github.com/minio/minio/issues/20120 but the access key does seem to exist and loki (the service using the minio storage) is able to connect just fine using the very same credentials that the operator is using so I'm sure that the credentials itself are fine. The file system inside the minio pods seems to agree:

$ ls -la /export0/data/.minio.sys/config/iam/users/
total 12
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 .
drwxr-sr-x 5 1000 1000 4096 Jun 26 07:34 ..
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 <redacted>

$ ls -la /export1/data/.minio.sys/config/iam/users/
total 12
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 .
drwxr-sr-x 5 1000 1000 4096 Jun 26 07:34 ..
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 <redacted>

$ ls -la /export2/data/.minio.sys/config/iam/users/
total 12
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 .
drwxr-sr-x 5 1000 1000 4096 Jun 26 07:34 ..
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 <redacted>

$ ls -la /export3/data/.minio.sys/config/iam/users/
total 12
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 .
drwxr-sr-x 5 1000 1000 4096 Jun 26 07:34 ..
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 <redacted>

The value is the same in each location and each pod. It is exactly the value of the CONSOLE_ACCESS_KEY key inside the storage-user secret that is referenced in the minio tenant. The tenant is not completely new but has been around for a while and the operator was previously able to connect to it and provision it correctly.

Possible Solution

N/A

Steps to Reproduce (for bugs)

  1. Take secret from https://github.com/minio/operator/blob/master/examples/kustomization/base/storage-user.yaml
  2. Deploy tenant that references that secret, e.g. https://github.com/minio/operator/blob/master/examples/kustomization/base/tenant.yaml
  3. ???
  4. See operator logs

Context

We use the operator to provision minio tenants. We only just updated the operator to v6 in this environment and were running v5 of the operator previously. I cannot delete and re-create this tenant because it contains production data which we do not want to lose.

Regression

Yes - I'm sure that older versions of the operator were able to connect just fine.

Your Environment

sebhoss commented 1 month ago

I just restarted the operator and now get what I previously described in https://github.com/minio/operator/issues/2238 ...

sebhoss commented 1 month ago

We just tried to provision a complete new tenant and got the same error:

I1002 13:26:22.525407       1 main-controller.go:986] Detected we are updating a legacy tenant deployment                                                                                                                                                                                             
I1002 13:26:22.726082       1 main-controller.go:1031] 'minio-tenant-loki/loki': Deploying pool pool-0                                                                                                                                                                                                
I1002 13:26:22.942871       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"bd2774d1-c5e4-4803-9af4-1dbec02a2033", APIVersion:"minio.min.io/v2", ResourceVersion:"303654799", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant poo
l pool-0 created
I1002 13:26:23.924634       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"bd2774d1-c5e4-4803-9af4-1dbec02a2033", APIVersion:"minio.min.io/v2", ResourceVersion:"303654834", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy'
 Waiting for MinIO to be ready: %!s(<nil>)
I1002 13:26:24.326941       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.86.18:443: connect: connection refused
I1002 13:26:25.343604       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp: lookup minio.minio-tenant-loki.svc.cluster.local on 10.24.64.10:53: no such host
I1002 13:26:26.362655       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.86.18:443: connect: connection refused
I1002 13:26:27.379977       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.86.18:443: connect: connection refused
I1002 13:26:28.397626       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.86.18:443: connect: connection refused
I1002 13:26:28.959847       1 minio-services.go:156] Headless Services don't match: service ports don't match
I1002 13:26:28.974730       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"bd2774d1-c5e4-4803-9af4-1dbec02a2033", APIVersion:"minio.min.io/v2", ResourceVersion:"303654913", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Servi
ce Updated
I1002 13:26:29.024133       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"bd2774d1-c5e4-4803-9af4-1dbec02a2033", APIVersion:"minio.min.io/v2", ResourceVersion:"303654913", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy'
 Waiting for MinIO to be ready: %!s(<nil>)
I1002 13:26:29.416118       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.86.18:443: connect: connection refused
I1002 13:26:30.434004       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.86.18:443: connect: connection refused
I1002 13:26:31.452460       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.86.18:443: connect: connection refused
I1002 13:26:32.484569       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.86.18:443: connect: connection refused
I1002 13:26:33.502608       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.86.18:443: connect: connection refused
I1002 13:26:39.611538       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.86.18:443: connect: connection refused
I1002 13:26:40.659456       1 monitoring.go:166] 'minio-tenant-loki/loki' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I1002 13:26:44.237884       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"bd2774d1-c5e4-4803-9af4-1dbec02a2033", APIVersion:"minio.min.io/v2", ResourceVersion:"303655127", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.
I1002 13:26:49.416264       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"bd2774d1-c5e4-4803-9af4-1dbec02a2033", APIVersion:"minio.min.io/v2", ResourceVersion:"303655202", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.
I1002 13:26:54.412388       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"bd2774d1-c5e4-4803-9af4-1dbec02a2033", APIVersion:"minio.min.io/v2", ResourceVersion:"303655255", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.
I1002 13:26:59.434339       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"bd2774d1-c5e4-4803-9af4-1dbec02a2033", APIVersion:"minio.min.io/v2", ResourceVersion:"303655323", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.
I1002 13:27:04.481406       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"bd2774d1-c5e4-4803-9af4-1dbec02a2033", APIVersion:"minio.min.io/v2", ResourceVersion:"303655376", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.

The tenant is configured to use RELEASE.2024-09-22T00-33-43Z and roughly looks like this:

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: loki
  namespace: minio-tenant-loki
  labels:
    app.kubernetes.io/name: minio-tenant
    app.kubernetes.io/instance: loki
spec:
  image: quay.io/minio/minio:RELEASE.2024-09-22T00-33-43Z
  imagePullPolicy: IfNotPresent
  configuration:
    name: storage-configuration
  users:
    - name: storage-user
  buckets:
    - name: chunks
    - name: ruler
    - name: admin
  env:
    - name: MINIO_PROMETHEUS_AUTH_TYPE
      value: "public"
    - name: MINIO_UPDATE
      value: "off"
  exposeServices:
    console: false
    minio: false
  serviceMetadata:
    minioServiceLabels:
      app.kubernetes.io/name: minio-tenant
      app.kubernetes.io/instance: loki
  pools:
    - servers: 1
      name: pool-0
      volumesPerServer: 1
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          storageClassName: csi-rbd
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 2Gi
      securityContext:
        fsGroup: 1000
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
      containerSecurityContext:
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: minio-tenant
              app.kubernetes.io/instance: loki
          nodeTaintsPolicy: Honor
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                topologyKey: kubernetes.io/hostname
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: minio-tenant
              weight: 100
  mountPath: /export
  subPath: /data
  requestAutoCert: false
  externalCertSecret:
    - name: tenant-certmanager-tls
      type: cert-manager.io/v1
  features:
    bucketDNS: false
    enableSFTP: false
  podManagementPolicy: Parallel

Looking at the /export/data folder we can see that no user was created (there is no users directory):

$ ls -la /export/data/.minio.sys/config/iam/
total 12
drwxr-sr-x 3 1000 1000 4096 Oct  2 13:26 .
drwxr-sr-x 4 1000 1000 4096 Oct  2 13:26 ..
drwxr-sr-x 2 1000 1000 4096 Oct  2 13:26 format.json
sebhoss commented 1 month ago

So steps for us to reproduce this are - which we can do 100% of the time:

  1. Apply the secrets storage-configuration and storage-user
  2. Apply the tenant
  3. Copy CA secret to operator namespace
  4. Look at operator logs

Log output again:

I1002 14:26:07.797099       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303701793", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' MinIO Servi
ce Created                                                                                                                                                                                                                                                                                            
I1002 14:26:07.807354       1 status.go:55] Hit conflict issue, getting latest version of tenant                                                                                                                                                                                                      
I1002 14:26:07.842477       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303701800", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Console Ser
vice Created                                                                                                                                                                                                                                                                                          
I1002 14:26:07.850876       1 status.go:55] Hit conflict issue, getting latest version of tenant                                                                                                                                                                                                      
I1002 14:26:07.876846       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303701807", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Headless Se
rvice created                                                                                                                                                                                                                                                                                         
I1002 14:26:07.937498       1 sync.go:151] created key minio-tenant-loki/loki-sa kind /v1, Kind=ServiceAccount diff [ObjectMeta.UID:  != eb5e7efa-b277-4d7c-943c-3db41b9baf54 ObjectMeta.ResourceVersion:  != 303701812 ObjectMeta.CreationTimestamp.Time: 0001-01-01 00:00:00 +0000 UTC != 2024-10-02
 14:26:07 +0000 UTC ObjectMeta.OwnerReferences: <nil slice> != [{minio.min.io/v2 Tenant loki 8656fcc2-4b6d-4954-bc5b-bf525cd36220 0xc001b0719c 0xc001b0719d}] ObjectMeta.ManagedFields: <nil slice> != [{minio-operator Update v1 2024-10-02 14:26:07 +0000 UTC FieldsV1 {"f:metadata":{"f:ownerRefere
nces":{".":{},"k:{\"uid\":\"8656fcc2-4b6d-4954-bc5b-bf525cd36220\"}":{}}}} }]]                                                                                                                                                                                                                        
I1002 14:26:07.960162       1 sync.go:151] created key minio-tenant-loki/loki-role kind rbac.authorization.k8s.io/v1, Kind=Role diff [ObjectMeta.UID:  != 2f81f8fd-6ed2-4227-8114-e57bf5b5136c ObjectMeta.ResourceVersion:  != 303701813 ObjectMeta.CreationTimestamp.Time: 0001-01-01 00:00:00 +0000 
UTC != 2024-10-02 14:26:07 +0000 UTC ObjectMeta.OwnerReferences: <nil slice> != [{minio.min.io/v2 Tenant loki 8656fcc2-4b6d-4954-bc5b-bf525cd36220 0xc001b075b4 0xc001b075b5}] ObjectMeta.ManagedFields: <nil slice> != [{minio-operator Update rbac.authorization.k8s.io/v1 2024-10-02 14:26:07 +0000
 UTC FieldsV1 {"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"8656fcc2-4b6d-4954-bc5b-bf525cd36220\"}":{}}},"f:rules":{}} }]]                                                                                                                                                                
I1002 14:26:07.985863       1 sync.go:151] created key minio-tenant-loki/loki-binding kind rbac.authorization.k8s.io/v1, Kind=RoleBinding diff [ObjectMeta.UID:  != 884b8fe0-c946-47e7-87ae-b748484c3f9d ObjectMeta.ResourceVersion:  != 303701815 ObjectMeta.CreationTimestamp.Time: 0001-01-01 00:00
:00 +0000 UTC != 2024-10-02 14:26:07 +0000 UTC ObjectMeta.OwnerReferences: <nil slice> != [{minio.min.io/v2 Tenant loki 8656fcc2-4b6d-4954-bc5b-bf525cd36220 0xc001b07c34 0xc001b07c35}] ObjectMeta.ManagedFields: <nil slice> != [{minio-operator Update rbac.authorization.k8s.io/v1 2024-10-02 14:2
6:07 +0000 UTC FieldsV1 {"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"8656fcc2-4b6d-4954-bc5b-bf525cd36220\"}":{}}},"f:roleRef":{},"f:subjects":{}} }]]                                                                                                                                    
I1002 14:26:08.135847       1 status.go:89] Hit conflict issue, getting latest version of tenant                                                                                                                                                                                                      
I1002 14:26:08.536697       1 main-controller.go:986] Detected we are updating a legacy tenant deployment                                                                                                                                                                                             
I1002 14:26:08.736339       1 main-controller.go:1031] 'minio-tenant-loki/loki': Deploying pool pool-0                                                                                                                                                                                                
I1002 14:26:08.951570       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303701827", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant poo
l pool-0 created                                                                                                                                                                                                                                                                                      
I1002 14:26:09.936817       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303701859", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy'
 Waiting for MinIO to be ready: %!s(<nil>)                                                                                                                                                                                                                                                            
I1002 14:26:10.338834       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp: lookup minio.minio-tenant-loki.svc.cluster.local on 10.24.64.10:53: no such host
I1002 14:26:11.356083       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp: lookup minio.minio-tenant-loki.svc.cluster.local on 10.24.64.10:53: no such host                      
I1002 14:26:12.374191       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp: lookup minio.minio-tenant-loki.svc.cluster.local on 10.24.64.10:53: no such host
I1002 14:26:13.390771       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp: lookup minio.minio-tenant-loki.svc.cluster.local on 10.24.64.10:53: no such host                      
I1002 14:26:14.409788       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp: lookup minio.minio-tenant-loki.svc.cluster.local on 10.24.64.10:53: no such host
I1002 14:26:14.970408       1 minio-services.go:156] Headless Services don't match: service ports don't match
I1002 14:26:14.979798       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303701941", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Servi
ce Updated
I1002 14:26:15.029857       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303701941", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy'
 Waiting for MinIO to be ready: %!s(<nil>)
I1002 14:26:15.428273       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp: lookup minio.minio-tenant-loki.svc.cluster.local on 10.24.64.10:53: no such host
I1002 14:26:16.446643       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp: lookup minio.minio-tenant-loki.svc.cluster.local on 10.24.64.10:53: no such host
I1002 14:26:17.464550       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.88.245:443: connect: connection refused
I1002 14:26:18.482739       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.88.245:443: connect: connection refused
I1002 14:26:19.501029       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.88.245:443: connect: connection refused
I1002 14:26:20.110702       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702019", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy'
 Waiting for MinIO to be ready: %!s(<nil>)
I1002 14:26:20.519296       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.88.245:443: connect: connection refused
I1002 14:26:20.643929       1 main-controller.go:1483] Secret 'minio-operator/operator-ca-tls-minio-tenant-loki' changed
I1002 14:26:20.644071       1 operator.go:215] Added certificates in field 'ca.crt' of 'minio-operator/operator-ca-tls-minio-tenant-loki' secret to trusted RootCA's
I1002 14:26:21.538662       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.88.245:443: connect: connection refused
I1002 14:26:22.555611       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.88.245:443: connect: connection refused
I1002 14:26:23.573236       1 monitoring.go:123] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.88.245:443: connect: connection refused
I1002 14:26:24.618866       1 monitoring.go:166] 'minio-tenant-loki/loki' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I1002 14:26:25.212875       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702093", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.
I1002 14:26:30.354643       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702157", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:26:35.395511       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702225", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:26:40.402459       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702282", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:26:45.435497       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702341", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:26:50.473951       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702398", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:26:55.513893       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702470", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:27:00.536729       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702527", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:27:05.577178       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702587", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:27:10.605627       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702653", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:27:15.638032       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702722", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:27:20.676762       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702778", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:27:25.706551       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702840", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.                                                                                                                                                                                                                   
I1002 14:27:30.737110       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303702898", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' U
sers creation failed: The Access Key Id you provided does not exist in our records.

Btw. why is it complaining about updating a legacy tenant deployment? This is a completely new tenant using the latest available apiVersion?

sebhoss commented 1 month ago

Ok so we downgraded the operator to the latest v5 version and it was able to provision the tenant just fine:

I1002 15:11:56.686637       1 controller.go:80] Starting MinIO Operator
I1002 15:11:56.705266       1 main-controller.go:323] Setting up event handlers
I1002 15:11:56.716631       1 main-controller.go:556] Using Kubernetes CSR Version: v1
I1002 15:11:56.716678       1 main-controller.go:576] STS Api server is not enabled, not starting
I1002 15:11:56.716705       1 leaderelection.go:250] attempting to acquire leader lease minio-operator/minio-operator-lock...
I1002 15:11:56.722349       1 main-controller.go:623] minio-operator-6fdd77bbf5-vdghm: is the leader, removing any leader labels that I 'minio-operator-698db9cfbc-mz5td' might have
I1002 15:12:04.797378       1 leaderelection.go:260] successfully acquired lease minio-operator/minio-operator-lock
I1002 15:12:04.797523       1 main-controller.go:605] minio-operator-698db9cfbc-mz5td: I am the leader, applying leader labels on myself
I1002 15:12:04.797786       1 main-controller.go:442] Waiting for Upgrade Server to start
I1002 15:12:04.797904       1 main-controller.go:446] Starting Tenant controller
I1002 15:12:04.797934       1 main-controller.go:449] Waiting for informer caches to sync
I1002 15:12:04.797974       1 main-controller.go:398] Starting HTTP Upgrade Tenant Image server
I1002 15:12:04.798013       1 main-controller.go:460] Starting workers and Job workers
I1002 15:12:04.798124       1 main-controller.go:496] Console TLS is not enabled
I1002 15:12:04.838881       1 operator.go:184] External secret found: operator-ca-tls-minio-tenant-loki
I1002 15:12:04.840279       1 operator.go:184] External secret found: operator-ca-tls-minio-tenant-loki
I1002 15:12:04.841017       1 minio-services.go:164] Headless Services don't match: service ports don't match
I1002 15:12:05.004371       1 operator.go:188] Appending cert from operator-ca-tls-minio-tenant-loki secret
I1002 15:12:05.204117       1 operator.go:188] Appending cert from operator-ca-tls-minio-tenant-loki secret
I1002 15:12:05.415614       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303737005", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated
I1002 15:12:05.615426       1 status.go:89] Hit conflict issue, getting latest version of tenant
I1002 15:12:05.643898       1 monitoring.go:170] 'minio-tenant-loki/loki' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I1002 15:12:05.815432       1 status.go:89] Hit conflict issue, getting latest version of tenant
I1002 15:12:05.836091       1 monitoring.go:170] 'minio-tenant-loki/loki' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I1002 15:12:06.810431       1 monitoring.go:122] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.88.245:443: connect: connection refused
I1002 15:12:22.204814       1 monitoring.go:122] 'minio-tenant-loki/loki' Failed to get cluster health: Get "https://minio.minio-tenant-loki.svc.cluster.local/minio/health/cluster": dial tcp 10.24.88.245:443: i/o timeout
I1002 15:12:22.258385       1 status.go:171] Hit conflict issue, getting latest version of tenant
I1002 15:12:22.276627       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303737005", FieldPath:""}): type: 'Warning' reason: 'UsersCreatedFailed' Users creation failed: Operation cannot be fulfilled on tenants.minio.min.io "loki": the object has been modified; please apply your changes to the latest version and try again
I1002 15:12:24.876889       1 minio-services.go:164] Headless Services don't match: service ports don't match
I1002 15:12:24.887229       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303737320", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated
I1002 15:12:25.175133       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303737320", FieldPath:""}): type: 'Normal' reason: 'UsersCreated' Users created
I1002 15:12:25.183632       1 helper.go:782] Successfully created bucket chunks
I1002 15:12:25.192439       1 helper.go:782] Successfully created bucket ruler
I1002 15:12:25.202146       1 helper.go:782] Successfully created bucket admin
I1002 15:12:25.210920       1 status.go:197] Hit conflict issue, getting latest version of tenant
I1002 15:12:25.231338       1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"8656fcc2-4b6d-4954-bc5b-bf525cd36220", APIVersion:"minio.min.io/v2", ResourceVersion:"303737320", FieldPath:""}): type: 'Normal' reason: 'BucketsCreated' Buckets created
I1002 15:12:25.240866       1 status.go:55] Hit conflict issue, getting latest version of tenant
I1002 15:12:25.271381       1 pdb.go:191] PodDisruptionBudget: v1

So we are pretty sure that this is a regression in v6 which currently blocks us from creating new tenants.