minio / operator

Simple Kubernetes Operator for MinIO clusters :computer:
https://min.io/docs/minio/kubernetes/upstream/index.html
GNU Affero General Public License v3.0
1.19k stars 450 forks source link

Tenant stuck at provisioning initial users #2285

Closed aqeelat closed 1 month ago

aqeelat commented 1 month ago

I'm deploying the tenant using this values.yaml:

secrets:
tenant:
  name: minio
  image:
    repository: quay.io/minio/minio
    pullPolicy: IfNotPresent
  configuration:
    name: minio-env-configuration
  configSecret:
    name: minio-env-configuration
    existingSecret: true
  pools:
    - servers: 4
      storageClassName: xfs
  metrics:
    enabled: false
  buckets:
    - name: bucket1
    - name: bucket2
  users:
    - name: "minio-user-dev"

However, the cluster does not get fully initialized and throws this event Users creation failed: The Access Key Id you provided does not exist in our records.

Operator logs:

I0822 07:11:33.687999       1 minio.go:185] Generating private key
I0822 07:11:33.688056       1 minio.go:198] Generating CSR with CN=*.minio-hl.minio.svc.cluster.local
I0822 07:11:33.702789       1 csr.go:183] Start polling for certificate of csr/minio-minio-csr, every 5s, timeout after 20m0s
I0822 07:11:33.702882       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761320", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' MinIO CSR Created
I0822 07:11:38.707749       1 csr.go:209] Certificate successfully fetched, creating secret with Private key and Certificate
I0822 07:11:43.959644       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761492", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' MinIO Service Created
I0822 07:11:43.967932       1 status.go:55] Hit conflict issue, getting latest version of tenant
I0822 07:11:44.009715       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761499", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Console Service Created
I0822 07:11:44.020353       1 status.go:55] Hit conflict issue, getting latest version of tenant
I0822 07:11:44.117219       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761505", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Headless Service created
I0822 07:11:44.136501       1 sync.go:151] created key minio/minio-sa kind /v1, Kind=ServiceAccount diff [ObjectMeta.UID:  != ab0f04c2-995a-4b66-a988-46ef7f225229 ObjectMeta.ResourceVersion:  != 144761512 ObjectMeta.CreationTimestamp.Time: 0001-01-01 00:00:00 +0000 UTC != 2024-08-22 07:11:44 +0000 UTC ObjectMeta.OwnerReferences: <nil slice> != [{minio.min.io/v2 Tenant minio efcb0a18-41cb-4af6-a632-af2c1000c060 0xc001a3fef5 0xc001a3fef6}] ObjectMeta.ManagedFields: <nil slice> != [{minio-operator Update v1 2024-08-22 07:11:44 +0000 UTC FieldsV1 {"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"efcb0a18-41cb-4af6-a632-af2c1000c060\"}":{}}}} }]]
I0822 07:11:44.176652       1 sync.go:151] created key minio/minio-role kind rbac.authorization.k8s.io/v1, Kind=Role diff [ObjectMeta.UID:  != 1defb74a-7507-4657-b919-9bf3530a9710 ObjectMeta.ResourceVersion:  != 144761513 ObjectMeta.CreationTimestamp.Time: 0001-01-01 00:00:00 +0000 UTC != 2024-08-22 07:11:44 +0000 UTC ObjectMeta.OwnerReferences: <nil slice> != [{minio.min.io/v2 Tenant minio efcb0a18-41cb-4af6-a632-af2c1000c060 0xc000f14be5 0xc000f14be6}] ObjectMeta.ManagedFields: <nil slice> != [{minio-operator Update rbac.authorization.k8s.io/v1 2024-08-22 07:11:44 +0000 UTC FieldsV1 {"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"efcb0a18-41cb-4af6-a632-af2c1000c060\"}":{}}},"f:rules":{}} }]]
I0822 07:11:44.214944       1 sync.go:151] created key minio/minio-binding kind rbac.authorization.k8s.io/v1, Kind=RoleBinding diff [ObjectMeta.UID:  != 334ff367-5939-4d1f-8871-2c9a3809b4de ObjectMeta.ResourceVersion:  != 144761515 ObjectMeta.CreationTimestamp.Time: 0001-01-01 00:00:00 +0000 UTC != 2024-08-22 07:11:44 +0000 UTC ObjectMeta.OwnerReferences: <nil slice> != [{minio.min.io/v2 Tenant minio efcb0a18-41cb-4af6-a632-af2c1000c060 0xc001233565 0xc001233566}] ObjectMeta.ManagedFields: <nil slice> != [{minio-operator Update rbac.authorization.k8s.io/v1 2024-08-22 07:11:44 +0000 UTC FieldsV1 {"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"efcb0a18-41cb-4af6-a632-af2c1000c060\"}":{}}},"f:roleRef":{},"f:subjects":{}} }]]
I0822 07:11:44.230981       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0822 07:11:44.270291       1 main-controller.go:985] Detected we are updating a legacy tenant deployment
I0822 07:11:44.473768       1 main-controller.go:1030] 'minio/minio': Deploying pool pool-0
I0822 07:11:44.687930       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761529", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant pool pool-0 created
I0822 07:11:45.672032       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761591", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:11:48.714573       1 minio-services.go:156] Headless Services don't match: service ports don't match
I0822 07:11:48.724080       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761644", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated
I0822 07:11:48.767165       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761644", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:11:53.797707       1 minio-services.go:156] Headless Services don't match: service ports don't match
I0822 07:11:53.803247       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761732", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated
I0822 07:11:53.846845       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761732", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:11:58.863896       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761830", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:01.068455       1 monitoring.go:123] 'minio/minio' Failed to get cluster health: Get "https://minio.minio.svc.cluster.local/minio/health/cluster": dial tcp 10.57.235.22:443: i/o timeout
I0822 07:12:03.896762       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144761940", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:08.924281       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762041", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:08.972778       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0822 07:12:09.007298       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:09.050063       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:10.148741       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:11.193489       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:12.235314       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:13.278542       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:13.989062       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762115", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:14.327557       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:15.372854       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:16.416350       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:17.463190       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:18.513950       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:19.061076       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762211", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:19.559816       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:20.604286       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:21.650320       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:22.694493       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:23.737728       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:24.134795       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762297", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:24.801100       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:25.853544       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:26.933422       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:27.983578       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:29.030070       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:29.211035       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762399", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:30.073877       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:31.119628       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:32.166078       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:33.208655       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:34.242599       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0822 07:12:34.271857       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:34.283145       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762489", FieldPath:""}): type: 'Warning' reason: 'WaitingMinIOIsHealthy' Waiting for MinIO to be ready: %!s(<nil>)
I0822 07:12:35.318628       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:36.367337       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:39.432785       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762577", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.
I0822 07:12:41.785686       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:44.569890       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762671", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.
I0822 07:12:49.709243       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762748", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.
I0822 07:12:50.340084       1 monitoring.go:166] 'minio/minio' Failed to get storage info: The Access Key Id you provided does not exist in our records.
I0822 07:12:54.701937       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762827", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.
I0822 07:12:59.804930       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762909", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.
I0822 07:13:04.833746       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio", UID:"efcb0a18-41cb-4af6-a632-af2c1000c060", APIVersion:"minio.min.io/v2", ResourceVersion:"144762986", FieldPath:""}): type: 'Warning' reason: 'UsersCreationFailed' Users creation failed: The Access Key Id you provided does not exist in our records.

Your Environment

ramondeklein commented 1 month ago

Please provide a valid values.yaml file, because the one that you submitted isn't valid:

I first applied the following namespace/secret that:

apiVersion: v1
kind: Namespace
metadata:
  name: my-minio
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: minio-env-configuration
  namespace: my-minio
data:
  config.env: ZXhwb3J0IE1JTklPX1NUT1JBR0VfQ0xBU1NfU1RBTkRBUkQ9IkVDOjIiCmV4cG9ydCBNSU5JT19CUk9XU0VSPSJvbiIKZXhwb3J0IE1JTklPX1JPT1RfVVNFUj0ibWluaW8iCmV4cG9ydCBNSU5JT19ST09UX1BBU1NXT1JEPSJtaW5pbzEyMyIK
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: minio-user-dev
  namespace: my-minio
data:
  CONSOLE_ACCESS_KEY: dGVzdHRlc3QK
  CONSOLE_SECRET_KEY: dGVzdHRlc3QK

Then ran helm install --namespace my-minio --create-namespace my-minio -f tenant.yaml minio-operator/tenant with the following tenant.yaml:

secrets: ~
tenant:
  name: minio-test
  image:
    repository: quay.io/minio/minio
    pullPolicy: IfNotPresent
  configuration:
    name: minio-env-configuration
  configSecret:
    name: minio-env-configuration
    existingSecret: true
  pools:
    - servers: 4
      name: pool-0
  metrics:
    enabled: false
  buckets:
    - name: bucket1
    - name: bucket2
  users:
    - name: "minio-user-dev"

It creates the tenant, buckets and user.