search

minio / operator

Simple Kubernetes Operator for MinIO clusters :computer:
https://min.io/docs/minio/kubernetes/upstream/index.html
GNU Affero General Public License v3.0
1.3k stars 473 forks source link

deploy minio tenant: Missing root credentials in the configuration #2442

Open Aragorn1026 opened 3 weeks ago

Aragorn1026 commented 3 weeks ago

There is a problem confused me serverl days, please help me, thank you ~

I have deployed a k8s 1.28.0 environment with 3 master nodes, and delete all taints on node.

When I deploy minio tenant, its yaml is:

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: minio-tenant
  namespace: minio
spec:
  pools:
  - servers: 3
    name: minio-pool
    volumesPerServer: 4
    volumeClaimTemplate:
      metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: directpv-min-io
  image: repo.my.cn/minio/minio:RELEASE.2025-04-22T22-12-26Z-cpuv1
  configuration:
    name: storage-configuration
  requestAutoCert: false
  externalCertSecret:
  - name: minio-tls
    type: kubernetes.io/tls

PV and PVCs are created normal, but the minio-tenant-pools pods run failed

[root@kylin-milvus-1 operator-6.0.4]# kubectl -n minio get pod -owide
NAME                        READY   STATUS                  RESTARTS         AGE     IP              NODE             NOMINATED NODE   READINESS GATES
minio-tenant-minio-pool-0   0/2     Init:Error              53 (5m5s ago)    4h5m    10.188.183.88   kylin-milvus-1   <none>           <none>
minio-tenant-minio-pool-1   0/2     Init:CrashLoopBackOff   268 (76s ago)    22h     10.188.7.92     kylin-milvus-2   <none>           <none>
minio-tenant-minio-pool-2   0/2     Init:CrashLoopBackOff   56 (2m30s ago)   4h23m   10.188.64.221   kylin-milvus-3   <none>           <none>
[root@kylin-milvus-1 minio-tenant]# kubectl -n minio logs minio-tenant-minio-pool-1 -c validate-arguments
2025/04/29 10:25:02 validator.go:79: Missing root credentials in the configuration.
2025/04/29 10:25:02 validator.go:80: MinIO won't start

The configuration is 

[root@kylin-milvus-1 minio-tenant]# kubectl -n minio get secret storage-configuration
NAME                    TYPE     DATA   AGE
storage-configuration   Opaque   1      26h
[root@kylin-milvus-1 minio-tenant]#
[root@kylin-milvus-1 minio-tenant]# kubectl -n minio get secret storage-configuration -oyaml
apiVersion: v1
data:
  config.env: TUlOSU9fUk9PVF9VU0VSPW1pbmlvCk1JTklPX1JPT1RfUEFTU1dPUkQ9WnpiYW5rQDQ1NgpNSU5JT19DT05TT0xFX1BBU1NXT1JEPVp6YmFua0AxMjMK
kind: Secret
metadata:
  creationTimestamp: "2025-04-29T06:06:29Z"
  name: storage-configuration
  namespace: minio
  resourceVersion: "1490510"
  uid: 3c77bc3e-e24a-419a-a900-201288bf44bb
type: Opaque
[root@kylin-milvus-1 minio-tenant]#

And the data.config.env is :

MINIO_ROOT_USER=XXX
MINIO_ROOT_PASSWORD=XXXX
MINIO_CONSOLE_PASWORD=XXXX

I deployED minio operator v6.0.4, the tar package from github:

https://github.com/minio/operator/archive/refs/tags/v6.0.4.tar.gz

BY:

kubectl creat -k operator-6.0.4

only changed minio operator's image to: repo.my.cn/minio/operator:v6.0.4 then the two minio operator run normal.

An the directpv runs normal:

[root@kylin-milvus-1 minio-tenant]# kubectl -n directpv get pod -owide
NAME                          READY   STATUS    RESTARTS   AGE    IP              NODE             NOMINATED NODE   READINESS GATES
controller-59459c7fbf-7j2hc   3/3     Running   0          2d3h   10.188.7.84     kylin-milvus-2   <none>           <none>
controller-59459c7fbf-7v6md   3/3     Running   0          2d3h   10.188.64.212   kylin-milvus-3   <none>           <none>
controller-59459c7fbf-xz96f   3/3     Running   0          2d3h   10.188.183.81   kylin-milvus-1   <none>           <none>
node-server-d44pz             4/4     Running   0          2d3h   10.188.64.213   kylin-milvus-3   <none>           <none>
node-server-mb5r5             4/4     Running   0          2d3h   10.188.183.82   kylin-milvus-1   <none>           <none>
node-server-mqgqt             4/4     Running   0          2d3h   10.188.7.85     kylin-milvus-2   <none>           <none>
[root@kylin-milvus-1 minio-tenant]#
[root@kylin-milvus-1 minio-tenant]# kubectl directpv info
┌──────────────────┬──────────┬───────────┬─────────┬────────┐
│ NODE             │ CAPACITY │ ALLOCATED │ VOLUMES │ DRIVES │
├──────────────────┼──────────┼───────────┼─────────┼────────┤
│ • kylin-milvus-1 │ 30 GiB   │ 20 GiB    │ 4       │ 1      │
│ • kylin-milvus-2 │ 30 GiB   │ 20 GiB    │ 4       │ 1      │
│ • kylin-milvus-3 │ 30 GiB   │ 20 GiB    │ 4       │ 1      │
└──────────────────┴──────────┴───────────┴─────────┴────────┘

60 GiB/90 GiB used, 12 volumes, 3 drives

And the minio tenant pool's sts is :

[root@kylin-milvus-1 minio-tenant]# kubectl -n minio get sts minio-tenant-minio-pool -oyaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2025-04-29T10:19:36Z"
  generation: 3
  labels:
    v1.min.io/pool: minio-pool
    v1.min.io/tenant: minio-tenant
  name: minio-tenant-minio-pool
  namespace: minio
  ownerReferences:
  - apiVersion: minio.min.io/v2
    blockOwnerDeletion: true
    controller: true
    kind: Tenant
    name: minio-tenant
    uid: 460e8692-f025-434b-9173-e78598333d18
  resourceVersion: "1777048"
  uid: 4a86a57b-040a-4e74-9142-9a013702f6ce
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      v1.min.io/console: minio-tenant-console
      v1.min.io/pool: minio-pool
      v1.min.io/tenant: minio-tenant
  serviceName: minio-tenant-hl
  template:
    metadata:
      annotations:
        min.io/revision: "0"
      creationTimestamp: null
      labels:
        v1.min.io/console: minio-tenant-console
        v1.min.io/pool: minio-pool
        v1.min.io/tenant: minio-tenant
    spec:
      containers:
      - args:
        - server
        - --certs-dir
        - /tmp/certs
        - --console-address
        - :9443
        env:
        - name: MINIO_CONFIG_ENV_FILE
          value: /tmp/minio/config.env
        image: repo.my.cn/minio/minio:RELEASE.2025-04-22T22-12-26Z-cpuv1
        imagePullPolicy: IfNotPresent
        name: minio
        ports:
        - containerPort: 9000
          protocol: TCP
        - containerPort: 9443
          protocol: TCP
        resources: {}
        securityContext:
          runAsGroup: 1000
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp/minio/
          name: cfg-vol
        - mountPath: /export0
          name: data0
        - mountPath: /export1
          name: data1
        - mountPath: /export2
          name: data2
        - mountPath: /export3
          name: data3
        - mountPath: /tmp/certs
          name: minio-tenant-tls
      - args:
        - sidecar
        - --tenant
        - minio-tenant
        - --config-name
        - storage-configuration
        env:
        - name: CLUSTER_DOMAIN
          value: cluster.local
        image: quay.io/minio/operator-sidecar:v6.0.2
        imagePullPolicy: IfNotPresent
        name: sidecar
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /ready
            port: 4444
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 1
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          runAsGroup: 1000
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp/minio/
          name: cfg-vol
        - mountPath: /tmp/minio-config
          name: configuration
      dnsPolicy: ClusterFirst
      initContainers:
      - args:
        - validate
        - --tenant
        - minio-tenant
        env:
        - name: CLUSTER_DOMAIN
          value: cluster.local
        image: quay.io/minio/operator-sidecar:v6.0.2
        imagePullPolicy: IfNotPresent
        name: validate-arguments
        resources: {}
        securityContext:
          runAsGroup: 1000
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp/minio/
          name: cfg-vol
        - mountPath: /tmp/minio-config
          name: configuration
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
      serviceAccount: minio-tenant-sa
      serviceAccountName: minio-tenant-sa
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: cfg-vol
      - name: minio-tenant-tls
        projected:
          defaultMode: 420
          sources:
          - secret:
              items:
              - key: tls.crt
                path: public.crt
              - key: tls.key
                path: private.key
              - key: tls.crt
                path: CAs/public.crt
              name: minio-tls
      - name: configuration
        projected:
          defaultMode: 420
          sources:
          - secret:
              name: storage-configuration
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data0
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: directpv-min-io
      volumeMode: Filesystem
    status:
      phase: Pending
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data1
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: directpv-min-io
      volumeMode: Filesystem
    status:
      phase: Pending
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data2
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: directpv-min-io
      volumeMode: Filesystem
    status:
      phase: Pending
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data3
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: directpv-min-io
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  availableReplicas: 0
  collisionCount: 0
  currentReplicas: 3
  currentRevision: minio-tenant-minio-pool-6dd85d8469
  observedGeneration: 3
  replicas: 3
  updateRevision: minio-tenant-minio-pool-6dd85d8469
  updatedReplicas: 3
pjuarezd commented 3 weeks ago

please upgrade to operator 7.1.1

Aragorn1026 commented 3 weeks ago

please upgrade to operator 7.1.1

my k8s is 1.28.0, it seems does not support minio 7.1.1?

JoelRuizRojas commented 2 weeks ago

Hi @Aragorn1026 k8s version 1.28 is no longer being supported by Kubernetes, please use a recent version. Can you also provide some steps to reproduce the issue?