Minio pool pods report tls errors when expanding tenants

[oghoneim]

Hello,

I am using Minio k8s operator v6.0.3 and when i try to expand a tenant by adding a second pool Minio fails to start and the following error logs are showing up on the Minio pool pods.

INFO: Unable to use the drive https://test-minio-pool-1-0.test-minio-hl.test.svc.cluster.local:9000/export0: drive not found, will be retried 2024-09-02T06:30:35.285379676Z INFO: Unable to use the drive https://test-minio-pool-1-0.test-minio-hl.test.svc.cluster.local:9000/export1: drive not found, will be retried 2024-09-02T06:30:35.285385751Z INFO: Waiting for a minimum of 1 drives to come online (elapsed 34s) 2024-09-02T06:30:35.285390037Z

2024-09-02T06:30:35.822418197Z API: SYSTEM.grid Time: 06:30:35 UTC 09/02/2024 DeploymentID: 29ff0903-2526-49d7-8f55-047c0fc385a5 2024-09-02T06:30:35.822460850Z Error: grid: https://test-minio-pool-0-0.test-minio-hl.test.svc.cluster.local:9000 re-connecting to https://test-minio-pool-1-0.test-minio-hl.test.svc.cluster.local:9000: tls: failed to verify certificate: x509: certificate is valid for test-minio-pool-0-0.test-minio-hl.test.svc.cluster.local, minio.test.svc.cluster.local, minio.test, minio.test.svc, ., .test.svc.cluster.local, not test-minio-pool-1-0.test-minio-hl.test.svc.cluster.local (tls.CertificateVerificationError) Sleeping 1.906s (3) (fmt.wrapError)

This is my tenant manifest, at first the tenant was deployed with pool-0 then i added pool-1

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: test
spec:
  pools:
    - servers: 1
      name: pool-0
      volumesPerServer: 2
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
    - servers: 1
      name: pool-1
      volumesPerServer: 2
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi

Expected Behavior

Adding a second pool should work

Your Environment

• Version used (minio --version): v6.0.3
• Server setup and configuration: k8s 1.28.9

[jiuker]

Looks like your yaml is incomplete, please share the full yaml

[oghoneim]

Sorry here is the full yaml. Also i forgot to mention i am creating the tenant using ArgoCD and kustomize.

apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"minio.min.io/v2","kind":"Tenant","metadata":{"annotations":{"prometheus.io/path":"/minio/v2/metrics/cluster","prometheus.io/port":"9000","prometheus.io/scrape":"true"},"labels":{"app":"minio","app.kubernetes.io/instance":"minio-operator-resources"},"name":"test-minio","namespace":"test"},"spec":{"certConfig":{},"configuration":{"name":"storage-configuration"},"env":[],"externalCaCertSecret":[],"externalCertSecret":[],"externalClientCertSecrets":[],"features":{"bucketDNS":false,"domains":{}},"image":"quay.io/minio/minio:RELEASE.2024-08-17T01-24-54Z","imagePullSecret":{},"mountPath":"/export","podManagementPolicy":"Parallel","pools":[{"name":"pool-0","servers":1,"volumeClaimTemplate":{"metadata":{"name":"data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}}}},"volumesPerServer":2},{"name":"pool-1","servers":1,"volumeClaimTemplate":{"metadata":{"name":"data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}}}},"volumesPerServer":2}],"priorityClassName":"","requestAutoCert":true,"serviceAccountName":"","serviceMetadata":{"consoleServiceAnnotations":{},"consoleServiceLabels":{},"minioServiceAnnotations":{},"minioServiceLabels":{}},"subPath":"","users":[{"name":"storage-user"}]}}
    prometheus.io/path: /minio/v2/metrics/cluster
    prometheus.io/port: "9000"
    prometheus.io/scrape: "true"
  labels:
    app: minio
    app.kubernetes.io/instance: minio-operator-resources
  name: test-minio
  namespace: test
spec:
  certConfig: {}
  configuration:
    name: storage-configuration
  env: []
  externalCaCertSecret: []
  externalCertSecret: []
  externalClientCertSecrets: []
  features:
    bucketDNS: false
    domains: {}
  image: quay.io/minio/minio:RELEASE.2024-08-17T01-24-54Z
  imagePullSecret:
    name: ""
  mountPath: /export
  podManagementPolicy: Parallel
  pools:
  - name: pool-0
    servers: 1
    volumeClaimTemplate:
      metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
    volumesPerServer: 2
  - name: pool-1
    servers: 1
    volumeClaimTemplate:
      metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
    volumesPerServer: 2
  priorityClassName: ""
  requestAutoCert: true
  serviceAccountName: ""
  serviceMetadata:
    consoleServiceAnnotations: {}
    consoleServiceLabels: {}
    minioServiceAnnotations: {}
    minioServiceLabels: {}
  subPath: ""
  users:
  - name: storage-user

[oghoneim]

I found the issue, i was missing the securitycontext and container securitycontext