minio on k8s, one node always says:Waiting for the first server to format the disks

[orxvan]

Expected Behavior

• 1) 4 node works fine.
• 2) service works finel.

Current Behavior

• 1) minio-2 logs “Waiting for the first server to format the disks”
• 2) curl minio-2.minio:9000 response “Server not initialized, please try again.”
• 3) other 3 nodes works fine.BUT the whole service is unsteadiness.

Possible Solution

Steps to Reproduce (for bugs)

may NOT reproduce.

• 1) deploy a minio cluster on k8s, version is RELEASE.2020-02-20T22-51-23Z,storage use local volume on every ECS，4nodes.every nodes has 500GB files.
• 2) I deploy a new minio,it's fine.AND I transfer the data.(use scp or cp)
• 3) Stop the old minio and new minio service；
• 4) copy the data from old to the new;
• 5) start the new minio service.

Context

kubectl delete po minio-2,can't work redeploy the service , may work, but It's a online service, redpeloy makes the service down.

Regression

I google /search the key words ,it always says it's a network error.But I can curl minio-2.minio:9000，just reponse 503 means the node is not fine。

Your Environment

• Version used RELEASE.2020-02-20T22-51-23
• Server setup and configuration:

  name:               minio
  Namespace:          prod
  CreationTimestamp:  Sat, 19 Feb 2022 09:36:37 +0800
  Selector:           app=minio
  Labels:             app=minio
  Annotations:        cpaas.io/creator: admin
                  cpaas.io/updated-at: 2022-02-19T02:04:12Z
  Replicas:           4 desired | 4 total
  Update Strategy:    RollingUpdate
  Partition:        0
  Pods Status:        4 Running / 0 Waiting / 0 Succeeded / 0 Failed
  Pod Template:
  Labels:       app=minio
  Annotations:  cpaas.io/creator: admin
  Containers:
  minio-persist:
  Image:      ***minio:RELEASE.2020-02-20T22-51-23Z
  Port:       <none>
  Host Port:  <none>
  Command:
    minio
    server
  Args:
    http://minio-0.minio:9000/data
    http://minio-1.minio:9000/data
    http://minio-2.minio:9000/data
    http://minio-3.minio:9000/data
  Limits:
    cpu:     1
    memory:  1G
  Requests:
    cpu:     100m
    memory:  512M
  Environment:
    MINIO_ACCESS_KEY:  ***
    MINIO_SECRET_KEY:  ***
  Mounts:
    /data from minio-data (rw)
  Volumes:  <none>
  Volume Claims:
  Name:          minio-data
  StorageClass:  local-storage
  Labels:        <none>
  Annotations:   <none>
  Capacity:      10G
  Access Modes:  [ReadWriteOnce]
  Events:
  Type    Reason            Age                From                    Message
  ----    ------            ----               ----                    -------
  Normal  SuccessfulCreate  24m (x4 over 48m)  statefulset-controller  create Pod minio-2 in StatefulSet minio successful

• Operating System and version (uname -a):Linux worker-2 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[harshavardhana]

1. deploy a minio cluster on k8s, version is RELEASE.2020-02-20T22-51-23Z,storage use local volume on every ECS，4nodes.every nodes has 500GB files. *

this looks like we can't write to your drives

[orxvan]

1. deploy a minio cluster on k8s, version is RELEASE.2020-02-20T22-51-23Z,storage use local volume on every ECS，4nodes.every nodes has 500GB files.

this looks like we can't write to your drives

Thx, but how to reslove or debug it. I try to rm the path where mounted by the local volume,and restart the minio-2,but it not worked

[orxvan]

1. deploy a minio cluster on k8s, version is RELEASE.2020-02-20T22-51-23Z,storage use local volume on every ECS，4nodes.every nodes has 500GB files.

this looks like we can't write to your drives

Maybe I troubled you,

deploy a minio cluster on k8s, version is RELEASE.2020-02-20T22-51-23Z,storage use local volume on every ECS，4nodes.every nodes has 500GB files.

the old minio cluster works fine for 2 years. It only find in the dest cluster when transfer the date

[klauspost]

I transfer the data.(use scp or cp) copy the data from old to the new

Whatever it is you are doing here is not going to work. You should never touch backend files.

We do not provide community support for older MinIO versions. Start with a clean cluster on the newest release.