search

minio / minio

MinIO is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license.
https://min.io/download?license=agpl&platform=linux
GNU Affero General Public License v3.0
47.94k stars 5.49k forks source link

site replication only sync part of files #19978

Closed li-zheng-hao closed 4 months ago

li-zheng-hao commented 4 months ago

NOTE

If this case is urgent, please subscribe to Subnet so that our 24/7 support team may help you faster.

I have a cluster server with 4 minio nodes and a single-node minio server for backup , now i'm trying to use site replication to sync files from cluster server to backup server. However, after synchronizing a part of the files, the synchronization operation stopped.

Source Server:

image

Backup Server:

image

i had tried to recreate backup server and reconfigure site replication , the result is same. and the strange thing is that the number of files copied is the same.

Expected Behavior

all files should sync to backup server

Current Behavior

after copy part of files ,it stopped.

Possible Solution

Steps to Reproduce (for bugs)

  1. create a cluter server with 4 single drive node by docker-compose on 4 servers 192.168.5.111,112,113,116:
version: "3"
services:
  minio:
    image: minio/minio
    user: "root:root"
    network_mode: "host"
    restart: always
    container_name: minio-1   # 1....4
    entrypoint:
      - sh
      - -euc
      - |
        #        mkdir -p /data1 &&         mkdir -p /data2 &&
        minio server --address :9000 --console-address :9001 http://minio-{1...4}/data
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=XXXX
      - MINIO_UPDATE=off
    volumes:
      - /data/minio/data:/data
      - /etc/hosts:/etc/hosts:ro
  1. create a single node server with docker compose file (change container name to minio-5).
version: "3"
services:
  minio:
    image: minio/minio
    user: "root:root"
    network_mode: "host"
    restart: always
    container_name: minio-5
    entrypoint:
      - sh
      - -euc
      - |
        #        mkdir -p /data1 &&         mkdir -p /data2 &&
        minio server --address :9000 --console-address :9001 /data
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=XXXX
      - MINIO_UPDATE=off
    volumes:
      - /data/minio/data:/data
      - /etc/hosts:/etc/hosts:ro
  1. upload some files to cluster server
  2. config site replication on console ui

image

  1. wait for file sync to complete

I'm not sure if the above steps can reproduce the problem.

Context

here are more information :

[root@minio-5 config]# mc admin replicate info minio_cluster
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication   
                                     |                 |                                                |      | Per Bucket |                          
536905e1-6131-4f8d-89c4-2c1418f4d21d | backup          | http://192.168.5.115:9000                      |      | N/A        | false                    
d744ce8e-2cc3-4407-973c-2e6ee652e26b | cluster         | http://192.168.5.111:9000                      |      | N/A        | false

source server ips : 192.168.5.111, 192.168.5.112,192.168.5.113,192.168.5.114,192.168.5.116 backup server ip: 192.168.5.115

[root@minio-5 config]# mc   admin  replicate status  minio_cluster
Bucket replication status:
●  4/4 Buckets in sync

Policy replication status:
●  5/5 Policies in sync

User replication status:
●  6/6 Users in sync

Group replication status:
No Groups present

ILM Expiry Rules replication status:
No ILM Expiry Rules present

Object replication status:
Replication status since 3 days 
192.168.5.115:9000
Replicated:    278 objects (69 MiB)
Transfer Rate: 0 B/s (avg: 0 B/s; max 0 B/s)
Latency:       1ms (avg: 1ms; max 23ms)
Link:          ● online (total downtime: 45 seconds)
Errors:        0 in last 1 minute; 0 in last 1hr; 0 since uptime

192.168.5.115:9000
Replicated:    366 objects (90 MiB)
Transfer Rate: 0 B/s (avg: 402 B/s; max 1.2 MB/s)
Latency:       1ms (avg: 1ms; max 23ms)
Link:          ● online (total downtime: 45 seconds)
Errors:        0 in last 1 minute; 0 in last 1hr; 0 since uptime

192.168.5.115:9000
Replicated:    365 objects (89 MiB)
Transfer Rate: 0 B/s (avg: 742 kB/s; max 1.2 MB/s)
Latency:       1ms (avg: 1ms; max 23ms)
Link:          ● online (total downtime: 45 seconds)
Errors:        0 in last 1 minute; 0 in last 1hr; 0 since uptime

Replicated:    278 objects (69 MiB)
Transfer Rate: 0 B/s (avg: 154 kB/s; max 2.0 MB/s)
Latency:       0s (avg: 0s; max 0s)
Link:          ● offline 0 milliseconds (total downtime: 0 milliseconds)
Errors:        0 in last 1 minute; 0 in last 1hr; 0 since uptime

192.168.5.115:9000
Replicated:    367 objects (90 MiB)
Transfer Rate: 0 B/s (avg: 425 B/s; max 1.2 kB/s)
Latency:       1ms (avg: 1ms; max 23ms)
Link:          ● online (total downtime: 45 seconds)
Errors:        0 in last 1 minute; 0 in last 1hr; 0 since uptime

192.168.5.115:9000
Replicated:    367 objects (90 MiB)
Transfer Rate: 0 B/s (avg: 18 MB/s; max 19 MB/s)
Latency:       1ms (avg: 1ms; max 23ms)
Link:          ● online (total downtime: 45 seconds)
Errors:        0 in last 1 minute; 0 in last 1hr; 0 since uptime

Summary:
Replicated:    2,021 objects (496 MiB)
Queued:        ● 0 objects, (0 B) (avg: 0 objects, 0 B; max: 0 objects, 0 B)
Received:      75,105 objects (24 GiB)
[root@minio-5 config]# mc   admin  replicate status  minio_backup
Bucket replication status:
●  4/4 Buckets in sync

Policy replication status:
●  5/5 Policies in sync

User replication status:
●  6/6 Users in sync

Group replication status:
No Groups present

ILM Expiry Rules replication status:
No ILM Expiry Rules present

Object replication status:
Replication status since 3 minutes 
Summary:
Replicated:    0 objects (0 B)
Queued:        ● 0 objects, (0 B) (avg: 0 objects, 0 B; max: 0 objects, 0 B)
Received:      0 objects (0 B)

192.168.5.115 is the IP of the backup server

[root@minio-5 config]# mc replicate diff minio_cluster/pdm --full
Unreplicated versions summary
─────────────   
Total Unreplicated: 62 •    

┌─────────────────────────────────────────────────────────────────────────────────────┐
│ Attempted At              Created                   Status      VersionID                             Op    Object                                                       │
│─────────────────────────────────────────────────────────────────────────────────────│
│ 0001-01-01 00:00:00 UTC  2024-06-08 03:30:03 UTC             3c553295-d3b9-474f-87e1-ff6397c3e589  PUT  00e32fb2c085519340db71d565047d27.STEP                            │
│ 0001-01-01 00:00:00 UTC  2024-06-08 03:30:06 UTC             59eb8f57-b9da-4b4e-ad80-387a429c8edb  PUT  01811e9c9fd8a05b6e130a7bd909365b.stl                             │
│ 0001-01-01 00:00:00 UTC  2024-06-18 05:11:50 UTC             0b79c360-5b78-4450-8434-593a6bdd8cd1  PUT  0a63d45eb91d875d54ec4c889f91e28a.step                            │
│ 0001-01-01 00:00:00 UTC  2024-06-18 05:12:44 UTC             15873719-2344-4afc-83a7-816a0473c20e  PUT  0bc498a12dc2a001fba0579e107a4eac.stl                             │
│ 0001-01-01 00:00:00 UTC  2024-06-11 10:07:35 UTC             cf6229e4-da12-483b-8450-8a6b1d970dff  PUT  0bf6cdd64ca6ddba79e82a0b94ee7263.stl                             │
│ 0001-01-01 00:00:00 UTC  2024-06-08 03:30:48 UTC             3039eafe-7c79-40bf-8cd3-1e105f3e9733  PUT  0d87a44b982eb53805dfc8514c5dee33.stl                             │
│ 0001-01-01 00:00:00 UTC  2024-06-08 03:30:26 UTC             5f396e83-4f6d-41bb-ad09-2e3048f8d075  PUT  11c9807b9b52d1221a32fe1404200de9.stl                             │
│ 0001-01-01 00:00:00 UTC  2024-06-08 03:30:25 UTC             aa4e90bf-efd4-470d-a67e-e19d65208632  PUT  1448e3529f53443c82ae0691013ae361.stl                             │
│ 0001-01-01 00:00:00 UTC  2024-06-08 03:30:19 UTC             e13d7aba-9efe-4c75-9097-68e305f4a0fc  PUT  15903e6b85a54be08384268378c54fa1.igs                             │
│ 0001-01-01 00:00:00 UTC  2024-06-08 03:30:34 UTC             6bb44d91-88cb-491b-a746-dc039967cf40  PUT  190fe4506d24385004d8685cd385f208.stl                             │
└─────────────────────────────────────────────────────────────────────────────────────┘
[root@minio-5 config]# mc replicate diff minio_backup/pdm --full
Unreplicated versions summary
─────────────   
Total Unreplicated: 2 •     

┌─────────────────────────────────────────────────────────────────────────────────────┐
│ Attempted At              Created                   Status      VersionID                             Op    Object                                                       │
│─────────────────────────────────────────────────────────────────────────────────────│
│ 0001-01-01 00:00:00 UTC  2024-06-22 02:11:54 UTC             34ad3d48-1859-4301-953f-87947cb7aedb  PUT  736b1d4c8160204c8edfc96f1be958ce.dwg                             │
│ 0001-01-01 00:00:00 UTC  2024-06-22 06:30:14 UTC             f3c2a6ae-e6fe-4d84-9b7b-215400edf9b4  PUT  9c23fa26a96bc400a88152cba3305987.stl                             │
│                                                                                                                                                                          │
│                                                                                                                                                                          │
│                                                                                                                                                                          │
│                                                                                                                                                                          │
│                                                                                                                                                                          │
│                                                                                                                                                                          │
│                                                                                                                                                                          │
│                                                                                                                                                                          │
└─────────────────────────────────────────────────────────────────────────────────────┘
enter/spacebar  • ↓/j Move down • ↑/k Move up • q quit

Regression

i dont know

Your Environment

harshavardhana commented 4 months ago

Please upgrade your setup.

menocrat commented 2 weeks ago

I have the same problem, the amount of data is not the same between the two sites

I've tried changing the Scanner configuration and the same and latest version is used between both sites

root@localhost:~# minio --version
minio version RELEASE.2024-10-02T17-50-41Z (commit-id=ded0b19d97bb9f73f4d78382eb131ff80003e272)
Runtime: go1.22.8 linux/amd64
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
Copyright: 2015-2024 MinIO, Inc.