search

minio / mc

Unix like utilities for object store
https://min.io/download
GNU Affero General Public License v3.0
2.86k stars 548 forks source link

mc mirror takes long time when target bucket contains a lot of files #4873

Open archsh opened 7 months ago

archsh commented 7 months ago

Expected behavior

Using mc mirror to mirror local folder to a target bucket, it should be exit soon if local folder is empty or few files.

Actual behavior

It took a long time to start transfer files if the target bucket contains a lot of objects.

Steps to reproduce the behavior

mc mirror --json ./localfolder target/bucket/path

mc --version

System information

CentOS release 6.9 (Final) x64

zveinn commented 7 months ago

Yes, mc mirror does a full source to target comparison when it's ran. This is due to the --remove flag. However, it might be possible to speed up the mirror given that the --remove flag is not present. I will look into this in the coming weeks, no promises on delivery date though since my schedule is rather full.

PrisedRabbit commented 5 months ago

Same here. It takes a lot of time to mirror folder with 20 000+ files ( using aws cli solved the problem )

zveinn commented 4 weeks ago

I will be looking into this next week, if there is any additional information you want to share, then now is the time :)

karsapeng commented 2 weeks ago

helloHello, I wrote a script to use “mc mirror --remove --overwrite --limit-download $LIMIT_DOWNLOAD --limit-upload $LIMIT_UPLOAD minio-master minio-slave” to synchronize the data of the master node. Even if the data of the two minio nodes are consistent, if the master node is shut down and restarted due to a fault, I will change the direction of the synchronization and transfer the data to the slave node. Then the master will run “mc mirror --remove --overwrite --limit-download $LIMIT_DOWNLOAD --limit-upload $LIMIT_UPLOAD minio-slave minio-master”for synchronization. However, even if the data of the two minio nodes is consistent, minio-master will perform a full synchronization, which will take up a lot of io and cause the server to crash. My mc version is mc:RELEASE.2024-08-26T10-49-58Z

karsapeng commented 1 week ago

Hello, has this problem been solved?

zveinn commented 1 week ago

Hey, I have not had time to look into this yet. Just as I was about to start, something else came up. I will see if someone else can take it off my hands.