Performance issue when use azcopy sync on huge file count (from 10.5.0)

[mslichao]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

AzCopy 10.5.0

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy sync 'https://ocrdatasetprodwu2.blob.core.windows.net/ocr-dataset/dataset/CDIP-IMAGES/20200611T022412/' 'https://ocrdatasetprod.blob.core.windows.net/ocr-dataset/dataset/CDIP-IMAGES/20200611T022412/' --recursive --delete-destination true

What problem was encountered?

azcopy sync will scan files both in target and source, but it is very slow. In my case, I have 20M files in both source and target. It took 14 hours but still not finish the scanning in target folder. And what's more, the CPU usage always 90+ percent. I did the same actions with 10.4.3, it only took 2 hours.

How can we reproduce the problem in the simplest way?

1. Prepare a folder with 20M files in it.
2. Use azcopy copy to copy them to another container
3. Use azcopy sync '***' '***' --recursive --delete-destination true to sync them again In the third step, expect it can finish the scanning in 2 hours, but in my case it took 14 hours but still not finish scanning target folder.

OR

Ping me on Microsoft Teams, my alias is lchao

Have you found a mitigation/solution?

Roll back to 10.4.3

[TiTi]

Hello, I also have this issue: azcopy sync --recursive --delete-destination true is very slow on file share in my case.

This is the command i used for troubleshooting: ./azcopy_10.13.0 sync "https://<storage_source>.file.core.windows.net/asshared/?se=2021-11-04T17%3A11Z&sp=rl&spr=https&sv=2018-03-28&ss=f&srt=sco&sig=<sig>" "https://<storage_dest>.file.core.windows.net/asshared/?se=2021-11-04T17%3A11Z&sp=rwdl&spr=https&sv=2018-03-28&ss=f&srt=sco&sig=<sig>" --log-level ERROR --recursive=true --delete-destination=true

I made my tests on a Standard_A2m_v2 linux VM (2 vCPUs, 16GB RAM). The destination storage does not get other traffic than my test.

Here are the number of files scanned at destination in 60s:

azcopy version	Nb files scanned
10.4.3	69185
10.5.0	17964
10.5.1	24966
10.6.0	26160
10.8.0	17365
10.9.0	2710
10.10.0	2265
10.13.0	3462

I made several tests and numbers change a little but stay in the same order of magnitude.

10.5.0 : "Improved scanning performance for most cases by adding support for parallel local and Blob enumeration." 1.9.0 : "Added scanning logs"

We can see a huge drop in scan performance! I'm available to make more tests.