Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
614 stars 222 forks source link

Copy does not multi thread blob container creation #2872

Open TylerBurnett opened 4 days ago

TylerBurnett commented 4 days ago

I am working on a use case where I need to replicate 75'000 containers that contain data from one blob storage account to another. When trying to do this copy, the scanning itself (Which I have noticed to include pre-creating the containers on the dest) takes upwards of 70 hours. Watching the logs, I have noticed there is no threading that occurs in the scanning phase for containers which accounts for this massive slowdown.

Copying the data itself after the scan only takes 20 minutes. Are there any alternatives or settings I could try to speed this up?