It's slow when trying with code example

[EmmaZhu]

================================================================== @nowenL I try run this example and it's so slow, and by the way it's using my internet network to download and upload the blobs to destination container. What is wrong?

Linked issue

[alexsandro-xpt]

@EmmaZhu I'm copying full container of blobs between different accounts.

Just see:

CloudBlobContainer blobContainer = FirstAccount();
CloudBlobContainer blobContainer2 = SecondAccount();

CopyDirectoryOptions options = new CopyDirectoryOptions()
{
    Recursive = true
};

var task = TransferManager.CopyDirectoryAsync(blobContainer.GetDirectoryReference(string.Empty), blobContainer2.GetDirectoryReference(string.Empty), true, options, null);//context

task.Wait();

[EmmaZhu]

@alexsandro-xpt From the code you shared, you are using a server side copy that DMLib sends requests to start copying that Azure Storage will do the actually copying work. Its speed is decided by Azure Storage that if the two accounts are in different DCs, it might be slow than downloading/uploading.

Could you share the scale of data you are copying and environment you are using? It might be faster if change the copying invoking like following to do a sync copy: var task = TransferManager.CopyDirectoryAsync(blobContainer.GetDirectoryReference(string.Empty), blobContainer2.GetDirectoryReference(string.Empty), false, options, null);

In a sync copying, DMLib will do the actual download and upload work. It could be faster than a server side copying if DMLib is running on a VM which locates in the same DC with the destination.

[alexsandro-xpt]

@EmmaZhu It's under same DC, using AzCopy it's is really fast. May I should record a video to show it working...

[EmmaZhu]

AzCopy is actually using DMLib in the background to do actual data transferring, their performance should similar. Did you try with following setting at the very beginning of your program? ServicePointManager.DefaultConnectionLimit = Environment.ProcessorCount * 8;

If it's still slow with the above setting, could you share me a dump and exactly time when you try it?

[alexsandro-xpt]

Thank you for replay @EmmaZhu this is my full script https://gist.github.com/alexsandro-xpt/b2c8d6fca959df8ba5c2e02087311944

[vinjiang]

@EmmaZhu , please continue to follow-up.

[EmmaZhu]

@alexsandro-xpt

Sorry for the signaficant late reply. Do you still need my help?

The method you were calling is to start service side copying, the copying is actually done by azure storage server which doesn't have SLA.

It should be faster to change code like following to use DMLib's sync copying functionality:

TransferManager.CopyDirectoryAsync(blobContainer.GetDirectoryReference(string.Empty), blobContainer2.GetDirectoryReference(string.Empty), false, options, null);//context

Thanks Emma

[alexsandro-xpt]

Alright @EmmaZhu

Actually I'm not working with Azure Storage Copy, but I will be back soon. I will test it, thank you a lot!