search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
604 stars 217 forks source link

Azcopy sync option for file share is not supported #2650

Open yamorish opened 5 months ago

yamorish commented 5 months ago

Which version of the AzCopy was used?

10.24.0 (latest)

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

azcopy sync [local file share] [Azure files share]

What problem was encountered?

The customer found the data loss issues when they used "Azcopy sync" to sync their on-premises files (millions of files) to Azure files. They detected few thousands of files are corrupted, but the metadata-information (filename, filesize, timestamp) is the same as original files for these files. Also, Azcopy completed successfully without any error.

How can we reproduce the problem in the simplest way?

Haven't confirmed yet, but it can be reproduced, since this is documented.

https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-files#synchronize-files https://learn.microsoft.com/en-us/azure/storage/files/storage-files-migration-overview

AzCopy sync is supported but not fully recommended for Azure Files. AzCopy sync doesn't support differential copies at scale, and some file fidelity might be lost. To learn more, see [Migrate to Azure file shares](https://learn.microsoft.com/en-us/azure/storage/files/storage-files-migration-overview#file-copy-tools).

Have you found a mitigation/solution?

Given the circumstances, detecting corruption is very difficult, and then we should consider limiting the functionality of the tool or issuing strong warnings upon execution until the issue is resolved.

tanyasethi-msft commented 5 months ago

Hi @yamorish, thanks for reporting this! We are discussing this internally within the team and will get back on this thread.

seanmcc-msft commented 5 months ago

Hi @yamorish, we will add warnings for this case to the logs and console output, we are targeting this for our September release.

ld0614 commented 1 month ago

Hi @seanmcc-msft do you know if the fix for this issue is still targeting the September release?

Are there any options to mitigate such as using md5 hashing rather than metadata comparisons and running a sync multiple times?