Azcopy file syncing with old files

danielsjf commented 5 years ago

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

8.1.0

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy /Source:x /Pattern:x /Dest:x /DestKey:x /XO /Y

What problem was encountered?

I want to update a file, but only if the local file is newer. I had a more difficult example though:

Old file from 2 years ago -> after upload, the last modification date on Azure becomes the upload date
New file from 1 year ago -> will never upload since one year ago is before the date that azure knows which is the original upload date

Of course, as far as I know, the file from one year ago is actually the newer file and I want this file to be uploaded in favour of the one on Azure.

Typically on Windows, copying a file keeps the original modification date.

For me specifically, there are two ways to tackly this issue:

Only upload when hash is different (as far as I could find, this doesn't seem to be supported by azcopy)
Adapt the azure modification time to the original file modification date (also not possible as far as I could find)

How can we reproduce the problem in the simplest way?

Create two files with a certain time in between. First upload the oldest, then the newest. As you will see, the newest will never be uploaded (unless you always upload without the only if newer setting).

Have you found a mitigation/solution?

Not yet apart from reuploading all files always (which takes long for us)

scsummers commented 5 years ago

I am struggling with the same issue:

The last modified time stamp gets changed.
Subsequent copy jobs just re-copy everything because they are seen as "newer"

Tried to follow the process to mount an Azure file share so I could use Robocopy to see if that would help but unfortunately the ISP blocks port 445 which is required... :(

EmmaZhu commented 5 years ago

@danielsjf & @scsummers

Currently, the last modified time of blob/azure file is not supported to be changed by REST API. As a client side tool, AzCopy may not be able to do anything about this.

@zezha-msft For @danielsjf's first requirement, to only upload when hash is different, could you help to evaluate whether AzCopy v10 could support this scenario?

scsummers commented 5 years ago

I haven’t been able to test yet but I know you CAN use Robocopy to mount an Azure File Share; do you happen to know if that would keep the date/timestamp intact? Sounds like it is controlled on the backend and not the tool you are using?

In principle AZCopy (or rather the ability to use something like AZCopy/Robocopy) is very beneficial in cases like mine where we acquire a company and need to ingest fileshare data prior to our networks being connected. I can copy data to Azure then down into our environment. However, right now I can’t do anything in advance of a cutover time window because everytime it sees files as newer on the source and overwrites them.

From: EmmaZhu-MSFT notifications@github.com Reply-To: Azure/azure-storage-azcopy reply@reply.github.com Date: Wednesday, March 6, 2019 at 19:19 To: Azure/azure-storage-azcopy azure-storage-azcopy@noreply.github.com Cc: "Summers, Chris" csummers@ecolab.com, Mention mention@noreply.github.com Subject: Re: [Azure/azure-storage-azcopy] Azcopy file syncing with old files (#257)

Caution: This email originated from outside of the organization. DO NOT CLICK on links or open attachments unless you recognize the sender and know the content is safe.

@danielsjfhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_danielsjf&d=DwMFaQ&c=clRTYxLjfWTYQkksq4Trqw&r=eH4Z5_MjrcZGmVvDCEZR9xURltSzhEOv1pNJcWAuu_M&m=3Z0er1Di0VPXSjiUHKHVJQTMUs9us1fIMIuyejtAOS4&s=cdrSzjoO7aKEV8qOIvjICIy8BgIwepnn0w9_8nEiuAY&e= & @scsummershttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_scsummers&d=DwMFaQ&c=clRTYxLjfWTYQkksq4Trqw&r=eH4Z5_MjrcZGmVvDCEZR9xURltSzhEOv1pNJcWAuu_M&m=3Z0er1Di0VPXSjiUHKHVJQTMUs9us1fIMIuyejtAOS4&s=8DxdFZmyVE4rN_nYWwCn19pVPUe22bETpNPpgqD37GQ&e=

Currently, the last modified time of blob/azure file is not supported to be changed by REST API. As a client side tool, AzCopy may not be able to do anything about this.

@zezha-msfthttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_zezha-2Dmsft&d=DwMFaQ&c=clRTYxLjfWTYQkksq4Trqw&r=eH4Z5_MjrcZGmVvDCEZR9xURltSzhEOv1pNJcWAuu_M&m=3Z0er1Di0VPXSjiUHKHVJQTMUs9us1fIMIuyejtAOS4&s=4fsH5jmwp8vYq3hqFLlCKTNs0XZTo6Xercv0Gl1BH-8&e= For @danielsjfhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_danielsjf&d=DwMFaQ&c=clRTYxLjfWTYQkksq4Trqw&r=eH4Z5_MjrcZGmVvDCEZR9xURltSzhEOv1pNJcWAuu_M&m=3Z0er1Di0VPXSjiUHKHVJQTMUs9us1fIMIuyejtAOS4&s=cdrSzjoO7aKEV8qOIvjICIy8BgIwepnn0w9_8nEiuAY&e='s requirement, Could you help to evaluate whether AzCopy v10 could support this scenario?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Azure_azure-2Dstorage-2Dazcopy_issues_257-23issuecomment-2D470343933&d=DwMFaQ&c=clRTYxLjfWTYQkksq4Trqw&r=eH4Z5_MjrcZGmVvDCEZR9xURltSzhEOv1pNJcWAuu_M&m=3Z0er1Di0VPXSjiUHKHVJQTMUs9us1fIMIuyejtAOS4&s=jQckm6KT2Mi7B_Og6ZFDt9m_5vvG7bHbxBsDGDM4ipw&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_Ajd3sPDtMzDf0I0GRRBwUN-5FLVhedtUb5ks5vUGk6gaJpZM4beVxu&d=DwMFaQ&c=clRTYxLjfWTYQkksq4Trqw&r=eH4Z5_MjrcZGmVvDCEZR9xURltSzhEOv1pNJcWAuu_M&m=3Z0er1Di0VPXSjiUHKHVJQTMUs9us1fIMIuyejtAOS4&s=9a-wCpD_q0NAnGlvrdnZlP9UEHFEAHAPHu7cDEvWGss&e=.

CONFIDENTIALITY NOTICE: This e-mail communication and any attachments may contain proprietary and privileged information for the use of the designated recipients named above. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

ricolaten commented 3 years ago

Is there an update on this issue? I'm having issues with the sync function caused by Last Modified Date comparison too.

@danielsjf : I agree with your suggestion of only uploading when hash is different. This would help to have this option to choose instead of Last Modified Date (since upload to blob changes it).

@zezha-msft: Could this hash comparison be implemented for local > blob sync instead of date comparison?

zezha-msft commented 3 years ago

Unfortunately the hash comparison is currently not a reliable option, since the stored MD5 on the blob could be inaccurate. This scenario can be enabled if the service can enable a new API.

For Azure File, we may be able to use the x-ms-file-last-write-time which is settable. I'll note that down as a work item in our backlog.

EurikasHP commented 3 years ago

@zezha-msft azcopy does md5 checks so I wonder why the MD5 hash on the blob could be considered inaccurate. Are you looking for a different hash algorithm in the service API or does it need enhancements to be accurate?

zezha-msft commented 3 years ago

@EurikasHP the stored MD5 value for the blob service can become outdated:

x-ms-blob-content-md5	Optional. An MD5 hash of the blob content. Note that this hash is not validated, as the hashes for the individual blocks were validated when each was uploaded.The Get Blob operation returns the value of this header in the Content-MD5 response header.If this property is not specified with the request, then it is cleared for the blob if the request is successful.

zezha-msft commented 3 years ago

Closing due to inactivity. As far as I can tell there is no viable alternative (from the service perspective) to using the lmts for now.

Azure / azure-storage-azcopy