Sync remote file <-> local file does not work

[Jendker]

Which version of the AzCopy was used?

10.11.0

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run?

azcopy sync https://jedrzej.blob.core.windows.net/rail-data/some_data.npy ~/some_data.npy

What problem was encountered?

I get the following output:

INFO: azcopy: A newer version 10.11.0 is available to download

INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job 5a85c569-c0b0-324f-6f03-82d8d658b02c has started
Log file is located at: /home/jedrzej/.azcopy/5a85c569-c0b0-324f-6f03-82d8d658b02c.log

Cannot perform sync due to error: cannot scan the path /home/jedrzej/basic_data.npy, please verify that it is a valid

If I touch ~/some_data.npy first, azcopy exits without downloading the file (file has still size 0) and I get the following output:

INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job 13e80161-37f0-d245-4914-162b8c1e7c09 has started
Log file is located at: /home/jedrzej/.azcopy/13e80161-37f0-d245-4914-162b8c1e7c09.log

0 Files Scanned at Source, 1 Files Scanned at Destination

Job 13e80161-37f0-d245-4914-162b8c1e7c09 Summary
Files Scanned at Source: 1
Files Scanned at Destination: 1
Elapsed Time (Minutes): 0.0333
Number of Copy Transfers for Files: 0
Number of Copy Transfers for Folder Properties: 0 
Total Number Of Copy Transfers: 0
Number of Copy Transfers Completed: 0
Number of Copy Transfers Failed: 0
Number of Deletions at Destination: 0
Total Number of Bytes Transferred: 0
Total Number of Bytes Enumerated: 0
Final Job Status: Completed

How can we reproduce the problem in the simplest way?

Run the command azcopy sync https://jedrzej.blob.core.windows.net/rail-data/some_data.npy ~/some_data.npy and touch ~/some_data.npy && azcopy sync https://jedrzej.blob.core.windows.net/rail-data/some_data.npy ~/some_data.npy

Have you found a mitigation/solution?

azcopy copy https://jedrzej.blob.core.windows.net/rail-data/some_data.npy ~/some_data.npy works fine, but I would prefer if azcopy sync would work properly with its delta scans.

[adreed-msft]

Sync works by comparing the source and the destination paths. If the destination path has a newer file, it will not copy it. Perhaps what you wanted to do was start with a copy, and then continue with syncs (or target entire folders instead of single files).

[adreed-msft]

Unfortunately, there's a bit of weirdness in that, if the destination does not exist (and this is intended for folders), sync won't run because it "can't" scan it.

[Jendker]

Do you think there is any solution for it, like checking if the path's dirname exists instead of the path if we are copying the file and then sync?

[g7cnp]

Sync works by comparing the source and the destination paths. If the destination path has a newer file, it will not copy it. Perhaps what you wanted to do was start with a copy, and then continue with syncs (or target entire folders instead of single files).

Surely that should be 'newer or same file' as currently sync re-copies files even if unchanged!

[adreed-msft]

@Jendker

Do you think there is any solution for it, like checking if the path's dirname exists instead of the path if we are copying the file and then sync?

I think that makes sense, yes.

@g7cnp

Surely that should be 'newer or same file' as currently sync re-copies files even if unchanged!

That sounds like incorrect behavior. Could you open a Github issue regarding this? Thanks! Nevermind, there's already one open.

[g7cnp]

I think you'll find #1666 covers this issue?

Regards

Huw

From: adreed-msft @.> Sent: 16 May 2022 16:15 To: Azure/azure-storage-azcopy @.> Cc: Huw Weatherhead @.>; Comment @.> Subject: Re: [Azure/azure-storage-azcopy] Sync remote file <-> local file does not work (#1490)

Do you think there is any solution for it, like checking if the path's dirname exists instead of the path if we are copying the file and then sync?

I think that makes sense, yes.

Surely that should be 'newer or same file' as currently sync re-copies files even if unchanged!

That sounds like incorrect behavior. Could you open a Github issue regarding this? Thanks!

- Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-storage-azcopy%2Fissues%2F1490%23issuecomment-1127800959&data=05%7C01%7Chuw%40weatherheadsweb.co.uk%7Cbb1d6d7ea2d2449c05ac08da374ec79b%7C1af5220bbf7a4ba48b854c45223fad45%7C0%7C0%7C637883108732750574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4L5KDk4dztIovSZ9zZrS2mW5mBhhnsEPji%2FxUg%2FXFBo%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJCYGOKOWH4G7J5P6GENGQLVKJQ5PANCNFSM477ZV2IA&data=05%7C01%7Chuw%40weatherheadsweb.co.uk%7Cbb1d6d7ea2d2449c05ac08da374ec79b%7C1af5220bbf7a4ba48b854c45223fad45%7C0%7C0%7C637883108732750574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5VAdylm6OaiOiZi7L9d2rPvHwwcsTWqnc0y155Dctcg%3D&reserved=0. You are receiving this because you commented.Message ID: @.**@.>>

[adreed-msft]

No, #1490 is it's own issue.