gapra-msft
commented
10 months ago Hi @verveanix thank you for reporting this issue. We are currently working on investigating this issue.
Closed verveanix closed 10 months ago
gapra-msft
commented
10 months ago Hi @verveanix thank you for reporting this issue. We are currently working on investigating this issue.
verveanix
commented
10 months ago Thanks @gapra-msft. I tweaked some stuff to improve my logging documentation and I'm seeing some interesting behavior - some of our blobs sync successfully from the same private [staging-host] to a different storage account ([production-private]) where we store internal tracking data about our accounts.
Is it possible that syncing from a private account to a public account is causing some sort of routing error when AzCopy attempts to read the binary data from the private account, but that routing change is not in place on a private -> private account?
verveanix
commented
10 months ago I did some more experimentation here - based on this Stack Overflow answer I fiddled with my script to use azcopy cp and a SAS token for the source URL that is explicitly scoped for the blob - I get the same behavior.
2023/11/15 16:19:46 AzcopyVersion 10.21.2
2023/11/15 16:19:46 OS-Environment linux
2023/11/15 16:19:46 OS-Architecture amd64
2023/11/15 16:19:46 Log times are in UTC. Local time is 15 Nov 2023 16:19:46
2023/11/15 16:19:46 ISO 8601 START TIME: to copy files that changed before or after this job started, use the parameter --include-before=2023-11-15T16:19:41Z or --include-after=2023-11-15T16:19:41Z
2023/11/15 16:19:47 WARN: Failed to create destination container mycontainer. The transfer will continue if the container exists
2023/11/15 16:19:47 Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2023/11/15 16:19:47 Job-Command cp --s2s-preserve-access-tier=false https://[private-staging-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15t17%3A49%3A43z&sig=-REDACTED-&ske=2023-11-15t17%3A49%3A43z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15t16%3A19%3A46z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rl&sr=b&sv=2023-01-03 https://[public-production-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15t17%3A49%3A43z&sig=-REDACTED-&ske=2023-11-15t17%3A49%3A43z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15t16%3A19%3A45z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rcwdl&sr=c&sv=2023-01-03
2023/11/15 16:19:47 Number of CPUs: 3
2023/11/15 16:19:47 Max file buffer RAM 1.500 GB
2023/11/15 16:19:47 Max concurrent network operations: 32 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2023/11/15 16:19:47 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2023/11/15 16:19:47 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2023/11/15 16:19:47 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override)
2023/11/15 16:19:47 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2023/11/15 16:19:47 Max open files when downloading: 1048152 (auto-computed)
2023/11/15 16:19:47 Final job part has been created
2023/11/15 16:19:47 JobID=993eaa9f-5646-1f49-6fe4-c720f7f75ac1, credential type: Anonymous
2023/11/15 16:19:47 Final job part has been scheduled
2023/11/15 16:19:47 INFO: [P#0-T#0] Starting transfer: Source "https://[private-staging-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15T17%3A49%3A43Z&sig=-REDACTED-&ske=2023-11-15T17%3A49%3A43Z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15T16%3A19%3A46Z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rl&sr=b&sv=2023-01-03" Destination "https://[public-production-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15T17%3A49%3A43Z&sig=-REDACTED-&ske=2023-11-15T17%3A49%3A43Z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15T16%3A19%3A45Z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rcwdl&sr=c&sv=2023-01-03". Specified chunk size 8388608
2023/11/15 16:19:47 ==> REQUEST/RESPONSE (Try=1/324.295011ms, OpTime=415.886833ms) -- RESPONSE STATUS CODE ERROR
PUT https://[public-production-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?blockid=MDAwMDCfqj6ZRlZJH2%2FkxyD391rBMDAwMDAwMDAwMDAwMDI0&comp=block&se=2023-11-15T17%3A49%3A43Z&sig=-REDACTED-&ske=2023-11-15T17%3A49%3A43Z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15T16%3A19%3A45Z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rcwdl&sr=c&sv=2023-01-03
Accept: application/xml
Content-Length: 0
User-Agent: AzCopy/10.21.2 azsdk-go-azblob/v1.1.0 (go1.19.12; linux)
X-Ms-Client-Request-Id: 44431710-0710-4b9b-5800-0544edea6fe0
x-ms-copy-source:
x-ms-source-range:
x-ms-version:
--------------------------------------------------------------------------------
RESPONSE Status: 403 This request is not authorized to perform this operation.
Content-Length: 248
Content-Type: application/xml
Date: Wed, 15 Nov 2023 16:19:47 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Client-Request-Id: 44431710-0710-4b9b-5800-0544edea6fe0
X-Ms-Error-Code: CannotVerifyCopySource
X-Ms-Request-Id: 8d5e2152-201e-0004-43df-173200000000
X-Ms-Version: 2020-10-02
Response Details: <Code>CannotVerifyCopySource</Code><Message>This request is not authorized to perform this operation. </Message>
2023/11/15 16:19:47 ERR: [P#0-T#0] COPYFAILED: https://[private-staging-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15T17%3A49%3A43Z&sig=-REDACTED-&ske=2023-11-15T17%3A49%3A43Z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15T16%3A19%3A46Z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rl&sr=b&sv=2023-01-03 : 403 : 403 This request is not authorized to perform this operation.. When Staging block from URL. X-Ms-Request-Id: 8d5e2152-201e-0004-43df-173200000000
Dst: https://[public-production-host].blob.core.windows.net/mycontainer/Blobdir/2.37.0/binary-blob-123456.bin?se=2023-11-15T17%3A49%3A43Z&sig=-REDACTED-&ske=2023-11-15T17%3A49%3A43Z&skoid=01b10450-230c-4aae-92c0-7c792ab955d5&sks=b&skt=2023-11-15T16%3A19%3A45Z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rcwdl&sr=c&sv=2023-01-03
2023/11/15 16:19:47 JobID=993eaa9f-5646-1f49-6fe4-c720f7f75ac1 canceled
2023/11/15 16:19:47 all parts of entire Job 993eaa9f-5646-1f49-6fe4-c720f7f75ac1 successfully completed, cancelled or paused
2023/11/15 16:19:47 all parts of entire Job 993eaa9f-5646-1f49-6fe4-c720f7f75ac1 successfully cancelled
2023/11/15 16:19:47 is part of Job which 1 total number of parts done
2023/11/15 16:19:49 PERF: primary performance constraint is Unknown. States: W: 0, F: 0, S: 0, E: 1, T: 1, GRs: 32
2023/11/15 16:19:49 0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 2-sec Throughput (Mb/s): 1072.8358
2023/11/15 16:19:49
Diagnostic stats:
IOPS: 13
End-to-end ms per request: 418
Network Errors: 0.00%
Server Busy: 0.00%
Job 993eaa9f-5646-1f49-6fe4-c720f7f75ac1 summary
Elapsed Time (Minutes): 0.0334
Number of File Transfers: 1
Number of Folder Property Transfers: 0
Number of Symlink Transfers: 0
Total Number of Transfers: 1
Number of File Transfers Completed: 0
Number of Folder Transfers Completed: 0
Number of File Transfers Failed: 0
Number of Folder Transfers Failed: 0
Number of File Transfers Skipped: 0
Number of Folder Transfers Skipped: 0
TotalBytesTransferred: 0
Final Job Status: CancelledHi @verveanix here is a doc on how to use AzCopy between storage accounts with network restrictions. Please take a look an ensure your setup is appropriate
verveanix
commented
10 months ago Thank you - the resolution to my issue (as documented somewhat opaquely on that doc) was to add a private endpoint for the public storage accounts, which allowed the traffic to go through.
Which version of the AzCopy was used?
10.21.2
Which platform are you using? (ex: Windows, Mac, Linux)
Linux
What command did you run?
azcopy sync --recursive=true --exclude-pattern <excluded> --delete-destination=true --s2s-preserve-access-tier=false https://[staging-host].blob.core.windows.net/mycontainer/?se=2023-11-14t16%3A00%3A22z&sig=-REDACTED-&ske=2023-11-14t16%3A00%3A22z&skoid=660d6430-659f-4878-861d-fb5f3b219247&sks=b&skt=2023-11-14t14%3A30%3A23z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rl&sr=c&sv=2023-01-03 https://[production-host].blob.core.windows.net/mycontainer/?se=2023-11-14t16%3A00%3A22z&sig=-REDACTED-&ske=2023-11-14t16%3A00%3A22z&skoid=660d6430-659f-4878-861d-fb5f3b219247&sks=b&skt=2023-11-14t14%3A30%3A24z&sktid=07e4ccf9-8832-40b3-9731-2709e0742154&skv=2023-01-03&sp=rcwdl&sr=c&sv=2023-01-03Code that generates the SAS tokens (uses Powershell Az module):
Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.
What problem was encountered?
Some of the files synced fine ; here's an example of a failed transfer:
[staging-host] is a Storage Account with public access completely disabled and a private endpoint configured on a private VNet.
[production-host] is a Storage Account with public access enabled.
The system running the code is on a VM with access to both Storage Accounts.
How can we reproduce the problem in the simplest way?
Have you found a mitigation/solution?
No