search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
610 stars 220 forks source link

azcopy sync to storage account that doesn’t exists returns no error and keep retrying until it’s timeout. #1685

Closed v-takebi closed 2 years ago

v-takebi commented 2 years ago

Which version of the AzCopy was used?

version 10.13.0

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

azcopy sync src-path https://storageaccount.blob.core.windows.net/container

What problem was encountered?

azcopy sync doesn’t fail with error but keep retrying even if the storage account doesn’t exists.

How can we reproduce the problem in the simplest way?

To run this command to a storage account that doesn't exists.

Have you found a mitigation/solution?

No.

Description

I’d like to ask you if the following azcopy’s behavior is designed. azcopy sync doesn’t fail and keep retrying even if the storage account doesn’t exists.

If it fails when the client get the first “dial tcp: lookup storageaccountthatdoesntexists.blob.core.windows.net: no such host” and write error messages on the console, I think it’s more user friendly. If it's a designed behavior, I’d l like you to think of fix this behavior in the future updates if possible.

The process keep retrying until time is up.

PS C:\Users\takebi> azcopy login
PS C:\Users\takebi> azcopy sync <src-path> https://storageaccountthatdoesntexists/container
INFO: Authenticating to destination using Azure AD
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job *** has started
Log file is located at: C:\Users\username\.azcopy\****.log

1 Files Scanned at Source, 0 Files Scanned at Destination

In the azcopy’s log, the following logs are recorded.

2022/02/25 00:49:51 ==> REQUEST/RESPONSE (Try=1/68.308ms, OpTime=68.308ms) -- REQUEST ERROR
   HEAD https://storageaccountthatdoesntexists.blob.core.windows.net/container?timeout=901
   Authorization: REDACTED
   User-Agent: [AzCopy/10.13.0 Azure-Storage/0.14 (go1.16; Windows_NT)]
   X-Ms-Client-Request-Id: [****]
   X-Ms-Version: [2019-12-12]
   --------------------------------------------------------------------------------
   ERROR:
-> github.com/Azure/azure-pipeline-go/pipeline.NewError, /home/vsts/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.3/pipeline/error.go:157
HTTP request failed

Head "https://storageaccountthatdoesntexists.blob.core.windows.net/container?timeout=901": dial tcp: lookup storageaccountthatdoesntexists.blob.core.windows.net: no such host

goroutine 1 [running]:
github.com/Azure/azure-storage-azcopy/v10/ste.stack(0xc00003ab10, 0xc0003e5100, 0x0)
    /home/vsts/work/1/s/ste/xferLogPolicy.go:237 +0xa5
github.com/Azure/azure-storage-azcopy/v10/ste.NewRequestLogPolicyFactory.func1.1(0x19f1b80, 0xc000278480, 0xc0003e5100, 0x10, 0x0, 0x0, 0xc000274f20)
    /home/vsts/work/1/s/ste/xferLogPolicy.go:151 +0x7c5
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc000081db0, 0x19f1b80, 0xc000278480, 0xc0003e5100, 0xc000274fa8, 0x0, 0xc000be0910, 0xd160b6)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x4b
github.com/Azure/azure-storage-azcopy/v10/ste.NewVersionPolicyFactory.func1.1(0x19f1b80, 0xc000278480, 0xc0003e5100, 0xd1557e, 0xc000ef4007, 0xc000eae000, 0x2af7)
    /home/vsts/work/1/s/ste/mgr-JobPartMgr.go:83 +0x1c9
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc0000946a8, 0x19f1b80, 0xc000278480, 0xc0003e5100, 0x2af7, 0x2af7, 0x1, 0xc000ef4000)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x4b
github.com/Azure/azure-storage-blob-go/azblob.responderPolicy.Do(0x19e1e00, 0xc0000946a8, 0xc000ea0790, 0x19f1b80, 0xc000278480, 0xc0003e5100, 0x2, 0x2, 0xc000ef4000, 0xc000274f20)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.13.1-0.20210823171415-e7932f52ad61/azblob/zz_generated_responder_policy.go:33 +0x5a
github.com/Azure/azure-storage-blob-go/azblob.(*tokenCredential).New.func1(0x19f1b80, 0xc000278480, 0xc0003e5100, 0xcceebf, 0x10, 0x1754e00, 0x1346f4b4b01)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.13.1-0.20210823171415-e7932f52ad61/azblob/zc_credential_token.go:135 +0x212
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc000beaee0, 0x19f1b80, 0xc000278480, 0xc0003e5100, 0xd18c2e2800, 0x0, 0xc0003f4360, 0xc0003f4360)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x4b
github.com/Azure/azure-storage-azcopy/v10/ste.(*retryNotificationPolicy).Do(0xc00037c4c0, 0x19f1b80, 0xc000278480, 0xc0003e5100, 0x0, 0xc000278490, 0x18d8878, 0xc000be0c00)
    /home/vsts/work/1/s/ste/xferRetryNotificationPolicy.go:59 +0x62
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc00037c4d0, 0x19f1b80, 0xc000278480, 0xc0003e5100, 0xc000278480, 0xc00037c5b0, 0xc000000001, 0x0)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x4b
github.com/Azure/azure-storage-azcopy/v10/ste.NewBlobXferRetryPolicyFactory.func1.1(0x19f1bb8, 0xc0003c90b0, 0xc0003e5000, 0x10, 0x16df920, 0x64492d747301, 0xc000274dc0)
    /home/vsts/work/1/s/ste/xferRetrypolicy.go:384 +0x762
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc000081e00, 0x19f1bb8, 0xc0003c90b0, 0xc0003e5000, 0xc000274e78, 0x20, 0x19c725a, 0xc000be0f90)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x4b
github.com/Azure/azure-storage-blob-go/azblob.NewUniqueRequestIDPolicyFactory.func1.1(0x19f1bb8, 0xc0003c90b0, 0xc0003e5000, 0x10, 0x16df920, 0xccee01, 0xc000274dc0)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.13.1-0.20210823171415-e7932f52ad61/azblob/zc_policy_unique_request_id.go:22 +0xd4
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc000094738, 0x19f1bb8, 0xc0003c90b0, 0xc0003e5000, 0xc000274e60, 0x36, 0xc0000be480, 0xc000be1048)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x4b
github.com/Azure/azure-storage-blob-go/azblob.NewTelemetryPolicyFactory.func1.1(0x19f1bb8, 0xc0003c90b0, 0xc0003e5000, 0x1, 0x0, 0x1, 0xc0000060a8)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.13.1-0.20210823171415-e7932f52ad61/azblob/zc_policy_telemetry.go:34 +0x169
github.com/Azure/azure-pipeline-go/pipeline.PolicyFunc.Do(0xc0003c9980, 0x19f1bb8, 0xc0003c90b0, 0xc0003e5000, 0xc0003c9980, 0x0, 0xc000be1118, 0xcceebf)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.3/pipeline/core.go:43 +0x4b
github.com/Azure/azure-pipeline-go/pipeline.(*pipeline).Do(0xc0000dc2c0, 0x19f1bb8, 0xc0003c90b0, 0x19e1f00, 0xc000ea0790, 0xc0003e5000, 0x19, 0xc0000a8360, 0x5, 0x0)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-pipeline-go@v0.2.3/pipeline/core.go:129 +0x88
github.com/Azure/azure-storage-blob-go/azblob.blobClient.GetProperties(0xc0001bb4d0, 0x5, 0x0, 0x0, 0x0, 0xc0001bb4d8, 0x19, 0xc0000a8360, 0x5, 0x0, ...)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.13.1-0.20210823171415-e7932f52ad61/azblob/zz_generated_blob.go:1009 +0x405
github.com/Azure/azure-storage-blob-go/azblob.BlobURL.GetProperties(0xc0001bb4d0, 0x5, 0x0, 0x0, 0x0, 0xc0001bb4d8, 0x19, 0xc0000a8360, 0x5, 0x0, ...)
    /home/vsts/go/pkg/mod/github.com/!azure/azure-storage-blob-go@v0.13.1-0.20210823171415-e7932f52ad61/azblob/url_blob.go:188 +0x17f
github.com/Azure/azure-storage-azcopy/v10/cmd.(*blobTraverser).getPropertiesIfSingleBlob(0xc000278300, 0x5, 0x0, 0x0, 0x0)
    /home/vsts/work/1/s/cmd/zc_traverser_blob.go:93 +0x428
github.com/Azure/azure-storage-azcopy/v10/cmd.(*blobTraverser).Traverse(0xc000278300, 0x0, 0xc00003eec0, 0x201dfa0, 0x0, 0x0, 0x0, 0x0)
    /home/vsts/work/1/s/cmd/zc_traverser_blob.go:129 +0xe5
github.com/Azure/azure-storage-azcopy/v10/cmd.(*syncEnumerator).enumerate(0xc000bbc370, 0x19f1b00, 0xc0003c90b0)
    /home/vsts/work/1/s/cmd/zc_enumerator.go:631 +0xfb
github.com/Azure/azure-storage-azcopy/v10/cmd.(*cookedSyncCmdArgs).process(0xc000262a80, 0x62, 0x0)
    /home/vsts/work/1/s/cmd/sync.go:683 +0x425
github.com/Azure/azure-storage-azcopy/v10/cmd.init.19.func2(0xc0003d3900, 0xc0003d0400, 0x2, 0x4)
    /home/vsts/work/1/s/cmd/sync.go:718 +0x17a
github.com/spf13/cobra.(*Command).execute(0xc0003d3900, 0xc0003d03c0, 0x4, 0x4, 0xc0003d3900, 0xc0003d03c0)
    /home/vsts/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:860 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x1fc9e20, 0xe139882638f8827e, 0x30, 0x17975e0)
    /home/vsts/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
    /home/vsts/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
github.com/Azure/azure-storage-azcopy/v10/cmd.Execute(0xc0000aad50, 0x17, 0xc0000aad50, 0x17, 0xc0000a7120, 0x1d, 0x7fffffff)
    /home/vsts/work/1/s/cmd/root.go:165 +0xfa
main.main()
    /home/vsts/work/1/s/main.go:82 +0x397
mohsha-msft commented 2 years ago

Hey @v-takebi ,

Thanks for reaching out! AzCopy has retry with exponential back-off policy to retry a failed request 20 times. There can be n number of network errors while transferring data over internet and AzCopy cannot account for all of them. Afterall, an error may or may not be fatal. If not fatal, it may be transient and go away with retries. So, it is by design.

It is generally expected to have correct StorageAccountName in the source/destination URLs provided to AzCopy command.

v-takebi commented 2 years ago

Mohit Sharma-san,

Thanks for your quick and precise answer. Understand. This behavior is not about azcopy but networking, and should be by design. We'd like to use some conditional logic to validate before running into azcopy's process.

Many thanks as always! :)

mohsha-msft commented 2 years ago

Glad you found that useful!

You can use Azure CLI to get the properties of a storage Account. Please find this link

If you're invoking AzCopy from a program, I suggest you write a simple REST API call to get properties of a storage account. If it fails, it means that storage account wasn't valid.

v-takebi commented 2 years ago

Thanks for your alternative plan! Sure. I will use it to get better at it myself.

pdarvasi commented 3 months ago

AzCopy has retry with exponential back-off policy to retry a failed request 20 times. There can be n number of network errors while transferring data over internet and AzCopy cannot account for all of them. Afterall, an error may or may not be fatal. If not fatal, it may be transient and go away with retries. So, it is by design.

The concern here is that AzCopy retries even 401 and 403 HTTP responses, your argument does not stand as HTTP error codes could determine what calls do not make sense to retry. The HTTP codes valid for retry are e.g. 408, 502, 503 and 504. It seems to me that it was easier on your side not to deal with this problem at all ,which is sad.