AzCopy v10 does not appear to work with static web sites

jflam commented 5 years ago

Which version of the AzCopy was used?

v10.0.4.preview

Note: The version is visible when running AzCopy without any argument

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

copy

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

What problem was encountered?

I'm trying to copy to an static web site container, i.e., $web. I can copy just fine to a container ("foo") that I create in the same storage account. However, any attempt to copy to the $web container fails.

Here's a sample snippet from the log files with sig elided:

2019/01/04 02:00:27 ==> REQUEST/RESPONSE (Try=1/158.0369ms, OpTime=158.0369ms) -- RESPONSE SUCCESSFULLY RECEIVED
   PUT https://jlamiunknownblogstorage.blob.core.windows.net/=2019-01-04T01:45:09Z&se=2019-01-05T01:45:09Z&sp=rwdl&sv=2018-03-28&sr=c&sig=<elided>=/public/css/academicons.min.css?timeout=901
   Content-Length: [4087]
   User-Agent: [AzCopy/10.0.4-Preview Azure-Storage/0.3 (go1.10.3; Windows_NT)]
   X-Ms-Blob-Cache-Control: []
   X-Ms-Blob-Content-Disposition: []
   X-Ms-Blob-Content-Encoding: []
   X-Ms-Blob-Content-Language: []
   X-Ms-Blob-Content-Type: [text/css; charset=utf-8]
   X-Ms-Blob-Type: [BlockBlob]
   X-Ms-Client-Request-Id: [b499a4a2-243e-45fe-5616-0bea39b2ef2a]
   X-Ms-Version: [2018-03-28]
   --------------------------------------------------------------------------------
   RESPONSE Status: 404 The specified resource does not exist.
   Content-Length: [223]
   Content-Type: [application/xml]
   Date: [Fri, 04 Jan 2019 02:00:26 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [ResourceNotFound]
   X-Ms-Request-Id: [d99242e1-101e-007f-33d1-a3400e000000]
   X-Ms-Version: [2018-03-28]

How can we reproduce the problem in the simplest way?

Run the copy using a SAS token generated for the $web container.

Have you found a mitigation/solution?

No.

jflam commented 5 years ago

I wanted to add some color to my scenario: I'm trying to build a continuous integration Azure Pipeline that will generate a static website using hugo on a Ubuntu 16.04 LTS worker. Ideally I would:

Install hugo
Install azcopy
git clone my blog
run the hugo command in the root of my blog dir
azcopy the generated public directory to my blob storage account in the $web container

This does not appear possible based on my experiments. I might be able to do a convoluted workaround using the az storage blob batch-upload command as well.

If you search for "hugo azure pipeline azure blob storage" there's a bunch of folks who have gone down this path in a semi-automated fashion. The missing piece seems to be azcopy supporting copying to $web containers.

Is there an ETA for this support?

jflam commented 5 years ago

Updating this thread: I do have a workaround using az storage blob batch-upload:

az storage blob upload-batch --account-name <name> --source . -d \$web --account-key <storage account key>

Escaping the $ in $web is necessary as I'm running in bash.

I think that AzCopy will ultimately be a better solution than upload-batch because:

The sync command which should be more efficient over the wire
Avoid having to install all the other unnecessary software that comes with the Azure CLI which currently takes a long time on my build VM.

Does that make sense?

zezha-msft commented 5 years ago

Hi @jflam, thanks for reaching out!

AzCopy already supports uploading to $web, could you please make sure to escape the $ properly in the URL?

jflam commented 5 years ago

Thanks. There are two things that I found that I needed to do:

Ensure that $ is escaped correctly (even when it is in quotes)
Ensure that both the origin and the destination are in quotes

I can't remember if this was documented anywhere. I seem to recall seeing doc folks not wanting to document things specifically for bash. I think this is a mistake.

Super happy with my Azure Pipelines powered automated build for my blog. Now I just have to finish struggling with getting SSL enabled (that also suffers from doc problems).

I really like the fact that v3 is compiled to a single executable. Please do make an effort to make sure that older docs point to the latest version (and to make clear what the benefits of v3 are!).

zezha-msft commented 5 years ago

Hi @jflam, thanks for the suggestions!

We do show the quotes in the example when you run azcopy cp --help, but in hindsight we should probably show single quotes instead of double quotes.

Azure / azure-storage-azcopy