Azure / blobxfer

Azure Storage transfer tool and data movement library
MIT License
151 stars 38 forks source link

Cannot download file in a subfolder when using container SAS and no-recursive #123

Closed tonybendis closed 3 years ago

tonybendis commented 4 years ago

Problem Description

Blobxfer fails to download a blob when the following conditions are met:

  1. Blob sits in a "subfolder"
  2. SAS token is at the container level (read+list)
  3. no-recursive option is specified

Azure blobxfer parameters output

         blobxfer version: 1.9.4
                 platform: Linux-5.3.0-1034-azure-x86_64-with
               components: CPython=3.7.5-64bit azstor.blob=2.1.0 azstor.file=2.1.0 crypt=2.8 req=2.22.0
       transfer direction: Azure -> local
                  workers: disk=4 xfer=3 (msoc=8) md5=0 crypto=0
                 log file: None
                  dry run: False
              resume file: None
                  timeout: connect=10 read=200 max_retries=1000
                     mode: StorageModes.Auto
                  skip on: fs_match=False lmt_ge=False md5=False
                   delete: extraneous=False only=False
                overwrite: True
                recursive: True
            rename single: True
         strip components: 0
         chunk size bytes: 0
         compute file md5: False
       restore properties: attr=False lmt=False
          rsa private key: None
        local destination: /tmp/xyz

Steps to Reproduce

WORKS when blob is in container root: docker run -it mcr.microsoft.com/blobxfer download --storage-url 'https://abendiscromwell.blob.core.windows.net/httptest/Hello.txt?sv=2019-02-02&st=2020-08-17T19%3A20%3A24Z&se=2020-12-15T20%3A20%3A00Z&sr=c&sp=rl&sig=JqUHAvXX3XVwZoHOx6H8d8nLqcqx6BbAv5HCnJXBbmQ%3D' --local-path '/tmp/Hello.txt' --rename --no-recursive -v --enable-azure-storage-logger

FAILS (doesn't download), when blob is in "subfolder": docker run -it mcr.microsoft.com/blobxfer download --storage-url 'https://abendiscromwell.blob.core.windows.net/httptest/subfolder/Hello.txt?sv=2019-02-02&st=2020-08-17T19%3A20%3A24Z&se=2020-12-15T20%3A20%3A00Z&sr=c&sp=rl&sig=JqUHAvXX3XVwZoHOx6H8d8nLqcqx6BbAv5HCnJXBbmQ%3D' --local-path '/tmp/Hello.txt' --rename --no-recursive -v --enable-azure-storage-logger

Expected Results

1 remote files processed, waiting for download completion of approx. 0.0000 MiB

Actual Results

0 remote files processed, waiting for download completion of approx. 0.0000 MiB

Additional Logs

Ensure that you execute your command with -v --enable-azure-storage-logger options.

INSERT ADDITIONAL LOGS HERE

Additonal Comments

alfpark commented 3 years ago

This is expected behavior with --no-recursive. Only blobs at the container level will be downloaded.