search

Azure / blobxfer

Azure Storage transfer tool and data movement library
MIT License
150 stars 39 forks source link

Support for blobs that don't require SAS #132

Open BMurri opened 2 years ago

BMurri commented 2 years ago

Feature Request Description

I'm not sure if this is a feature request or a bug report. That answer is contingent on whether anonymous (non-SAS) access to Azure Storage blobs is supported.

Describe Preferred Solution

Use the following command line: blobxfer download --storage-url "https://dragentestdata.blob.core.windows.net/reference-genomes/Hsapiens/hash-tables/hg38_altaware_nohla-cnv-anchored.v8.tar" --local-path "/cromwell-executions/PairedFastqToVcf/b1ec8b8f-8f6b-4190-b87c-57ff65782d87/call-DRAGEN/inputs/dragentestdata.blob.core.windows.net/reference-genomes/Hsapiens/hash-tables/hg38_altaware_nohla-cnv-anchored.v8.tar" --chunk-size-bytes 104857600 --rename --include 'Hsapiens/hash-tables/hg38_altaware_nohla-cnv-anchored.v8.tar'

Expected result: Successful download of the tar file (note that the url works in Edge and other browsers without any issues)

Actual result:

Traceback (most recent call last): File "/usr/local/bin/blobxfer", line 33, in sys.exit(load_entry_point('blobxfer==1.11.0', 'console_scripts', 'blobxfer')()) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1137, in call return self.main(args, kwargs) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 763, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 84, in new_func return ctx.invoke(f, obj, *args, *kwargs) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 763, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.9/site-packages/blobxfer-1.11.0-py3.9.egg/blobxfer_cli/cli.py", line 1094, in download ctx.initialize(settings.TransferAction.Download) File "/usr/local/lib/python3.9/site-packages/blobxfer-1.11.0-py3.9.egg/blobxfer_cli/cli.py", line 71, in initialize self.credentials = settings.create_azure_storage_credentials( File "/usr/local/lib/python3.9/site-packages/blobxfer-1.11.0-py3.9.egg/blobxfer_cli/settings.py", line 425, in create_azure_storage_credentials creds.add_storage_account(name, key, endpoint) File "/usr/local/lib/python3.9/site-packages/blobxfer-1.11.0-py3.9.egg/blobxfer/operations/azure/init.py", line 69, in add_storage_account self._storage_accounts[name] = StorageAccount( File "/usr/local/lib/python3.9/site-packages/blobxfer-1.11.0-py3.9.egg/blobxfer/operations/azure/init.py", line 106, in init raise ValueError( ValueError: no authentication credential given for storage account: dragentestdata

Describe Alternatives Considered

Additional Context

This is running in the linux blobxfr container on a CentOS 7.

alfpark commented 1 year ago

Public container downloads can be processed by other general tooling such as curl, wget, powershell or other generally available tooling. It is not currently supported in blobxfer and most likely will not be unless provided by the community. blobxfer focuses on authenticated workflows.