search

Azure / azure-storage-python

Microsoft Azure Storage Library for Python
https://azure-storage.readthedocs.io
MIT License
339 stars 241 forks source link

SDK currently does not address the scenario where a large file needs to be downloaded serially to a non-seekable stream #464

Closed williexu closed 6 years ago

williexu commented 6 years ago

Which service(blob, file, queue) does this issue concern?

blob/file

What problem was encountered?

referencing with some edits: https://github.com/Azure/azure-cli/issues/6608:

Currently the max_connections=1 arguments for downloads will only attempt to pull the larger files in one piece. When the file it is trying to download is larger than the memory on the workstation running the task the process consumes the workstation's entire memory and swap file then fails.

Looking to get the download methods to treat the max_connections=1 like it does when the max connections is set to 2 or more. That is to handle files larger than the memory of the workstation by breaking the file into multiple peices and make multiple requests to get the full file to download.

The Customer needs to download the blob to a named pipe so they must be able to use a single download connection to prevent seek operations on the pipe

zezha-msft commented 6 years ago

Hi @williexu, thanks for creating the issue.

I'll update this thread once this work is scheduled on our end.

v-tisheg commented 6 years ago

Has this been published?
Does the customer need to download a new version of the Azure CLI? He tested the command and got the same results.

zezha-msft commented 6 years ago

No @v-tisheg, this work will be scheduled soon.

zezha-msft commented 6 years ago

Hi @williexu and @v-tisheg, this issue has been fixed in 1.3.1.