Closed marco-rossi29 closed 5 years ago
Hi @marco-rossi29, thanks for reaching out!
Could you please clarify your use case? Are you trying to download a blob (in its entirety) while it's being modified by another process/user?
Which blob type are you using?
Hi Ze, Thank you for your answer: Our logic is here
We are using BlockBlobService(connection_string=conn_string)
(see here)
The use case is: there is a blob in Azure Storage that is being modified by another process/user and I'd like to download the blob, but I'm happy to not download any new portion of it.
To clarify, say I'm accessing Azure Storage and the file I want is 100MB. As I'm downloading it, another process increases the file to 110MB. I'm happy to download only the 100MB as long as the get_blob_to_path
command shouldn't throw. Currently, the workaround to avoid throwing is by setting max_connections = 1
. However, since v1.3.1 you removed such workaround, and even with max_connections = 1
the program throws
Hi @marco-rossi29, thanks for the clarification!
I assume your blob is being manipulated in an append-only pattern, right? E.g. the extra 10MB of data was added at the end of the blob.
To be clear, the SDK is locking onto the blob's etag to protect the integrity of the downloaded data. In most use cases, there's no guarantee that the data is being modified in a way that still allows the downloaded data to be "good", but if you are only appending blocks at the end of the blob, then you could work around the protection by specifying if_match=*
.
Perfect, that's a great tip and it will probably solve my issue. Thank you very much for your help!
Note: I understand that this breaking change was done on purpose in commit
8f68597815022e20df5df5402508a2569d8e6bfe
. Is there a better way to download a blob that may be written?Which service(blob, file, queue) does this issue concern?
get_blob_to_path()
Which version of the SDK was used? Please provide the output of
pip freeze
.azure-storage-blob-1.3.0 works azure-storage-blob-1.3.1 doesn't work
What problem was encountered?
download fails:
Have you found a mitigation/solution?
I downgraded to 1.3.0