Azure / azure-cli

Azure Command-Line Interface
MIT License
3.96k stars 2.94k forks source link

az storage blob upload - support for standard base16 (hex) md5sum #25020

Open hholst80 opened 1 year ago

hholst80 commented 1 year ago

As good as nobody in the world uses base64 encoded md5sums and most people would not even know how to generate it from a normal base16 encoded md5sum.

Please support both. If overhead is of concern consider compression. JSON is a large overhead format so the extra "compression" of base64 should be irrelevant here vis. user workflows.

trainer:/workdir/eee# az storage blob upload -c models --file model.onnx --name p11808/model.onnx --overwrite=true --content-md5 aba1a2322a80eee3b400679c8a50986b
Argument '--overwrite' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Finished[#############################################################]  100.0000%
The MD5 value specified in the request is invalid. MD5 value must be 128 bits and base64 encoded.
RequestId:208740f5-301e-008b-07b2-1e6451000000
Time:2023-01-02T13:57:18.5449066Z
ErrorCode:InvalidMd5
trainer:/workdir/eee#
hholst80 commented 1 year ago

In case anyone attempts to do this, here is a Python snippet:

>>> base64.b64encode(int("aba1a2322a80eee3b400679c8a50986b",16).to_bytes(16,'big')).decode()
'q6GiMiqA7uO0AGecilCYaw=='
>>>
hholst80 commented 1 year ago

Also, from a user workflow perspective, it is very unclear if this option even works when it is accepted.

The output hints on that the md5 is not even used:

trainer:/workdir/eee# az storage blob upload -c models --file model.onnx --name p11808/model.onnx --overwrite=true --content-md5 q6GiMiqA7uO0AGecilCYaw==
Argument '--overwrite' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Finished[#############################################################]  100.0000%
{
  "client_request_id": "d64a453c-8aa7-11ed-b97a-3a3f0cb41999",
  "content_crc64": "BqP5S4LU38E=",
  "content_md5": null,
  "date": "2023-01-02T14:14:56+00:00",
  "encryption_key_sha256": null,
  "encryption_scope": null,
  "etag": "\"0x8DAECCBBA844E98\"",
  "lastModified": "2023-01-02T14:14:57+00:00",
  "request_id": "c3746311-301e-0040-4ab4-1e6704000000",
  "request_server_encrypted": true,
  "version": "2021-06-08",
  "version_id": null
}
yonzhan commented 1 year ago

@evelyn-ys for awareness

ghost commented 1 year ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details
As good as nobody in the world uses base64 encoded md5sums and most people would not even know how to generate it from a normal base16 encoded md5sum. Please support both. If overhead is of concern consider compression. JSON is a large overhead format so the extra "compression" of base64 should be irrelevant here vis. user workflows. ``` trainer:/workdir/eee# az storage blob upload -c models --file model.onnx --name p11808/model.onnx --overwrite=true --content-md5 aba1a2322a80eee3b400679c8a50986b Argument '--overwrite' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus Finished[#############################################################] 100.0000% The MD5 value specified in the request is invalid. MD5 value must be 128 bits and base64 encoded. RequestId:208740f5-301e-008b-07b2-1e6451000000 Time:2023-01-02T13:57:18.5449066Z ErrorCode:InvalidMd5 trainer:/workdir/eee# ```
Author: hholst80
Assignees: evelyn-ys
Labels: `Storage`, `Service Attention`, `customer-reported`, `feature-request`, `Auto-Assign`, `Azure CLI Team`
Milestone: Backlog
hholst80 commented 1 year ago

The situation is not less obscure in Windows PowerShell:

https://stackoverflow.com/questions/62086677/azure-blob-upload-with-content-md5-set