search

Azure / azure-sdk-for-cpp

This repository is for active development of the Azure SDK for C++. For consumers of the SDK we recommend visiting our versioned developer docs at https://azure.github.io/azure-sdk-for-cpp.
MIT License
173 stars 122 forks source link

Include `WWW-Authenticate`, `x-ms-version`, `x-ms-date` to the default list of non-redacted headers in the log sanitizer #5782

Closed mmaitre314 closed 2 weeks ago

mmaitre314 commented 1 month ago

Is your feature request related to a problem? Please describe.

I am debugging REST requests to a Storage Account hitting auth failures in DuckDB and the logs redacted headers like WWW-Authenticate, x-ms-version, x-ms-date which could have helped root-causing the issue.

[2024-07-11T12:09:11.2782773Z T: 23464] INFO  : HTTP Request : HEAD https://azduckdb.blob.core.windows.net/data/HuggingFaceFW/fineweb/CC-MAIN-2013-20/000_00000.parquet
authorization : REDACTED
user-agent : azsdk-cpp-storage-blobs/12.10.0 (Windows 10 Enterprise 6.3 22631 22621.1.amd64fre.ni_release.220506-1250)
x-ms-client-request-id : ebac0027-0250-42aa-87dc-91b1713ab0ef
x-ms-date : REDACTED
x-ms-version : REDACTED

[2024-07-11T12:09:12.2519756Z T: 23464] INFO  : HTTP Response (972ms) : 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
date : Thu, 11 Jul 2024 12:09:10 GMT
server : Microsoft-HTTPAPI/2.0
transfer-encoding : chunked
www-authenticate : REDACTED
x-ms-error-code : REDACTED
x-ms-request-id : b596ec8d-701e-0010-2a8b-d36a02000000

Including x-ms-error-code might also be helpful but I could not find documentation on this header to understand what it can contain.

Describe the solution you'd like

Include WWW-Authenticate, x-ms-version, x-ms-date in g_defaultAllowedHttpHeaders

If this sounds OK I can likely contribute a PR.

Describe alternatives you've considered

It looks like DuckDB could customize redacted headers but doing that in the Azure SDK would be a more generic solution.

Additional context

Related issue: #3708

WWW-Authenticate is included in .NET and Java SDKs here:

x-ms-version and x-ms-date are not included there but the Date header is and the api-version query parameter also is, so that seems like a reasonable inclusion.

Information Checklist

Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

github-actions[bot] commented 1 month ago

Thank you for your feedback. Tagging and routing to the team member best able to assist.

ahsonkhan commented 1 month ago

@mmaitre314 for context and better understanding, which services and C++ client SDK are you using in your application, that motivated this? Is it storage blobs and the BlobClient or some other service/SDK?

mmaitre314 commented 1 month ago

DuckDB uses Azure::Storage::Blobs::BlobServiceClient. Here is the code: https://github.com/duckdb/duckdb_azure/blob/097bb13aadb186ca43ae9b5dc6a21c20e56ad4dd/src/azure_storage_account_client.cpp#L329

mmaitre314 commented 1 month ago

Investigating an issue with HTTP range requests, it would also be helpful to include x-ms-range,range, accept-ranges, content-range:

[2024-07-15T11:19:48.3297721Z T: 7ff86e6e9700] INFO  : HTTP Request : GET https://<snip>.blob.core.windows.net/<snip>.gz.parquet?se=REDACTED&sig=REDACTED&ske=REDACTED&skoid=REDACTED&sks=REDACTED&skt=REDACTED&sktid=REDACTED&skv=REDACTED&sp=REDACTED&sr=REDACTED&st=REDACTED&sv=REDACTED
if-match : "0x8DC7356EF4FDCDA"
user-agent : azsdk-cpp-storage-blobs/12.10.0 (Linux 5.15.0-1064-azure x86_64 #73~20.04.1-Ubuntu SMP Mon May 6 09:43:44 UTC 2024)
x-ms-client-request-id : 8afbe7a1-48fd-4857-8c86-47047fd756b9
x-ms-date : REDACTED
x-ms-range : REDACTED
x-ms-version : REDACTED
[2024-07-15T11:19:48.3297856Z T: 7ff86e6e9700] DEBUG : [CURL Transport Adapter]: Creating a new session.
[2024-07-15T11:19:48.3297975Z T: 7ff86e6e9700] DEBUG : [CURL Transport Adapter]: Re-using connection from the pool.
[2024-07-15T11:19:48.3298108Z T: 7ff86e6e9700] DEBUG : [CURL Transport Adapter]: No Host in request headers. Adding it
[2024-07-15T11:19:48.3298239Z T: 7ff86e6e9700] DEBUG : [CURL Transport Adapter]: Send request without payload
[2024-07-15T11:19:48.3298616Z T: 7ff86e6e9700] DEBUG : [CURL Transport Adapter]: Parse server response
[2024-07-15T11:19:48.3484566Z T: 7ff86e6e9700] DEBUG : [CURL Transport Adapter]: Request completed. Moving response out of session and session to response.
[2024-07-15T11:19:48.3484913Z T: 7ff86e6e9700] INFO  : HTTP Response (18ms) : 206 Partial Content
accept-ranges : REDACTED
content-length : 1048576
content-range : REDACTED
content-type : application/octet-stream
date : Mon, 15 Jul 2024 11:19:47 GMT
etag : "0x8DC7356EF4FDCDA"
last-modified : Mon, 13 May 2024 14:13:59 GMT
server : Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-blob-type : REDACTED
x-ms-client-request-id : 8afbe7a1-48fd-4857-8c86-47047fd756b9
x-ms-copy-completion-time : REDACTED
x-ms-copy-id : REDACTED
x-ms-copy-progress : REDACTED
x-ms-copy-source : REDACTED
x-ms-copy-status : REDACTED
x-ms-creation-time : REDACTED
x-ms-group : REDACTED
x-ms-lease-state : REDACTED
x-ms-lease-status : REDACTED
x-ms-meta-hdi_permission : REDACTED
x-ms-owner : REDACTED
x-ms-permissions : REDACTED
x-ms-request-id : 74787e37-f01e-0024-20a8-d61070000000
x-ms-resource-type : REDACTED
x-ms-server-encrypted : REDACTED
x-ms-version : REDACTED
[2024-07-15T11:19:48.3485269Z T: 7ff86e6e9700] INFO  : HTTP status code 206 won't be retried.