This PR includes HTTP info collected from Azure Storage in the explain analyze statement output.
changes:
utilizes theAzure::Core::Diagnostics::Logger to capture logs and uses a global instance of http_state to maintain stats
added an azure_http_stats setting option, which enables this feature (default value is false)
expanded the azure.test with additional statements
notes/limitations:
guarded global variables are used to allow sharing the stats across multiple files in parsing
the results may be inaccurate for more than one active duckdb connection
the extraction of received and sent bytes is not implemented. (I am not very familiar with Azure Storage, so) here are some links that may be useful for a future try:
after an HTTP request is sent to the storage service, Azure logger sends a response message with info related to the request status. There is a content-length field whose value, if the request was successful, is equal to the packet size. If a request is unsuccessful, this value may not be equal to that.
source: Azure Storage Documentation - log format
samansmink
commented
11 months ago
This PR includes HTTP info collected from Azure Storage in the
explain analyzestatement output.changes:
Azure::Core::Diagnostics::Loggerto capture logs and uses a global instance ofhttp_stateto maintain statsazure_http_statssetting option, which enables this feature (default value isfalse)azure.testwith additional statementsnotes/limitations:
content-lengthfield whose value, if the request was successful, is equal to the packet size. If a request is unsuccessful, this value may not be equal to that. source: Azure Storage Documentation - log format