databricks / databricks-sdk-go

Databricks SDK for Go
https://docs.databricks.com/dev-tools/sdk-go.html
Apache License 2.0
46 stars 38 forks source link

Add traceparent header to enable distributed tracing. #914

Closed mgyucht closed 1 month ago

mgyucht commented 1 month ago

Changes

Currently, it is difficult to correlate requests made by a client with a request made by a backend service, especially if the same request is made multiple times. The REST API supports the Trace Context standard which defines a standard way of propagating tracing information through HTTP headers.

This PR implements a traceparent generator to construct new traceparent headers in accordance with this standard. These traceparents are attached to the headers of each individual request. The resulting header is visible when debug logs are enabled and DATABRICKS_DEBUG_HEADERS is set.

Tests

Added a unit test to ensure that new traceparents are set for every request.

It's hard to test that a debug log actually contains this header without rewriting a lot of the matching logic for debug logs, but I did need to remove traceparent from those unit tests, suggesting that debug logging does include traceparent (like any other header).

Manually tested that the traceparent set by the client is visible to the server by checking our internal access logs.