This pull request adds more instrumentation in different areas of the project. It includes multiple commits addressing different concerns. The main four changes are:
It configures a tracer provider in both enduro-a3m-worker and enduro-am-worker (enduro was already configured) and enables telemetry passing environment strings to the corresponding k8s workloads,
It enables automatic instrumentation of database/sql via github.com/XSAM/otelsql,
It enables automatic instrumentation of the Temporal Go SDK using a client interceptor, and
It enables route tagging in Goa (required to bump its version).
With these changes, we'll see a significant increase in the number of traces captured by Tempo. We're not yet initiating a trace during the transfer submission via MinIO; however, an interesting example that can already be observed is the move workflow initiated through the API. If you've already moved a package, use this query to find related traces:
{ .http.route="/package/{id}/move" }
It should look like this:
Something that becomes apparent right away is the lag between activity executions - we know that's the cost of using Temporal to schedule activities. It also shows that local activities are much faster since they're not coordinated with the Temporal server.
You can also search for a workflow directly, e.g.:
{name="StartWorkflow:processing-workflow"}
This trace is interesting because it shows that the telemetry data originates from multiple services, specifically "enduro" and "enduro-a3m-worker", rather than a singular source.
This pull request adds more instrumentation in different areas of the project. It includes multiple commits addressing different concerns. The main four changes are:
enduro-a3m-workerandenduro-am-worker(enduro was already configured) and enables telemetry passing environment strings to the corresponding k8s workloads,database/sqlviagithub.com/XSAM/otelsql,With these changes, we'll see a significant increase in the number of traces captured by Tempo. We're not yet initiating a trace during the transfer submission via MinIO; however, an interesting example that can already be observed is the move workflow initiated through the API. If you've already moved a package, use this query to find related traces:
It should look like this:
Something that becomes apparent right away is the lag between activity executions - we know that's the cost of using Temporal to schedule activities. It also shows that local activities are much faster since they're not coordinated with the Temporal server.
You can also search for a workflow directly, e.g.:
This trace is interesting because it shows that the telemetry data originates from multiple services, specifically "enduro" and "enduro-a3m-worker", rather than a singular source.