⭐ Q3B1 Mitigate Observability expenses

jhchabran commented 2 years ago

This was brought to us by @bobheadxi over this Slack message.

Problem

Dotcom is seeing continuous growth in traces export that is amounting to significant expenses ($22k in Grafana this month) as well. @bobheadxi worked on a few mitigation efforts, but I think we need a more concerted strategy here about how we are going to handle tracing in the application (see below) and investigate why this is happening and how we can keep costs under control.

We are missing guidance on how to write good tracing and potential misuse of tracing policy due to how tracing policies have been implemented in Sourcegraph for a long time. I suspect a significant issue we are seeing is that background jobs always enable tracing (https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@af1c9d3a8ef24cc35f3b248b66b0e89dfed46258/-/blob/internal/workerutil/worker.go?L315) because there's no concept of "selective" for background jobs yet, which causes libraries like otelsql to emit large amounts of traces for every background job that gets run. @jhchabran and @burmudar did some work previously on options for otel-level sampling - IIRC (@bobheadxi) one of the conclusions is because of selective we can't really reliably suggest generalized probabilistic sampling, so we need a solution designed and implemented here (whether that be more feature flags, research into more filtering/sampling, etc).

Scope

We're not looking to refactor entirely the tracing API nor to make the selectiveness property work perfectly across all the components, as this would require to migrate all the existing opentracing code to OTEL.

Instead, by order of importance:

Mitigate the spending by fixing the biggest source of traces that do not respect the selective property.
Provide a clear path to write tracing code that respects the type of tracing.
Sketch a plan for taking up this further.

Boundaries

Do not migrate open tracing code in OTEL (unless we're 100% sure it's mandatory to reach 1., and if it's the case, we need to revisit this ticket).
Do not implement solely on our own the changes in the code, reach out for their respective owners for help. We have a lot on our place for this bet cycle, and we need to leverage any help we can get.

Approach

@jhchabran and @burmudar to review all the existing PRs submitted by @bobheadxi
@jhchabran and @burmudar to scrutinize the code to get better understanding from where those traces are coming from.
@jhchabran, @burmudar, and @bobheadxi to hold a meeting to chat about the approach.
Work