Switch to otlp/http protocol in charm_tracing

[mmkay]

Issue

Inclusion of charm_tracing library caused a dependency on GRPC that, in various contexts, added between 10MB to 100+MB to the charm size using this library (ref: canonical/prometheus-k8s-operator#545 ). We considered other protocols to send traces and decided that OTLP/HTTP seems to be the most relevant in the current context.

Resolves #53.

Solution

Replacement of opentelemetry-exporter-otlp-proto-grpc with opentelemetry-exporter-otlp-proto-http together with bumping up to the latest (1.21.0) version.

The main difference between exporters is that instead of certificate being passed as one of the parameters of OTLPSpanExporter, HTTP exporter asks for a path to a certificate file. That means that the library upgrade for charms that are using a certificate (traefik-k8s-operator) requires changing the annotation signature and providing certificate_file instead of the current certificate parameter.

Context

One interesting side effect of the version upgrade is that _get_tracer() couldn't find the context var for tracer in its context when an external function was instrumented using @trace. There was another context var reference with the same name pointing at the same tracer object in its context, though, so I added a fallback trace retrieval.

Testing Instructions

Integration tests set up tempo together with a tester charm that sends traces and confirms they were ingested correctly.

Release Notes

Replacement of OTLP/GRPC protocol with OTLP/HTTP in traces export.

[mmkay]

@PietroPasotti

I'm assuming you tried it out manually to see if it works?

Correct - manual reproduction steps, assuming tempo is mounted on the multipass host under ~/tempo-k8s-operator and libraries are copied to tester charm:

juju add-model cos
juju switch cos
juju deploy cos-lite --trust
cd tempo-k8s-operator
charmcraft pack 
charmcraft pack -p ./tests/integration/tester/
juju deploy ./tempo-k8s_ubuntu-22.04-amd64.charm tempo --resource tempo-image=grafana/tempo:1.5.0
juju deploy ./tester_ubuntu-22.04-amd64.charm tester -n 3 --resource workload=python:slim-buster
juju relate tempo:tracing tester:tracing
juju relate grafana:grafana-source tempo:grafana-source

You should be able to see traces from TempoTesterCharm in Grafana: