[DPE-4614] Add support for tracing through tempo-k8s

[shayancanonical]

Issue

We would like to be able to instrument our charm and generate traces to discover inefficiencies in charm code execution. The observability team has developed a charm for tempo that we would like to integrate with.

Solution

Integrate with the tempo charm

Considerations

1. We are not supporting sending traces to tempo-k8s using HTTPS are there are still some rough edges to be rounded with this integration
2. We need to use the edge version of cos-lite until they release to stable as it contains some critical fixes

[shayancanonical]

• debugging with pdb would be a bit more complex with the decorator as there will be additional function calls on the stack than just the charm code -- the decorator overwrites how the charm is invoked, injecting code to instrument a function call before and after an event handler/extra type method is called. i believe you will still be able to set breakpoints and jump to them. i would suspect that stepping through code may be a bit more complex
• if there's a relation with tempo, then yes, there will be decreased resiliency when there's an error + there will be a performance penalty. however, tempo is not meant to be used in production -- it is meant to be used locally to determine why some hook may take a long time to run
• i see no issues if the charm code changes. if it changes to not have the decorator, we wont instrument. we are instrumenting units independently, so no cluster implications

[carlcsaposs-canonical]

tempo is not meant to be used in production -- it is meant to be used locally to determine why some hook may take a long time to run

when I discussed this with @PietroPasotti a few months ago, my understanding was that this was meant to be used in production—has that changed?

if it's not meant to be used in production, should we merge to main?

[shayancanonical]

charm tracing is not expected to be enabled in production as i understand it, but i still think that there is value in pulling a charm from one of our tracks in a non-prod env and instrument tracing. the tracing integration is an optional one, and does not add any issues if not related to tempo