[Docs] Create end-to-end guide for OTEL with Tempo

[knylander-grafana]

@electron0zero noted that there are gaps in our documentation around instrumenting applications. Here's what we need to address how to configured the Grafana stack (agent, Tempo, libraries) based upon his comments:

• OTEL docs are not that great, new users have hard time instrumenting applications.
• Users don’t understand how to configure libraries, agent and tempo to ship traces successfully.
• There is a gap and we can have some guided docs, and that can help tempo become the default tracing backend for OTEL
• @electron0zero had hard time when instrumenting an internal application with OTEL.

Ideally, after these guides are done, we should see fewer support questions around instrumenting applications from the community.

Note: There are now OTEL docs hosted in grafana.com. Be aware of what belongs in Tempo and what should be handled by the OTEL team.

These guides should hold a user's hand end-to-end.

Tasks:

• Create a guide that starts with installation of OTEL and ends with visualizing traces using OTEL and the Grafana stack (Grafana Agent, Tempo, and Grafana)
• Make sure that basic concepts are explained (see https://github.com/grafana/tempo/issues/1606)
• We need info on Tempo received Open Telemetry info
• See [OTEL blog post from 2021'(https://grafana.com/blog/2021/04/13/how-to-send-traces-to-grafana-clouds-tempo-service-with-opentelemetry-collector/)

[knylander-grafana]

See also https://github.com/grafana/website/tree/opentelemetry

[GrafanaWriter]

@knylander-grafana - should this be owned by you? Or by the Otel team?

[knylander-grafana]

@GrafanaWriter I added it here since it was related to traces and Tempo. However, if you think it's better handled by the OTEL team, I'm happy to hand it over to them or work with them on it.

This is a needed document.

Today we received some feedback from self-serve support that a customer opted to use a competitor's solution since they had an end-to-end guide for using OTEL metrics and traces. The customer specifically asked for an end-to-end example showing the configuration required to have an application send OpenTelemetry data to a Grafana Agent and have Grafana Agent forward that data (traces, metrics) to Grafana Cloud.

[knylander-grafana]

@electron0zero We have a possible doc that addresses some of these topics. See this PR: https://github.com/grafana/opentelemetry-docs/pull/20

What do you think?

[electron0zero]

Yes, but this doc only covers the steps to configure the collector and send tarces to our databses, so this is only one step of the whole end to end guide.

I would like to have a guide where we showcase:

1. instrumention
2. setting up collecter (this doc touches on it, but only for kubernetes users using helm)
3. configuring collector (this doc covers this step nicely)
4. sending data and querying it in Grafana.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[knylander-grafana]

This blog post by Marty has some info on how to set up the OTEL collector: https://grafana.com/blog/2021/04/13/how-to-send-traces-to-grafana-clouds-tempo-service-with-opentelemetry-collector/

[knylander-grafana]

@peterolivo wrote a new Cloud doc that covers some of OTEL content: Ship Kubernetes OTEL traces using Grafana Agent

[knylander-grafana]

See also https://github.com/grafana/tempo-squad/issues/243

[GrafanaWriter]

@knylander-grafana - I'm triaging unassigned items in our backlog - given your work here, should you assign yourself?

[knylander-grafana]

@GrafanaWriter There may be cases where someone from the community may wish to write some doc for Tempo. I keep the doc issues updated as I can, but that doesn't mean I'll be the one writing them. I assign a doc issue to myself when I actually start on it.

Chances are I will be the one to write this one with some help from the field engineering team.