[DOC] Add section for writing good TraceQL queries with content for instrumentation, query examples for finding traces with issues

[knylander-grafana]

Tempo users need to be able to find traces that are causing issues easily. First, they have to instrument their apps to make sure the data they need to use is available (are they producing the right tracing data and metrics?). Next, Tempo users need to be able to find that data by writing TraceQL queries that find the data they need.

• Best practices on instrumentation when tracing with Tempo
  • Check the Best Practices doc - https://grafana.com/docs/tempo/latest/operations/best-practices/
  • Related: https://github.com/grafana/tempo-squad/issues/297
• How to write good TraceQL queries
  • Reference: https://github.com/grafana/tempo-squad/issues/343
  • Related: https://github.com/grafana/tempo-squad/issues/413
• How to use TraceQL to find things like long running traces, big traces, traces with lots of short spans? or traces with single spans, etc.
• Teach people about span links when tracing background job, batch job or long running processes like ETL jobs
• Put up a doc with what we think user should look for when building queries and write a section with it. Example: use scopes when possible, watch out for complex regexes, OR conditions vs a regex on the same attribute, etc.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.