Document tracing/honeycomb for team

[bloodearnest]

We need to document various bits of the work we've done

• [x] Trace semantics in job-runner. What the spans mean, how it works in the code.
• [x] General How To Honeycomb docs
• [x] Specific "use honeycomb to investigate a job" tek supprt dos
• [ ] Mermaid diagram of status code transitions in job-runner

[madwort]

some work here https://github.com/ebmdatalab/team-manual/pull/327

I think there's an overlapping pipeline team board card for this, that I can't link to (because it's not an issue attached to a repo...)

[madwort]

Dump of other ticket for merging:

• • Can we add a derived column to show time in minutes/hours?
  • [x] minutes
  • [ ] hours
• Can we see queue times?
• What questions do people on tech support typically want to answer?
  • (from Seb) can we compare cpu usage for python jobs vs R jobs -> no, not yet

Documentation

• Job runner docs about what we're recording - e.g an explanation of tracing
• Intro to honeycomb, in the team manual, linking to official resources, etc.
  • docs at: https://bennettinstitute-team-manual.pages.dev/tools-systems/honeycomb/
  • [x] document new job-runner traces

Related PRs

• adding honeycomb links from job-server
  • https://github.com/opensafely-core/job-server/pull/2186
  • https://github.com/opensafely-core/job-server/pull/2190
  • https://github.com/opensafely-core/job-server/pull/2193
• documenting
  • https://github.com/ebmdatalab/team-manual/pull/327

[madwort]

I think this PR https://github.com/ebmdatalab/team-manual/pull/327 addresses most of what we need for this ticket

[madwort]

@bloodearnest to create mermaid diagram of the status codes & transitions in job-runner

[madwort]

@bloodearnest to put the mermaid diagram in a PR for the job-runner repo, then this ticket is complete, see https://github.com/opensafely-core/job-runner/issues/486