Understand how we might count people using Codespaces

[StevenMaude]

We might want to count how long people are using Codespaces for. When/if people migrate to local development. Can we associate a commit with a dev environment?

Ideas

• Can we use the GitHub API? Should we add something to OpenSAFELY CLI?
  • Note: if we can use the GitHub API or something automated we can add a graph into Grafana.
• The organization audit log can filter for events involving Codespaces. We may be able to pull that data out to get an idea of the number of users.
  • I don't believe there's API access, but it is manually exportable — click a button in Settings for the organization — into JSON/CSV.
• GitHub adds some environment variables when running in Codespaces; these might be useful.
• Should we just do a Slack poll instead?

[lucyb]

For the purposes of testing what's available via the GitHub API, the following command may be useful: gh codespace list --org opensafely.

I believe this is equivalent to:

gh api \                          
  -H "Accept: application/vnd.github+json" \  
  -H "X-GitHub-Api-Version: 2022-11-28" \
  /orgs/opensafely/codespaces  

[Jongmassey]

GitHub adds some environment variables when running in Codespaces; these might be useful.

These could be useful if we add telemetry to opensafely-cli - there's bits in there in the vendored job runner but AFAICT it's not being used currently. This could also provide us with a "local run" denominator but raises possible issues around user consent to this telemetry and gracefully handling when a user is not connected to the internet.

For the purposes of testing what's available via the GitHub API, the following command may be useful: gh codespace list --org opensafely.

This appears to list the currently running codespaces, from the docs I can't see anything about historic codespace usage. We could poll this on a semi-regular basis to track it over time.

Should we just do a Slack poll instead?

This is my instinctive preference, but might suffer from low uptake

[iaindillingham]

If we wish to count how many Codespaces were started, then we could use one of the lifecycle scripts to communicate with an endpoint. If we sent both start and stop timestamps, then we could determine each Codespace's duration.

The issue title mentions counting people. We could pass values of one or more default environment variables to an endpoint, too. Doing so doesn't seem dramatically different to capturing other telemetry data.

[Jongmassey]

I've been poking at this for a couple of days and here are some of the harder edges I've come up against:

• we can easily add code to the existing postCreateCommand
• we want to record this creation somewhere useful, ideally where it can go into a dashboard
• Grafana is fed from a read-only copy of the Job Server database and from GitHub via the API and custom metrics code
• Therefore the telemetry code needs to hit either:
  • GitHub (the API? Add an issue/issue comment in the codespaces-initiative repo?)
  • Job Server (we already have APIs there for other purposes, we could add another one and a corresponding table in the DB) - this feels out of Job Server's scope.
  • A new API on the metrics dokku container (FastAPI or similar?)
  • somewhere else?
• Having any of these be authenticated feels like a bad idea. Even with careful design this feels like it's an opening for a DoS attack, but that might be OTT worrying. There's probably some secrets management fun ahead if we want to avoid distributing the required credentials in the devcontainer config itself.
• Any changes we do make to the research template will need to be pushed out to any repo that is derived from it

[lucyb]

For reference, there's a slack thread with some additional discussion.

I'm going to make a decision here and say that we poll the GitHub API every hour. This might not give us particularly detailed information, but it might tell us what additional information we do need and will be quick to do. Also, it will tell us if people are using Codespaces at all... if they're not, then there's no point in spending extra time gathering additional metrics.