Measure onboarding time

[jmbowman]

One suggestion for a metric that didn't make it into our earlier analysis is the time it takes a new developer to "finish onboarding". This could involve one or more metrics such as time from hire/transfer to:

• Completion of dev environment setup
• First PR creation
• First merged PR
• etc.

We probably also want qualitative input on what was easy, hard, took longer than it should have, etc. There isn't a good way to automatically measure this, so we'd want to tell new developers that we want to measure this, write down the relevant duration(s) as the milestone(s) are achieved, and send them to us when done. And then store the information somewhere where we can analyze it and track it over time.

A/C:

• Add at least one metric for this to the Developer Experience Metrics spreadsheet and fill out the fields accordingly
• Write up instructions for new developers explaining what we want to measure and how to submit the measurements to us
• Set up something to store the measurements (Google Sheet for now?)
• Add the data request to the standard onboarding process
• Try to collect data from people who recently finished onboarding (or are still in the middle of it). We probably also want to set up brief interviews with these people to get feedback on how the process went.

[dianakhuang]

There's a spreadsheet for tracking how long it takes to get devstack up and running: https://docs.google.com/spreadsheets/d/1gfMhicX65xHPAI2GnDNPvAaezbypdJUDDE5ZtHK_EH8/edit#gid=1863116243

[rgraber]

Taking a second look at GitHub metrics. I bet we could get something like first PR automatically.

[rgraber]

This turns out to be way more difficult to do automatically than it should. Several attempts were made to automate something like "For the x most recent additions to the open/edx github organization, give me the date they joined the organization and the date their first PR was merged." It was reasonably easy to get the date of the first PR, but seemingly impossible to get the date someone joined the org. That data was not available in the GitHub Rest API, and the GraphQL api only provides access to public members, even if the token used represents a user who is part of the organization. The information is also not present in the community kibana dashboard. I really don't like the idea of relying on a Google sheet since it's always difficult to get people to add one more thing to the onboarding process, and keeping track of trends will be annoying, but I don't see any good alternatives for now.

[rgraber]

One thing we could do is just have folks record their onboarding date in the spreadsheet and then get the information on first PR with a script that looks at the spreadsheet and GH API.

[rgraber]

this survey has been created and disseminated. We also have a POC for some processing scripts and a grafana dashboard.