Closed jmbowman closed 1 year ago
There's a spreadsheet for tracking how long it takes to get devstack up and running: https://docs.google.com/spreadsheets/d/1gfMhicX65xHPAI2GnDNPvAaezbypdJUDDE5ZtHK_EH8/edit#gid=1863116243
Taking a second look at GitHub metrics. I bet we could get something like first PR automatically.
This turns out to be way more difficult to do automatically than it should. Several attempts were made to automate something like "For the x most recent additions to the open/edx github organization, give me the date they joined the organization and the date their first PR was merged." It was reasonably easy to get the date of the first PR, but seemingly impossible to get the date someone joined the org. That data was not available in the GitHub Rest API, and the GraphQL api only provides access to public members, even if the token used represents a user who is part of the organization. The information is also not present in the community kibana dashboard. I really don't like the idea of relying on a Google sheet since it's always difficult to get people to add one more thing to the onboarding process, and keeping track of trends will be annoying, but I don't see any good alternatives for now.
One thing we could do is just have folks record their onboarding date in the spreadsheet and then get the information on first PR with a script that looks at the spreadsheet and GH API.
this survey has been created and disseminated. We also have a POC for some processing scripts and a grafana dashboard.
One suggestion for a metric that didn't make it into our earlier analysis is the time it takes a new developer to "finish onboarding". This could involve one or more metrics such as time from hire/transfer to:
We probably also want qualitative input on what was easy, hard, took longer than it should have, etc. There isn't a good way to automatically measure this, so we'd want to tell new developers that we want to measure this, write down the relevant duration(s) as the milestone(s) are achieved, and send them to us when done. And then store the information somewhere where we can analyze it and track it over time.
A/C: