Determine product & design goals for Github data

[joanneesteban]

User Story/Description

The VSP Analytics & Insights team has been working to retrieve Github data through BigQuery so that we can aggregate data to tell stories and provide insights on our customer service tickets. We have done some technical discovery into the data connection and believe we are able to get data through Google's public BigQuery dataset, given that the va.gov-team Github repo is public.

How might we understand if the data that is available can tell us if our customer support is successful and where we can find room for improvement? This ticket will be open to track those product and design questions so that we can determine if we have relevant data to answer those questions.

Tasks

• [ ] VSP PMs provide input for the product and design questions, particularly for customer support, they would like answered through Github data. Please add questions to comments to this ticket.
• [ ] OCTODE and VSP Leadership provide input to the types of product & design questions they would like answered through Github data. Please add questions to comments to this ticket.
• [ ] VSP Analytics synthesizes the questions into which metrics & datasets could answer those question
• [ ] VSP Analytics determines where there may be gaps in the datasets for the requested questions.

Questions to support thinking through the VSP Product Questions:

• Who is your audience?
• Why do you need this data?
• What are you hoping to find?
• What is your hypothesis that you're hoping the data will support?
• What area of VSP and Platform are you hoping to assess?
• How frequently do you need this data?

Acceptance Criteria

• [ ] VSP understands which product questions need to be answered by the Github data
• [ ] VSP Analytics understands where there may be gaps in the data for those product questions

[joanneesteban]

Testing Tools Team (Accessibility)

Outcomes to measure

• Adherence to Section 508 (accessibility) compliance. Are we getting better, staying the same, or need more improvement?
• Identify if VSP training, documentation, and support is improving VFS teams' adherence

GitHub API endpoints

• https://api.github.com/repos/department-of-veterans-affairs/va.gov-team/issues?labels=508-defect-0&state=open&per_page=100
• https://api.github.com/repos/department-of-veterans-affairs/va.gov-team/issues?labels=508-defect-1&state=open&per_page=100
• https://api.github.com/repos/department-of-veterans-affairs/va.gov-team/issues?labels=508-defect-2&state=open&per_page=100
• https://api.github.com/repos/department-of-veterans-affairs/va.gov-team/issues?labels=508-defect-3&state=open&per_page=100
• https://api.github.com/repos/department-of-veterans-affairs/va.gov-team/issues?labels=508-defect-4&state=open&per_page=100

KPIs

KPI	Filters
Count of open and closed accessibility issues All open and closed issues right now // Open and closed issues by team right now // All open and closed issues over time // Open and closed issues by team over time	Issues by: Label 508/Accessibility // Date Is Open // Date Is Closed
Count of accessibility issues by defect severity Severity 0-4 issues right now // Severity 0-4 issues over time	Issues by: Multiple labels (508-defect 0-4) // Counts of open issues by severity, stored and plotted over time
Count of accessibility issues by type Issues by type right now // Issues by type over time	Issues by: Multiple labels // Counts of open issues by type, stored and plotted over time
Average time to fix issues Average time to fix across all teams // Average time to fix for individual teams	Issues by: Sum of days between open and close divided by number of issues for time period // Same data with team labels applied
Early warnings Teams with number of issues 1-2 STDEV above the median average // Issues by type rising N% faster than historical average // Issues by defect severity rising N% faster than historical average // Issues open longer than N number of days	These would require more calculation but could be leading indicators of health, where the previous metrics are lagging indicators
Exploratory ideas JUnit output from CircleCI piped into dashboard or some notification // Filtered Sentry logs piped in as an early warning item	I have a theory that JUnit output such as we might be able to get from Axe-core or the Attest API and possibly Sentry logs could give us some near real-time warnings of issues

[joanneesteban]

Analytics & Insights Team (Customer Support)

Outcomes to measure

• To which analytics services can we add documentation and automation to improve the self-service and velocity of our services?
• Which services and which times of the year or quarters take up the most time and effort? Where can we improve efficiency?
• What percentage of VFS teams are getting quality analytics tracking? Which of those services are getting completed, and where are the efficiency gaps in those services?

KPIs

KPI	Filters
% of VFSs with quality GTM tracked	Issues by: Date is Open // Date is Closed // Closed Date - Open Date // Multiple Labels // Project Swim Lane // Project Swim Lane Date Change
% of VFSs with quality KPIs set & tracked	Issues by: Date is Open // Date is Closed // Closed Date - Open Date // Multiple Labels // Project Swim Lane // Project Swim Lane Date Change

[drorva]

Engineering teams

Outcomes to measure

• Measure the quality of support we're providing to VFS teams
• Measure the level of effort the engineering team is investing in support

KPIs Identical to the accessibility team's KPIs above with the addition of:

• Issues that are neglected: not answered in X (initially 3) number of days,
• Issues that are dormant: no activity in X (initially 15) number of days
• Average time to close an issue over time
• Issues that are closed using a link to documentation (identified by a label).

[RachaelMR]

Product

• The platform

Desired outcomes

• High performing VFS teams
• High performing VFSs

Potential KPIs we could draw from GitHub to help us understand the platform product's impact

• Customer support volume, by platform team, topic area, and time to close, and with VFS launches indicated along the timeline
• VFS production deployment rate (can this be measured through GH, and if so is that the best way? May need discovery)
• Volume of "standards" issues (open, closed, older than x days) by VFS team and topic area, and with VFS launches indicated along the timeline

(@shiragoodman cc'ing you for visibility into the desire around "standards" issue tracking)

---

• Who is your audience? (assuming "audience" means who's going to be looking at this data)
  • Platform crew leadership and teams, OCTO-DE, VA at large
• Why do you need this data?
  • To understand the impact of the platform product on the VA.gov ecosystem
• What are you hoping to find?
  • Opportunities for platform product improvement
  • Visibility into state of VFS standards, to help us drive OCTO-DE and VFS team behavior change
• What is your hypothesis that you're hoping the data will support?
  • As we release self-service tools, flexible deploys, and standards clarity, we'll see the quality and efficiency of VFS teams and their work improve
• How frequently do you need this data?
  • Open to recommendations here, maybe weekly? Daily?