QA Dashboard - Create a Product View

JoeTice commented 2 years ago

Problem Statement

Currently, the QA Dashboard provides a lot of information from testing tools across the platform, but it is very difficult for VFS team members, or for OCTO-DE leadership to determine how a specific product is performing across the different tools and methodologies due to the facts that - one product can be referred to in a different way in different testing tools making it tough to correlate products 1-1, and not all products are using all of the recommended testing tools or processes, so some may not be listed in certain areas altogether - increasing confusion.

How might we update the QA Dashboard to provide a view that will more clearly illustrate how a specific product is performing across our QA practices, therefore providing better insight for teams to identify issues and resolve them?

Hypothesis or Bet

By creating a dashboard where VFS team members and OCTO-DE leadership can view results of QA efforts of their product across our platform testing tools, we will provide VFS teams with a better tool set with which to assess how their product is performing and identify areas of improvement

We will know we're done when... ("Definition of Done")

Domo contains single-product dashboard visualizing quality assurance practice assessment for one or more products hosted on VA.gov
Domo contains a template for a single-product dashboard that can be accessed through Domo's API to create a new dashboard

Known Blockers/Dependencies

Some metrics may not be able to align directly to a product in their current state, these issues will be determined during the research/discovery portion of the initiative

Projected Launch Date

3/15/22 (End of Sprint 73)

Launch Checklist

Is this service / tool / feature...

... tested?

[ ] Usability test (TODO: link) has been performed, to validate that new changes enable users to do what was intended and that these changes don't worsen quality elsewhere. If usability test isn't relevant for this change, document the reason for skipping it.
- [ ] ... and issues discovered in usability testing have been addressed.
- Note on skipping: metrics that show the impact of before/after can be a substitute for usability testing.
[x] End-to-end manual QA or UAT is complete, to validate there are no high-severity issues before launching

... documented?

[x] QA Dashboard documentation is updated to reflect the Product View
[x] Product is included in the List of VSP Products
- List the existing product that this initiative fits within, or add a new product to this list.
[x] Internal-facing: there's a Product Outline

... measurable

[x] Metrics have been defined and are being tracked through dashboard
[x] Usage is being tracked

When you're ready to launch...

[x] Conduct a [go/no-go] (https://vfs.atlassian.net/wiki/spaces/AP/pages/1670938648/Platform+Crew+Office+Hours#Go%2FNo-Go) when you're almost ready to launch. Results from the 3/8/22 Go/No-Go are recorded here.
- [x] Nail down communications
  - [x] Add into help that if there are feature request to go to the feature request process/mechanism
  - [x] Add more clarity on how to get help (who do they contact in platform-support)
- [x] Roll out plan changes – make sure you’ve thought through it. Include announcement that QA standards will be following over the next week or two.
  - [x] Craft team messages
  - [x] Craft Slack announcement
  - [x] Craft Team of Team announcement

Required Artifacts

Documentation

PRODUCT_NAME: directory name used for your product documentation
Product Outline: Click here for Product Outline
User Guide: Click here for User Guide

Testing

Usability test: The small scale launch to 10 teams with a feedback survey built into the tool we hope will provide good feedback on the usability
Manual QA: None
Automated tests: N/A

Measurement

Success metrics:
- Tied to Team Objective for Q1 - Q1 2022 Objective 2: Launch QA Standards for VFS Products Usage/Engagement Statistics are tracked here (Requires Domo Access)
- KR 1: > 20 unique QA Dashboard users on average per sprint
- KR 2: > 8 of VA.gov products are getting weekly views
- KR 3: > 10 VFS Products have a QA Product View
Release plan: Release Plan

pjhill commented 2 years ago

Please add your planning poker estimate with ZenHub @JoeTice

JoeTice commented 2 years ago

Based on the results of the Go/No Go - we've revised our tracking mechanisms, we reached out to Analytics to communicate and coordinate our efforts with theirs to keep the experience consistent for users.

We are also waiting for a bug with TestRail's ingestionto be fixed, but don't have an ETA on that.

We will allow a couple of days to resolve these issues, otherwise will move forward

CBonade commented 2 years ago

The documentation is updated with all the proper procedures and will go out with the next "go live" of the documentation (currently scheduled for tomorrow).

I've drafted a couple of messages for slack and teams here:

Slack/Team of Teams:

Hello,VFS Teams! We are pleased to announce the initial launch of our Product QA Dashboards. The purpose of these dashboards is to help track the various QA status of products, to have a one-stop destination to gauge the state of a product. Our goal of the initial launch is to use these first 10 products to gather feedback. We will then be applying that feedback throughout the existing 10 dashboards, with a goal to have a dashboard for every product on va.gov. Extensive documentation is available for these dashboards to help you with how to use the dashboard as well as things such as requesting a new feature or needing support:

https://depo-platform-documentation.scrollhelp.site/developer-docs/QA-Dashboard-Documentation.2013200394.html

Additionally, we have created a survey to gauge your initial feedback as quickly as possible so we may make this dashboard as useful as it can be. Even if you do not have a dashboard for your product, we welcome your feedback of what you might like to see before we expand this out to your product. We would appreciate a few minutes of your time to fill it out:

https://adhoc.optimalworkshop.com/questions/68t04235

The survey will eventually close, and at that point there is an issue template that can be used to file a feature request addition (see docs above).

Thanks much to everyone, and we look forward to hearing from all of you!

Team:

Hello, [team name here]! We are pleased to announce that your product [product name] was selected to be part of our initial launch of our Product QA Dashboards. The purpose of this dashboard is to help you track the various QA status of your product, see where you are excelling and where there may be areas that can be made even better. Extensive documentation is available for these dashboards to help you with how to use the dashboard as well as things such as requesting a new feature or needing support:

https://depo-platform-documentation.scrollhelp.site/developer-docs/QA-Dashboard-Documentation.2013200394.html

Additionally, we have created a survey to gauge your initial feedback as quickly as possible so we may make this dashboard as useful as it can be. We would appreciate a few minutes of your time to fill it out:

https://adhoc.optimalworkshop.com/questions/68t04235

The survey will eventually close, and at that point there is an issue template that can be used to file a feature request addition (see docs above).

Thanks much to everyone, and we look forward to hearing from all of you!

JoeTice commented 2 years ago

This initiative's release is blocked, due to a TestRail issue - moving to Product Backlog until resolved.

department-of-veterans-affairs / va.gov-team