VAOS edge case dashboard - Design QA

[michelle-dooley]

Issue Description

As we roll out an VAOS edge case dashboard, we need to make sure that the dashboard is easy to use and understand.

---

Resources

• VAOS draft dashboard: https://va-gov.domo.com/page/202515362

• Past example tickets – recommend reading through comments:

  • Design QA for Forms KPIs
  • Design QA for Search KPIs
  • Design QA for Content KPIs
  • Design QA for All-VA Dashboard

Tasks

Use the Participant Tester account in Domo to QA for the following:

• [ ] All links should be clickable and correct
• [ ] All tooltips are consistently formatted and accurately describe their metric
• [ ] Directionality of change arrow/colors is correct for the metric
• [ ] All titles and text fully visible in the dashboard layout
• [ ] Filters should work properly
• [ ] Any feedback is documented

Acceptance Criteria

• [ ] The VAOS dashboard in Domo is usable and follows the design as closely as possible.

[michelle-dooley]

Usability checklist

Complete using Participant Tester account – please copy and paste into your own comments below to track QA progress. See example table below or refer to past QA tickets linked above.

Example

Test	Expected Behavior	Actual Behavior	Pass/Fail
Ex: Date filter	Every chart gets filtered for date and group by settings are changed	"Total logged in users" does not respond to filter changes	Fail ❌
Ex: Date filter	Every chart gets filtered for date and group by settings are changed	Every chart gets filtered for date and group by settings are changed	Pass ✅

Template

Test	Expected Behavior	Actual Behavior	Pass/Fail
Dashboard title	Text fully readable, without errors
Date filter	Every chart gets filtered for date and group by settings are changed
Section titles	Every section title & subtitle are spelled correctly
Chart titles	Every chart title is fully readable, without errors, relevant to chart
Chart descriptions	All chart descriptions are disabled, so nothing appears when hovering over title
Hover text	All hover text "makes sense" and is spelled correctly
Annotations	Every annotation "makes sense," is free of errors, and visible within text box
Text box formatting	Every annotation and section title box shows all text, does not generate scroll bar
Card interactions – filters	Clicking data point does not filter other cards
Card interactions – card detail	Clicking chart title opens card in new tab
Display settings – multi-value gauges	Date grouping ("by month") appears under each title
Display settings – summary numbers	All tables, bar charts, and filled gauges show date range
Related cards	In card detail view, all related card previews have been deleted
Appendix	No charts or notecards appear in appendix below dashboard

[michelle-dooley]

@jonathan-epstein13 and @pavanhothi for design QA for VAOS when ready. cc: @chloedotbrown

[jonathan-epstein13]

Design QA

VAOS Dashboard Test	Expected Behavior	Actual Behavior	Pass/Fail
Dashboard title	Text fully readable, without errors	Text fully readable, without errors	Pass ✅
Date filter	Every chart gets filtered for date and group by settings are changed	Every chart gets filtered for date and group by settings are changed	Pass ✅
Dashboard filter	Every card gets filtered by the dashboard level filter(s)	Landing page section isn't affected by the 'form name' dashboard level filter	Fail ❌
Section titles	Every section title & subtitle are spelled correctly	Every section title & subtitle are spelled correctly	Pass ✅
Chart titles	Every chart title is fully readable, without errors, relevant to chart	Every chart title is fully readable, without errors, relevant to chart	Pass ✅
Chart descriptions	All chart descriptions are disabled, so nothing appears when hovering over title (except title, but no different text)	All chart descriptions are disabled, so nothing appears when hovering over title (except title, but no different text)	Pass ✅
Hover text	All hover text "makes sense" and is spelled correctly	All hover text "makes sense" and is spelled correctly	Pass ✅
Annotations	Every annotation "makes sense," is free of errors, and visible within text box	Avg. minutes on page - positive/negative color is the blue shade (5492FF), Completion rate of step & Exit rate by step - steps aren't all aggregated. ie "multiple" 'Submission step'	Fail ❌
Text box formatting	Every annotation and section title box shows all text, does not generate scroll bar	Every annotation and section title box shows all text, does not generate scroll bar	Pass ✅
Card interactions – filters	Clicking data point does not filter other cards	Clicking data point does not filter other cards	Pass ✅
Card interactions – card detail	Clicking chart title opens card in new tab	Clicking chart title opens card in new tab	Pass ✅
Display settings – multi-value gauges	Date grouping ("by month") appears under each title	Date grouping ("by month") appears under each title	Pass ✅
Display settings – summary numbers	All tables, bar charts, and filled gauges show date range	All tables, bar charts, and filled gauges show date range	Pass ✅
Related cards	In card detail view, all related card previews have been deleted	Conversion rate by user' has a linked card.	Fail ❌
Appendix	No charts or notecards appear in appendix below dashboard	No charts or notecards appear in appendix below dashboard	Pass ✅

@pavanhothi if you would like to be a second pair of eyes on the Design QA as well, that would be great! I'd appreciate it.

[chloedotbrown]

Thanks for the QA, @jonathan-epstein13 ! Reproducing a failure-only version of QA table below with notes beneath:

Test	Expected Behavior	Actual Behavior	Pass/Fail
Dashboard filter	Every card gets filtered by the dashboard level filter(s)	Landing page section isn't affected by the 'form name' dashboard level filter	Fail ❌
Annotations	Every annotation "makes sense," is free of errors, and visible within text box	Avg. minutes on page - positive/negative color is the blue shade (5492FF), Completion rate of step & Exit rate by step - steps aren't all aggregated. ie "multiple" 'Submission step'	Fail ❌
Related cards	In card detail view, all related card previews have been deleted	Conversion rate by user' has a linked card.	Fail ❌

• Dashboard filter: This is fine - all VAOS pathways share the same landing page, so cards in this section aren't meant to respond to the pathway filter.
• Annotations:
  • Avg. min on page colors: Because an increase/decrease to this metric is pretty context-dependent, we keep the colors for the percent change a neutral blue for both directions. Same thing is done for equivalent card in the landing page section of Forms dashboard.
  • Step tables: Hmm, I'm not seeing an aggregation issue here. One thing that might be a cause for confusion is that each VAOS product should end with Submission and Successful Submission as the final two funnel steps - these are actually two separate steps. If that was the cause of concern, then we're good! But if you were seeing something else, can you send a screenshot? Happy to troubleshoot.
• Related cards: Good catch! Deleted relationship. ✅

[jonathan-epstein13]

When filtering through form_name, only Vaccine shows duplicates of Start Vaccine Flow When not filtering for form_name, as you can see below, the step_name shows duplicates for some, that's where I thought there would be aggregation. Let me know what you think. I appreciate the clarity on the Design QA! @chloedotbrown

[chloedotbrown]

Thanks so much for the screenshots, @jonathan-epstein13 ! Really helpful to see what you're looking at. If there's anything else you see beyond the two above, please send them along too!

Vaccine steps

Looks like the error is in the underlying forms_vaos table in BigQuery. This path should start with the 2nd "Start Vaccine Flow" step - I'll ping @obrienjulie in Slack so she can fix.

Non-filtered steps

Similar to our standardized dashboards, folks aren't supposed to ever look at the VAOS dash not filtered by one of the products - so this shouldn't be something they every see. But we can still unpack why you're seeing what you're seeing! Although it's not a column visible on the table, this card is sorted by the step_number field. And while many of these flows have similarly named steps, they often fall at different orders in the flow. Because these step_numbers are different, they're showing up as different rows on the unfiltered table.

Here's a screenshot of a version of the card where step_number is shown as a column, to make it easier to see. If you have any other questions, let me know!

[chloedotbrown]

Hi @jonathan-epstein13! The three user survey cards on the VAOS dashboard are ready for your design QA:

• Avg. experience rating
• Distribution of experience ratings
• Able to complete task

All cards in this section do not filter by product, just like the landing page section above (I've added small notes on their headings). Each has a drillpath chart configured - I've included % of total in the tooltip for each of these, though it specifies "% of total filtered", since users will need to remove the filter automatically applied by the drillpath in order for it to show anything other than 100%. There's documentation on this in our our existing guides, but we should still plan on calling it when we deliver this to the team.

Anyway! Let me know if you have any questions or find anything out of the ordinary. Also, reminder to check off column on tracking spreadsheet so we know what we've covered already.

[jonathan-epstein13]

Thank you @chloedotbrown! I appreciate the clarity of the drill paths and will look at the 'how to' guide for the documentation part, if needed. I'll post my QA results here to the card. Will check off all cards on the spreadsheet as well.

[jonathan-epstein13]

User satisfaction section QA

Test Cards	Expected Behavior	Actual Behavior	Pass/Fail
Avg. experience rating	All behavior from the full QA matches to the card	All behavior from the full QA matches to the card	Pass ✅
Distribution of experience ratings	All behavior from the full QA matches to the card	All behavior from the full QA matches to the card	Pass ✅
Able to complete task	All behavior from the full QA matches to the card	All behavior from the full QA matches to the card	Pass ✅

@chloedotbrown I did look through all the drill paths, and was trying to understand the part % of total in the tooltip for each of these . Was this the part (for example) in the Distribution of experience ratings card -->in hover text(%_PERCENT_OF_TOTAL of ratings). Just wanted to make sure I understood completely. All the cards look good!

[chloedotbrown]

Awesome, thanks so much for the speedy QA @jonathan-epstein13! Glad these were an easy pass.

And apologies if the note about the hover text caused more confusion than it solved! This was only meant to apply for the drillpath charts for "Distribution of experience ratings" and "Able to complete task" cards. For both, when you access the drillpath by clicking either response type, it filters for only that response – meaning the default for the %_PERCENT_OF_CATEGORY of total filtered in the tooltip initially reads as 100%. If folks remove the filter to see the distribution of answers over time, this updates to show the percent of the category within that month, which should be more useful! (screenshots below)

I believe we demo removing filters in our How-To guide and videos, and I'm also hoping that the "of total filtered" language will help clarify. But maybe we also note it when we pass this back to the VAOS team and walk them through it.