QA Content KPI Dashboards in Domo - Content/Design

joanneesteban commented 3 years ago

Issue Description

As we start to roll out our dashboards, we need to make sure that our dashboards are usable.

Tasks

QA for the following

[ ] Design
[ ] Content Other
[ ] All links should be clickable and correct
[ ] Filters should work properly
[ ] Document feedback

Acceptance Criteria

[ ] The production Content KPI Dashboard in Domo is usable and follows the design as close as possible

joanneesteban commented 3 years ago

@JasonTheBearded @chloedotbrown

Usability Checklist 07-09-2021

Completed using the participant tester.

<!DOCTYPE html>

Test	Expected Behavior	Actual Behavior	Pass/Fail
Default page	Page does not automatically filter, and description reflects lack of filter	Page does not automatically filter, and description reflects lack of filter	Pass ✅
Filter on Products	Every chart gets filtered for every product	Every chart gets filtered for every product	Pass ✅
Filter on Product URLs	Every chart gets filtered for filtered product URLs	Every chart gets filtered for filtered product URLs	Pass ✅ Is "translate.googleusercontent.com/translate_c" supposed to be a part of VA Medical Center Sites? One of the filters is "www.iowacity.va.gov/careers/physchintern/index.asp https://www.va.gov/pittsburgh-health-care/work-with-us/internships-fellowships/psychology-internships-and-fellowships"
Filter on Device Category	Every chart gets filtered on device category	Every chart gets filtered on device category	Pass ✅
Filter on Browser	Every chart gets filtered on browser	Every chart gets filtered on browse	Pass ✅
Date Filters	Every chart gets filtered on date	Every chart gets filtered on date	Pass ✅
How to use this dashboard - header	Links to Platform Website, "How to use this dashboard" page	Links to Platform Website, "How to use this dashboard" page	Pass ✅
How to use this dashboard - footer	Links to Platform Website, "How to use this dashboard" page	Links to Platform Website, "How to use this dashboard" page	Pass ✅
Chart Title	Corresponding filtered form appears as the page title	The title isn't consistent with the Forms KPI Dashboard. The Content dashboard starts with "Product," where the Forms KPI Dashboard just calls out the Product Name.	Fail ❌
Product description	Corresponding filtered product description appears with every filtered form	VA Medical Center Sites' description does not appear	Fail ❌
Data sources	Correct data sources appear	Correct data sources appear	Pass ✅
Section titles	Every section title & question is spelled correctly	In the Task completion Anchor links section, "How often did users interact with the aciton links?" This is supposed to be "anchor links" instead of "action links."	Fail ❌
Chart titles	Every chart title is spelled correctly	Every chart title is spelled correctly	Pass ✅
Annotations	Every annotation "makes sense" and is spelled correctly	- "Foresee" in Highlights section and User surveys section should be "ForeSee" - In Links section, "average links clicked per session on across all VA.gov content pages using this dashboard." - "on" should be removed. "Using this dashboard" doesn't make sense either. - In Links section, "Link engagement rate = visitors that clicked or tapped / total visitors" - what does tapped mean? Is it the total number of visitors who clicked or tapped a link? - In Buttons & action links section, "average buttons or action links clicked per session across all VA.gov content pages using this dashboard." - "using this dashboard" doesn't make sense. - In Anchor links section, "average anchor links clicked per session across all VA.gov content pages using this dashboard" - "using this dashboard" doesn't make sense. - In Anchor links section, "Anchor link engagement rate = visitors that clicked or tapped / total visitors" - what does tapped mean? Is it the total number of visitors who clicked or tapped an anchor link? - In Accordions section, "average accordions opened per session across all VA.gov content pages using this dashboard" - "using this dashboard" doesn't make sense. - In Accordions section, "Accordion opened rate = visitors who opened an accordion / total visitors" - what does tapped mean? Is it the total number of unique visitors who opened an accordion? "who" and "that" are not used consistently between the sections - In Interaction timing section, "Scroll depth funnel = how many visitors scrolled to each quarter fo the page / total visitors" - "fo" should be "of" - In the User surveys section, "(Add all VA comparison)" looks like it's incomplete. Can this be changed to "Waiting on Medallia for All VA comparison"?	Fail ❌
Labels	All data labels appear where expected.	- In the Buttons & action links section for the Translated Content product, "Buttons with highest engagement" - the top button is blank. The hover over is also blank. - In the Buttons & action links section for the VA Medical Center Sites product, "Buttons with highest engagement" - the 3rd to last data point label is blank. The hover over is also blank. - The up & down arrows are colored red or green, but many of these colors don't correspond with up or down.	Fail ❌
Hover text	All hover text "makes sense" and is spelled correctly	All hover text "makes sense" and is spelled correctly	Pass ✅
VA.gov Analytics Playbook link	Links to Github Analytics Playbook	Links to Github Analytics Playbook	Pass ✅
Google Analytics link	Links to Google Analytics 360	Links to Google Analytics "WBC- VA.gov Modernized Content" View home page	Pass ✅
VA.gov Data Dictionary link - header	Links to Platform Website, "Content KPIs Data Dictionary" page	Links to Confluence	Fail ❌
VA.gov Data Dictionary link - footer	Links to Platform Website, "Content KPIs Data Dictionary" page	Links to Confluence	Fail ❌
Domo Help Center link	Links to Domo Help Center	Links to Domo Help Center	Pass ✅
Export to PPT	Entire dashboard exports to PowerPoint without chart cutoffs on the slides or missing labels	All section headers end up getting cut off from the charts within them	Fail ❌
Export to PDF	Entire dashboard exports to PDF without chart cutoffs on the slides or missing labels	All section headers end up getting cut off from the charts within them	Fail ❌
Color accessibility	Any charts with differing colors has distinguishable values using a color blindness tester.	Any charts with differing colors has distinguishable values using a color blindness tester.	Pass ✅

chloedotbrown commented 3 years ago

🤔 Questions

Chart/Dashboard Title – Think this is a question for us about what seems most user-friendly and most uniform across all 3 dashboards? Currently, Search uses the prefix Search product: XYZ, Content is just Product: XYZ and Forms has no prefix (though each form product name does start with Form 123 -). Should we delete all prefixes? Match the Search one to Content? Open to anything, so let me know what seems clearest.
Labels – Final bullet mentions that red/green colors don't match what's expected. Which color a direction is coded should depend on how we hope the metric moves. I did a quick for the multi-value gauges, and they all looked alright, but send me the ones that look like they need to be flipped and I'll reverse them. Or was this about one of the drill paths?
Units of time (outside scope of initial testing) – Does looking at decimal places of minutes (< 1) seem weird? Should we change these units to seconds?

✅ Corrected

Section titles – Subheader reads "anchor links" instead of "action links."
Annotations
- Corrected capitalization in all instances of "ForeSee."
- Removed extraneous "on" from all-VA comparison of avg. links per session.
- Removed "or tapped" from all 4 user engagement rate definitions for clarity.
- Standardized user engagement rate definitions to use "who" > "that."
- Removed "... using this dashboard" from All-VA comparison cards.
- Corrected typo in scroll depth funnel definition.
- Language in User Survey section updated to clarify we're waiting on data rather than dashboard being unfinished.
Labels
- Blank labels in "Buttons with highest engagement" chart part of larger potential data issue. Have documented and asked Brian for support in Data QA ticket.

📥 Needs data/URL

Product description – Need this info from product teams, then will need to be added to Domo datasource.
VA.gov Data Dictionary link - header – Can't be updated until published on site right before.
VA.gov Data Dictionary link - footer – Can't be updated until published on site right before.

🔧 Domo limitations w/ workarounds

Export to PPT – Domo does not provide flexibility to fix or customize. How-To documentation instructs users to either take screenshots of portions of dashboard or export to other formats at card-level.
Export to PDF – (see above)

joanneesteban commented 3 years ago

@JasonTheBearded @chloedotbrown

A couple of old/new tweaks needed, including the Data Dictionary in the Platform Website.

Usability Test 07-19-2021

Completed using the participant tester:

chloedotbrown commented 3 years ago

Misc

This was me! It's now been removed, but it did unfortunately prove that the issue persists across an independently created card. As Jason suggested on another ticket, I'll reach out to Domo Doug this afternoon. I think we need backup, because this should just not be an issue.

Product description

I hadn't realized this field was added to the dataset, but now – it is also added to the card! One funky thing we will have to deal with here is that the VAMC description is very long and Domo won't wrap the text of summary numbers (which is how we're getting away with dynamic formatting).

Screen Shot 2021-07-19 at 2 53 41 PM

Misc

Good to know! We might need to put some sort of annotation flag on that chart if the issue persists by "launch." I'm going to assume that Domo tech won't have it sorted out by then 😅

Product Description

I checked in with Michelle, and your suggested edit is good to use!

Suggested edit: Pages for Veterans to learn about VAMC facilities and services and manage their healthcare online

Usability Test 07-20-2021

Completed using the participant tester:

Test	Expected Behavior	Actual Behavior	Pass/Fail
Filter on Product URLs	Every chart gets filtered for filtered product URLs	Every chart gets filtered for filtered product URLs	Fail ❌ Is "translate.googleusercontent.com/translate_c" supposed to be a part of VA Medical Center Sites?
Product description	Corresponding filtered product description appears with every filtered form	Corresponding filtered product description appears with every filtered form	Pass ✅
VA.gov Data Dictionary link - header	Links to Platform Website, "Content KPIs Data Dictionary" page	Links to Platform Website, "Content KPIs Data Dictionary" page	Pass ✅
VA.gov Data Dictionary link - footer	Links to Platform Website, "Content KPIs Data Dictionary" page	Links to Platform Website, "Content KPIs Data Dictionary" page	Pass ✅
Misc	Charts should be similar to approved design	Charts are similar to approved design	Pass ✅

chloedotbrown commented 3 years ago

@JasonTheBearded Can we do another Content data refresh pre-launch that recategorizes the URL Joanne flagged above?

JasonTheBearded commented 3 years ago

I think that's something coming in with the cleaned url. if it's a vamc site being translated in google, the vamc site would be included in the query and queries are getting cleaned out from the url.

chloedotbrown commented 3 years ago

Aha! Sounds like we may be good then?

joanneesteban commented 3 years ago

Yup! Thanks for looking into it, @JasonTheBearded & @chloedotbrown. Closing this ticket.

department-of-veterans-affairs / va.gov-team