QA Forms KPI Dashboard - Design/Content

joanneesteban commented 3 years ago

Issue Description

As we start to roll out our dashboards, we need to make sure that our dashboards are usable.

Tasks

QA for the following

[ ] Design
[ ] Content Other
[ ] All links should be clickable and correct
[ ] Filters should work properly
[ ] Document feedback

Acceptance Criteria

[ ] The production Forms KPI Dashboard in Domo is usable and follows the design as close as possible

joanneesteban commented 3 years ago

Some initial feedback (using our Participant Test User account):

Topic	Notes
Exports	Formatting for exports to PPT are a little off:
form_name Filter	When I go onto the dashboard, the title automatically populates to Higher Level Review. But the data reflects all of the Forms that are in the dataset. When I filter for `ALL` form_name`s, the title also says Higher Level Review, and when I filter for`NONE` of the form_name's, the title says Higher Level Review and filters for all of the data as well. Is there a way to edit the title for "All available Forms" or something of the equivalent?
Footer Links	The hyperlinks at the footer aren't clickable

mandy-uxd commented 3 years ago

Form dashboard - QA

chloedotbrown commented 3 years ago

Compiled changes to improve usability:

General / Intro / Footer

[x] Title default: Currently reads "Higher Level Review" when all forms selected. Ideally, should read "Please select a form" in default state.
[ ] Charts in default setting: Mandy also suggested having charts below set to something like "no data available" messages if no form has been chosen. Don't believe this functionality is possible, but will address in documentation and subheading dynamic text.
[x] Title formatting: Reduce text size and test to make sure no form names are cut off.
[x] Subheading: Add link to selected form's landing page (see mockup). Add link to "How to Use This Dashboard." Add instructions for asking further questions in Slack channel. (Still need to create "How to Use this Dashboard" doc to be able to link)
[ ] Filters: Start with capital letters and remove underscores. Domo's limitations mean this would have to be done through ETLs for each feeder datasource (can't be done with BQ).
[ ] Exports: Strange alignment when exported to PPT. Doug confirmed that Domo doesn't currently provide any ways to select where slide breaks occur or otherwise format exports.
[x] Footer: Make sure all links live/correct. (NOTE: still need to create "How to Use this Dashboard" doc to be able to link)
[x] Appendix: Should not exist!

Highlights

[ ] Forms by Status: Add "not submitted" category. @JasonTheBearded - do we still want to update to waterfall chart with a new ETL for this or add to current bar format?
[x] Unique users: Remove y-axis.
[x] Avg. minutes to complete: Add decimals to hundredths.

Form completion

[x] Forms started: Check % change settings - Mandy's screenshot was super high.
[x] Form completion rate: Add VA average, either to chart or text below for reference.
[ ] Completion funnel table: Add fill color to headers. Add sort functionality to all columns. Adjust column width to match mockup. Should be able to scroll through entire table.

Form timing & exits

[ ] Form exits: (same notes as above) Add fill color to headers. Add sort functionality to all columns. Adjust column width to match mockup. Should be able to scroll through entire table.
[x] Avg. sessions to complete: Change formatting to comparison value to be percent.

Finding the form

[ ] Top pages viewed before: (same notes as above) Add fill color to headers. Add sort functionality to all columns. Adjust column width to match mockup. Should be able to scroll through entire table.

Landing page

[x] All multi-value gauges: Reduce size - all too big.
[x] Top campaigns: Remove y-axis label. Left-align left axis text.

chloedotbrown commented 3 years ago

Still needs action

Domo chart options cannot do both HTML progress bars and scrollable tables in the dashboard view. We'll need to decide how else to represent funnel next to scrollable table, have Brian design custom chart, or other approach.
~~ETL to show form status with "not submitted" category instead of "started". Working with Jason to complete.~~ Resolved!

Additional changes made:

Changed spark line chart in Highlights to regular line chart to better handle formatting for dates with no submissions
Added "last week" to percent change labels on multi-value gauges for clarity
Updated dashboard title and intro text to encourage/reflect filter interaction

Changes not implemented due to Domo limitations:

Sentence case filter names (underscores removed)
Charts with no data when single form hasn't been selected
Edits/reformatting to PPT, slide, or PDF exports

chloedotbrown commented 3 years ago

Updated handling of dates in filters, graphs without date in X-axis, and units of comparison in multi-value gauges to be consistent with the new logic used in Search dashboard build (see comment here). This should make it easier for the user to understand how data is being compared and filtered.

joanneesteban commented 3 years ago

@chloedotbrown sorry about the delay here.

The following was tested using our "Participant" user account.

Another random thing: does the chart selection for "successful submissions" in the highlights section make sense? It looks like there were submissions during dates where there weren't because of the area fill.

Test	Expected Behavior	Actual Behavior	Pass/Fail
Default page	Page does not automatically filter, and description reflects lack of filter.	Page does not automatically filter, and description reflects lack of filter.	Pass ✅
Filter on Forms	Every chart gets filtered	Every chart gets filtered	Pass ✅
Filter on Device Category	Every chart gets filtered	(Form 1010EZ filtered on "mobile") The following charts do not appear to have the device category filter in the filter hovers: Highlights section: - Completed in 1 session Form timing & exits section: - Avg. sessions to complete - Sessions to complete form	Fail ❌
Filter on Browser	Every chart gets filtered	(Form 1010EZ filtered on "Firefox") The following charts do not appear to have the browser filter in the filter hovers: Highlights section: - Successful submissions - Forms by status - Unique users - Completed in 1 session Form completion section: - Forms started - Forms submitted - Form completion rate - Completion by device Form timing & exits section: - Avg. sessions to complete - Sessions to complete form Finding the form section: - Top pages viewed before accessing the form Landing page section: - Unique visitors - Avg. visits per user - Avg. seconds on page - Avg. scroll depth - Left without visiting form - Top sources - Top campaigns	Fail ❌
Date Filters	Every chart gets filtered on date	- `Top sources` chart in `How did people find the landing page` section does not filter for any form filter and non-default date filter - (Form 5655 filtered on "This Month")`Successful submissions` chart in `Highlights` section filters additional date ranges (screenshot should filter from June 1)	Fail ❌ second & third bullet points may be a Domo limitation.
How to use this dashboard - header	Links to Platform Website, "How to use this dashboard" page	Links to internal Confluence page	Fail ❌ Need to update to Platform Website link
How to use this dashboard - footer	Links to Platform Website, "How to use this dashboard" page	Links to internal Confluence page	Fail ❌ Need to update to Platform Website link
Chart Title	Corresponding filtered form appears as the page title	Corresponding filtered form appears as the page title	Pass ✅
See form landing page	Links for all available Forms go to the correct VA.gov production landing page	Form 22-1995 goes to https://www.va.gov/education/how-to-apply/ instead of https://www.va.gov/education/apply-for-education-benefits/application/1995/introduction	Fail ❌ Need to fix 1995 landing page (double check data source as well)
Form description	Corresponding filtered form description appears with every filtered form	Corresponding filtered form description appears with every filtered form	Pass ✅
Data sources	Correct data sources appear	Medallia appears as well as Google Analytics & ForeSee	Fail ❌ Is Medallia data included here? This will affect the annotation in User surveys section as well. "Foresee" should be "ForeSee"
Section titles	Every section title & question is spelled correctly	Every section title & question is spelled correctly	Pass ✅
Chart titles	Every chart title is spelled correctly	Every chart title is spelled correctly	Pass ✅
Annotations	Every annotation "makes sense" and is spelled correctly	Every annotation "makes sense" and is spelled correctly	Pass ✅
Labels	All data labels appear where expected.	All data labels appear where expected.	Pass ✅
Hover text	All hover text "makes sense" and is spelled correctly	Highlights section: - Forms by status title hover is "test description"	Fail ❌
Funnel hover text	All funnel hover text "makes sense" and is spelled correctly for each form	(Form 20-0996 filtered) `7 - Review and Submitt` should be `7 - Review and Submit`	Fail ❌
VA.gov Analytics Playbook link	Links to Github Analytics Playbook	Links to Github Analytics Playbook	Pass ✅
Google Analytics link	Links to Google Analytics 360	Links to Google Analytics 360 - WBC	Pass ✅
VA.gov Data Dictionary link	Links to Platform Website, "Forms Data Dictionary" page	Links to internal Confluence page	Fail ❌ Need to update to Platform Website link
Domo Help Center link	Links to Domo Help Center	Links to Domo Help Center	Pass ✅

JasonTheBearded commented 3 years ago

@joanneesteban The introduction page for 22-1995 is step 1 of the funnel. Does it make sense to have the landing page be the same as step 1? Or would https://www.va.gov/education/change-gi-bill-benefits/ work better?

joanneesteban commented 3 years ago

@JasonTheBearded was that intro page defined by the Education team? If so, then we can probably keep the landing page that's currently on the dashboard. Otherwise, the link that you have is probably better since the landing page can be the start to multiple education forms.

chloedotbrown commented 3 years ago

Filter on device category/browser: Failed filter tests caused by those fields not being included/named differently in some of our datasources. We've added both to Standard Forms User Completion and browser to Standard Forms Engagement and Standard Forms Landing Pages - will retest once updated datasets have reloaded. Date filters:

Top sources – Really couldn't figure out why this one card wasn't impacted by the date filters! Checked filters, card interactions, etc. Eventually ended up rebuilding a new version of the card and even making sure every element matched, only the new one responded to the filter. In the end, I replaced the card with the new version and added "archived" to the title of the old one. All seems to be working, but it's something to keep an eye out for in future QAs, as these page filters are still in beta.
Successful submissions – This isn't an error with the filter, but a quirk of how Domo aggregates. It's actually is showing data starting June 1, but because that week starts with 5/30, that's where it shows the data point. Including a screenshot with the date range subbed in the summary number slow (it's usually a form total) as an example. I don't have a good workaround for this, so I think we'll just have to be clear about it in the how-to documentation.

Data sources: Have removed "Medallia" from list of data sources in dashboard intro text and user survey section description. "S" in ForeSee now capitalized in both locations. Medallia task completion placeholder also removed. I've also added language to call out that device and browser filters don't work on the user survey data, which could be added to the data dictionary as well.

Hover text: "Test description" removed from card description field. No hover text appears above title for "Forms by status" card in the Highlights section.

Funnel hover text: Wondering what the best approach to fixing this typo is. @JasonTheBearded – could we correct in an ETL or is there another upstream solution?

I think most of the other failures are link updates once documentation is published on the Platform site, which we can dig into next week.

joanneesteban commented 3 years ago

Fantastic! Thank you both for the quick fixes here! 🎉

JasonTheBearded commented 3 years ago

@chloedotbrown For the funnel hover text it's an upstream fix, the step name was inserted into the table incorrectly. i've fixed that and am refreshing the datasource

JasonTheBearded commented 3 years ago

@chloedotbrown Data refresh complete for funnel and spelling error is corrected. Good to go :)

chloedotbrown commented 3 years ago

Double-checked device and browser filters and funnel text, now that all data has been refreshed. All passes! ✅

joanneesteban commented 3 years ago

Going through the QA checklist again. Is this chart supposed to be blank, @chloedotbrown @JasonTheBearded ?

joanneesteban commented 3 years ago

It's also blank for form 5655.

joanneesteban commented 3 years ago

Usability Test 6-29-2021

@chloedotbrown @JasonTheBearded almost there!

Test done using the Participant Tester:

chloedotbrown commented 3 years ago

@joanneesteban , thanks for catching! Looks like there was a rogue exclusion filter on the "Top sources" card. I've removed it, and both are now filtering correctly (screenshot below). I also removed the test description from "Forms by status"... and double checked all other cards for this. All seems clear, but feel free to give it another look!

joanneesteban commented 3 years ago

Thanks for the rapid work, @chloedotbrown !

Update to the test above 06-29-2021 17:36 EST

Test done using the Participant Tester:

All tests passed. Closing this ticket!

department-of-veterans-affairs / va.gov-team