Build Proof of Concept for Product Success Tool MVP in Domo

[nedierecel]

User Story or Problem Statement

As an Education Product Owner, I want to test the Product Success Tool MVP within Domo so I can make decisions and iterate on my products.

Goal

• Recreate custom Product Success Tool MVP within Domo to see how close we can make it to the custom approach

Objectives or Key Results this is meant to further

O2: Customers use qualitative and quantitative user data to build and iterate on VFSs.

Acceptance Criteria

• [x] Use Google Analytics native connector to build non-BigQuery dependent charts within Domo
• [x] Customize dashboards to fit VA.gov style guidelines
• [x] Calculate error rates using GA data
• [x] Align Foresee data with Education product area
• [ ] Communicate with Education stakeholders about using the POC
• [ ] Stretch: Customize dashboards to fit Grey Sky/customer focused designs
• [ ] Stretch: Map which features will need more work beyond a sprint (i.e. % Difference from Target over time)

Acceptance Criteria for Data to be Included

• [ ] Available Google Analytics data (using the native Google Analytics connection)
• [ ] Available historical ForeSee data (using the pulled CSV file)

How to configure this issue

• [ ] Attached to a Milestone (when will this be completed?)
• [ ] Attached to an Epic (what body of work is this a part of?)
• [ ] Labeled with Team (product support, analytics-insights, operations, triage, tools-improvements)
• [ ] Labeled with Practice Area (backend, frontend, devops, design, research, product, ia, qa, analytics, call center, research, accessibility, content)
• [ ] Labeled with Type (bug, request, discovery, documentation, etc.)

[nedierecel]

Domo Progress

• Failure Rate: The failure rate in our custom MVP proved to be the more problematic of the calculations. I am using Domo's Magic ETL to perform some transformations and see if we can calculate this. For visibility, the steps so far are can be found in this data flow:

  • Filter by Submissions and Failures
  • Group by form type
  • Aggregate total events by month and year

I am blocked at the moment to calculate the error percentage but will keep plugging away.

[nedierecel]

Domo Follow-Up Notes for Sprint 17:

• I'll request a time with Domo to review mapping form names more easily and calculating form failures; tentative hold for 1/9 at 3:30-4
• We'll use this discovery to see how to map the event names (e.g. edu--submission) to a Form; the objective is to allow us to filter at a dashboard level across all data sources
• Amy will review the Domo MVP to figure out what other Gray Sky charts/reports are missing for MVP

[joanneesteban]

ForeSee Comment Card 2019 Added:

Played around the with data here: ForeSee_Comment_Card_Education_2019.xlsx

Ex:

[nedierecel]

@amycesal @joanneesteban @jonwehausen

1/6/2020 - I requested updates on the new subdomain and the extended trial. I have a follow-up call with Domo tomorrow at 9:30 EST to talk through it.

Our technical follow-up is scheduled for 1/9 at 3:30-4 EST

[nedierecel]

@amycesal @joanneesteban

1/7/2020 Progress - Our Domo trial has been extended. Before we begin more work on the trial instance I set up, Domo will move our dashboards to va-gov.domo.com and @jonwehausen and I just need to reconnect the original GA and CSV Foresee data sources. This part shouldn't be a huge haul since we did not have many sources built therein.

[joanneesteban]

@nedierecel @amycesal

1/7/2020 Got access to va-gov.domo.com using the va.gov email and sent invites. I don't see the dashboards there, though.

[nedierecel]

Thanks for helping @joanneesteban. The new instance had to be created before they can clone the dashboards from the old instance to the new. I’ll check in with Domo tomorrow on their progress. May you add Jon and I as admins to the new instance?

[joanneesteban]

@nedierecel Ok, good to know. And yes, everyone should have admin access.

[nedierecel]

@joanneesteban @amycesal @jonwehausen I added the Google Analytics data sources back to va-gov.domo.com and the education dashboard here: https://va-gov.domo.com/page/636836229

Other Updates:

• Goal Conversion Rate: There is a limit of 10 metrics per data source, and we have 11 education goals. I created a blended data set to ensure all of the goals are pulled

Next implementation steps for today and tomorrow:

• Will meet with Domo today to see how we can transform data to (a) calculate error rate and (b) assigning form names so we can filter at the dashboard level.

[nedierecel]

@joanneesteban @amycesal @jonwehausen Thanks for the discussion with Domo. The Domo webform should be the first step we tackle so that we can join the friendly data names across the data sets. I haven't built out all of the 11 products in education but started with 2 so far.

Here is my first pass showing the user-friendly names and filtering it on the Form Submissions chart.

Next steps for me:

• Add to Domo lookup table for all 11 forms
• Join the readable names with all the Google Analytics data sets

[amycesal]

Latest Grey Sky

[nedierecel]

Next Steps:

• Call Center Data - @Kimberley2019 @clantosswett Is the mapping of issues to product categories updated? If so, I can import this to Domo.
• Failure Rate - @jonwehausen I'll set up a working session with you as I have a few thoughts on Failure rate but want to get your input on most efficient.
• Combining Unauthenticated Pageviews with Form Submissions - @joanneesteban I rebuilt this data set to blend Unauthenticated Pageviews with Sessions. I'll confirm if this gives us the output we had in Gray Sky.
• Design - @amycesal I'll continue focusing on the data sources and charts, but feel free to tweak any of the design elements surrounding the charts.

[joanneesteban]

To help with Call Center report mapping @clantosswett / @nedierecel : the product list that @amycesal created.

[nedierecel]

1/14/2020 Progress:

• Failure Rate: @jonwehausen @joanneesteban I was able to transform the Form 1990 Submissions and Failures in a Data Flow called WIP - NR - Event Failures (working title!) In it, I was able to uncollapse the data then calculate a percentage rate of form failures.

  • DataFlow Overview
  • Output for Failure Rate using Form 1990

Next Steps for Failure Rate:

• I'd like to pause here and QA before proceeding with building the remaining rates.
• I need to sync the Product Names with the lookup table so it will dynamically filter the report when all failure rates are there.

[nedierecel]

End-of-Day 1/14/2020 Progress:

• Failure Rate:

  • Transformed data to calculate error rates with all applicable forms/tools; Need to validate, as I found 1 form that has an error rate>100% (I think this may be due to error in code that is underreporting --submissions )
  • Block: Getting dashboard filters to work properly with error rates table

• Blending Unauthenticated Pages and Form Submissions: I tried to join this data but the ETL process is timing out.

[nedierecel]

@jonwehausen - Thanks for syncing up after grooming. To recap, I'm hitting an error on the blended Content+Event Submissions dataflow. I believe this was originally due to the wrong Join, which I have now fixed.

I'm checking to see we're reaching an API limit with the connector. Feel free to poke around and let me know if you see something else causing the issue.

[nedierecel]

1/16/2020 Progress

• Introduced Nancy to rep for follow up conversations
• Technical Blocks: I was able to blend the unauthenticated pageviews and event submissions and now trying to plot them so they show all form submission types. I reached out to my technical contact Ron and also to Domo chat. Ron will not be available til next week to do a walkthrough.

[nedierecel]

1/20/2020 Progress

• Met with Technical Domo rep Ron to get assistance with Dashboard Product Filter and to understand best method to plot error rates
• Added HRC as a data source

Next Steps

• Need to redo the Failure rates chart so that it filters correctly at the dashboard level
• Will add line graph for HRC data

[nedierecel]

@amycesal Summary of Design Blocks:

1. Failure Rates and Submission Rates - To get the error rate calculations to work, I had to build the Failure Rates as a set of independent series (See screenshot #1). The impact is that when filtering on the friendly Product Names, it does not give the user the same behavior with (a) selecting the same color and (b) hiding the legend. To recreate, visit the dashboard and select 1 or more of the Dashboard filters.

Screenshot 1

Screenshot 2

Other Updates:

1. I added the HRC line graph and currently populating the rest of the available data.

Next Steps:

1. Design Block - I will try to see if I can remap the data sources so the product filters work similarly across both charts.

[nedierecel]

@jonwehausen Here is the Google Analytics link to help me validate form failure rates: https://analytics.google.com/analytics/web/?authuser=0#/report/content-event-events/a50123418w177519031p184334251/_u.date00=20191201&_u.date01=20191231&explorer-segmentExplorer.segmentId=analytics.eventLabel&explorer-table.plotKeys=%5B%5D&explorer-table.filter=edu-1995

We completed 1995 together during our 1-on-1 today. Can we divide and conquer the rest? I can take on:

• [x] 1990
• [ ] 1990e
• [x] 0993
• [ ] 0994

May you help review:

• [x] 1990n
• [x] 5490
• [x] 5495
• [x] GI Bill Feedback Tool

Two products are omitted because they are views of GI Bill Facilities and Post 9-11 GI Bill Status. The GI Bill Facilities does not have a failure event; the Post 9-11 GI Bill Status has failures and successes but not a submission since it is not a form.

[joanneesteban]

Follow up to ensure we're moving forward & can deliver #5001 before the end of the sprint.

Chart Functionality

1. The filters at the top of the page should correspond to the Failure Rates chart as well (@nedierecel @jonwehausen )
2. Can we add targets to the current charts?

Tool Usage/Process

1. For the current metrics we have surfaced on the dashboard, determine baselines.
2. Determine a path forward for the VFS team if a KPI is not meeting the baseline.
3. Determine potential escalation if actions have not taken place for an "unhealthy" KPI.

Our goal here is to ensure that if we're not surfacing the correct metrics, we can fail fast at that and focus on ensuring we will surface the correct metrics sooner.

Access

@nedierecel - how many licenses should we have? Do we need to follow up with Nancy on this?

@amycesal have you reached out to Matt about this yet? Is there anything else we can help with to ensure you have a path forward to get this to stakeholders by end of sprint?

[nedierecel]

1. Filters - The error rates will also filter down when you use the Product filter. In my QA, I was disabling it temporarily so I can validate all of the error rates against GA.
2. Targets - I see our meeting with Chris on Monday. I can add a single target line to broad metrics like failure rate; the targets for the form submissions would require more effort so I can map each target to each form then plot. I will check into this resource on Domo on mapping targets.

Licenses - I will send a note back to Jan as I imagine they are configuring our org settings.

[nedierecel]

@jonwehausen I am going to fix my case statement in Domo. In its current state, it includes all event labels for failed-client but is missing a pattern that ends in failed only. I will send a note when this is ready for validation.

[jonwehausen]

@nedierecel Happy to go back and re-validate when needing to add the failed-client but the failed-submission is working fine.

Sample test procedure, used for validating numbers of each form. See example for 1990, checked each of the forms in your bottom half here, looks like it's working great!

1990n ✔️

[joanneesteban]

Other Follow Ups

User Tracking

The "Activity Log," referenced here, disappeared after the contract started on 1/24/20.

Design

@nedierecel is there a way to link the charts back to their source?

[nedierecel]

@jonwehausen The error rates have been updated and are ready for review. I updated the Product Name lookup table to account for the other version of the label.

• Forms 1995, 5495, and 5490 - These forms each have 1 month that is slightly off from Google Analytics. For example, Form 1995 for December is reporting .522% in Domo but GA calculation is .526%. I need to see if it sampling.

• @joanneesteban I believe there is a way with a table but not sure if it's supported with a line graph. Here is the Domo resource on adding links to a table

[joanneesteban]

It doesn't look like there is. I wonder what work arounds are available, then - if there's a way to add it to the Tooltips (but then it wouldn't be dynamic based on date).

[nedierecel]

@amycesal To align the colors on the form submissions and errors, I used the color rules option to apply the same color to the same products. Here is where this stands:

• I am trying troubleshoot as it didn't seem to inherit the settings. Can you also review?
• I picked the colors arbitrarily; the feature did not seem to offer any options for using HTML colors, so I'll turn it to you for selecting the colors so it meets color compliance

[joanneesteban]

Remaining AC for build (or for design QA)

@nedierecel & @amycesal to collab on how

• [ ] Data in graphs are accurate
• [ ] Targeting visualized
• [ ] Dashboard user flow is updated
• [ ] Titles & text details are clear, accurate, & 'helpful'
• [ ] Charts design & filters are updated

Next steps for testing is in 5001

[joanneesteban]

Remaining Question

• How do we provision access to users? How do we release it as well?

[amycesal]

Survey for Users:

https://forms.gle/VBpGeckdVnNiBSCz6

[joanneesteban]

@amycesal Good start! Can we add some questions to get us closer to:

• Does the Domo format (historical trend, access to dashboard) provide different analysis than the Google Sheets MVP? (to see if there's additional or different value in this format vs the original Google Sheet format)
• Would they still use the Google Sheets MVP? (to see if we can/should deprecate the Google Sheets MVP)
• Which of the charts are useful/not useful? (for how we cut the data)
• What their role is (for which of our users are actively looking at the data)

[amycesal]

Ok, I want to keep some of the questions the same so we can use them between both facility locator and Education. But I've added:

[amycesal]

I had @gmaciolek take a look, and she had some great suggestions.

https://forms.gle/zohCVFY8tXaaeLo46

[amycesal]

Updated the MVP with changes from @gmaciolek.

@nedierecel or @jonwehausen the CSAT/Forsee chart still needs the same time scale as the others.

[nedierecel]

@joanneesteban @amycesal I would recommend we create Participant level permissions for all stakeholders that want to view content. This would require in advance we provision access for education stakeholders and other leadership.

Resource: Sharing Content in Domo

For long-term roadmap and its discovery, we can continue looking at SSO or embedding.

CC: @jonwehausen

[nedierecel]

@joanneesteban @amycesal Timescale for CSAT data has been updated to reflect Feb-April 2019. Since comment card was not released until May 2019, the data points are empty for earlier part of the year.

[joanneesteban]

@nedierecel

Added ForeSee dataset to Domo. It's all URLS, so needs to be filtered by Education. Date range is 05/01/2019 - 01/31/2020, with dummy data added for Jan-Apr 2019.

[nedierecel]

@joanneesteban I updated the chart to reflect Education data through January 2020.

A few clean up items we can document in our Research folder:

• I downloaded the file and added Education URLs to the placeholder months from Jan-Apr 2019
• I then filtered on it using Magic ETL. Oddly, when I tried to add your new data source then filter on the card level, it wouldn't show the earlier 2019 months.

[joanneesteban]

Additional Follow Ups

CSAT Chart

• @nedierecel that's odd - so then we would have to do that for all products for the placeholder months?

Release

• @nedierecel @jonwehausen in thinking about releasing the Education POC a bit more assertively than the #vfs-analytics Slack channel, is there a way to show 1 chart, and then have a link for a 'click here to see more' login to Domo? How would providing links work in terms of provisioning access?
• Can we provision Kevin with access as well? I'll put together a list of folks we should give access to.

[nedierecel]

CSAT Chart

• Yes, I think we'd need to recreate.

Release

• Can we proactively add a list of stakeholders initially and share a screenshot/link to see more? If others are interested who don't fall on that list, can we then process requests for other users?
• Yes, we can provision Kevin with access.

[joanneesteban]

Release

• Working with Kevin H to schedule a demo with Education stakeholders

• Share on #vfs-analytics and #vfs-all-teams with the following (draft) message:

  🎉 📈 We're pleased to announce a Product Success Tool proof of concept with the Education products. Here is a preview of the dashboard. To request access to the entire dashboard, please emoji react to this post with :domo:

• With a comment:

  Many folks should have received an invite from Domo to create an account! If you did, this isn't spam! Please create an account to see 📊! If you did not, please ping us to add an account for you!

Ideal Launch Date - end of day, Monday, 3/2

---

Survey

• Provide the survey with the demo
• Determine when the survey will be released (with the report, or a few days after?)

[joanneesteban]

Other To Do

@nedierecel

For consistency, add dummy data to allow 13 month chart. Add annotations at the bottom to relay that those months are not rolling, but will need to wait until the end of month.

Annotations Also add annotations that the GA data is not complete until the end of the month. Provide the URLs that the education survey is coming from.

Discovery on how to add additional ForeSee data can be added monthly.

---

@joanneesteban Ping Kevin to set up Education stakeholder meeting.

---

@nedierecel & @joanneesteban

• Set up a list of stakeholders we'll give access to now.
• Is there a way to provision access without notifying them?

[joanneesteban]

Access List

@nedierecel - started a list of user access for Domo here.

[nedierecel]

@joanneesteban

Other Data Points

• GA Data - I added a note to the Appendix re: GA data reflects month-to-date activity
• ForeSee - Added February placeholder
• HRC - Added February placeholder; January data was not provided from HRC (from the Jan 2020 report provided by @clantosswett) so added that context

Blocks

• ForeSee Connector - My initial review of the connector didn't appear to have the URL column to filter by. I will keep poking, but we may need to continue with exporting data from ForeSee for MVP.

[joanneesteban]

@nedierecel

Access for Rollout

I've added the rest of the VA stakeholders to the Google Sheet, so we should be good to create the CSV for bulk user add when we release.

Initial Rollout Schedule

• Release on the #vfs-analytics DSVA Slack channel on Monday, 3/2/20 or no later than COB Tuesday, 3/3/20. We will need to add the final February CSAT before release. Let's also release the survey along with the release so that that stakeholders understand this can serve as a replacement for the original MVP/isn't a groundbreaking dashboard in terms of the data within it. Let's link the Google Sheets MVP to the survey so it's easier to see as well. (@amycesal )
• Demo with the Education stakeholders - tentatively set for Monday, 3/9/20 at 10-10:30am ET.
• Regroup on feedback at the end of Sprint 21.

[nedierecel]

@joanneesteban

1. The CSV sheet is ready for all the recipients marked "Add to Initial Release"
2. Groups for each recipient were created in Domo, then each user was associated with that group.
3. Each group, as noted in Col. C, was provided access to the Education MVP dashboard.

[joanneesteban]

Sprint 21 To Do

1. Demo with Education team
2. Retrieve feedback & synthesize

[nedierecel]

Sprint 21 Update

@joanneesteban @amycesal - I updated the Education MVP (Working Copy) Dashboard to reflect weekly time periods using dates in lieu of week numbers. There is also a slightly different version of notes in the Appendix.

@amycesal The weekly visualization is a bit crowded. Can you update?

[joanneesteban]

Initial Feedback

One piece of feedback in terms of usability so far has been that filtering by form is confusing. When they filter by form within the chart, it excludes that form instead of filters by it. This is probably an issue with how Domo does its charting, but if there's a way for us to workaround it or maybe put some sort of instructions?