Closed nedierecel closed 4 years ago
Failure Rate: The failure rate in our custom MVP proved to be the more problematic of the calculations. I am using Domo's Magic ETL to perform some transformations and see if we can calculate this. For visibility, the steps so far are can be found in this data flow:
I am blocked at the moment to calculate the error percentage but will keep plugging away.
edu--submission
) to a Form; the objective is to allow us to filter at a dashboard level across all data sourcesPlayed around the with data here: ForeSee_Comment_Card_Education_2019.xlsx
Ex:
@amycesal @joanneesteban @jonwehausen
1/6/2020 - I requested updates on the new subdomain and the extended trial. I have a follow-up call with Domo tomorrow at 9:30 EST to talk through it.
Our technical follow-up is scheduled for 1/9 at 3:30-4 EST
@amycesal @joanneesteban
1/7/2020 Progress - Our Domo trial has been extended. Before we begin more work on the trial instance I set up, Domo will move our dashboards to va-gov.domo.com and @jonwehausen and I just need to reconnect the original GA and CSV Foresee data sources. This part shouldn't be a huge haul since we did not have many sources built therein.
@nedierecel @amycesal
1/7/2020 Got access to va-gov.domo.com using the va.gov email and sent invites. I don't see the dashboards there, though.
Thanks for helping @joanneesteban. The new instance had to be created before they can clone the dashboards from the old instance to the new. I’ll check in with Domo tomorrow on their progress. May you add Jon and I as admins to the new instance?
@nedierecel Ok, good to know. And yes, everyone should have admin access.
@joanneesteban @amycesal @jonwehausen I added the Google Analytics data sources back to va-gov.domo.com and the education dashboard here: https://va-gov.domo.com/page/636836229
Other Updates:
Next implementation steps for today and tomorrow:
@joanneesteban @amycesal @jonwehausen Thanks for the discussion with Domo. The Domo webform should be the first step we tackle so that we can join the friendly data names across the data sets. I haven't built out all of the 11 products in education but started with 2 so far.
Here is my first pass showing the user-friendly names and filtering it on the Form Submissions chart.
Next steps for me:
Next Steps:
To help with Call Center report mapping @clantosswett / @nedierecel : the product list that @amycesal created.
1/14/2020 Progress:
Failure Rate: @jonwehausen @joanneesteban I was able to transform the Form 1990 Submissions and Failures in a Data Flow called WIP - NR - Event Failures
(working title!) In it, I was able to uncollapse the data then calculate a percentage rate of form failures.
Next Steps for Failure Rate:
End-of-Day 1/14/2020 Progress:
Failure Rate:
--submissions
)Blending Unauthenticated Pages and Form Submissions: I tried to join this data but the ETL process is timing out.
@jonwehausen - Thanks for syncing up after grooming. To recap, I'm hitting an error on the blended Content+Event Submissions dataflow. I believe this was originally due to the wrong Join, which I have now fixed.
I'm checking to see we're reaching an API limit with the connector. Feel free to poke around and let me know if you see something else causing the issue.
1/16/2020 Progress
1/20/2020 Progress
Next Steps
@amycesal Summary of Design Blocks:
Screenshot 1
Screenshot 2
Other Updates:
Next Steps:
@jonwehausen Here is the Google Analytics link to help me validate form failure rates: https://analytics.google.com/analytics/web/?authuser=0#/report/content-event-events/a50123418w177519031p184334251/_u.date00=20191201&_u.date01=20191231&explorer-segmentExplorer.segmentId=analytics.eventLabel&explorer-table.plotKeys=%5B%5D&explorer-table.filter=edu-1995
We completed 1995 together during our 1-on-1 today. Can we divide and conquer the rest? I can take on:
May you help review:
Two products are omitted because they are views of GI Bill Facilities and Post 9-11 GI Bill Status. The GI Bill Facilities does not have a failure
event; the Post 9-11 GI Bill Status has failures and successes but not a submission since it is not a form.
Follow up to ensure we're moving forward & can deliver #5001 before the end of the sprint.
Chart Functionality
Tool Usage/Process
Our goal here is to ensure that if we're not surfacing the correct metrics, we can fail fast at that and focus on ensuring we will surface the correct metrics sooner.
Access
@nedierecel - how many licenses should we have? Do we need to follow up with Nancy on this?
@amycesal have you reached out to Matt about this yet? Is there anything else we can help with to ensure you have a path forward to get this to stakeholders by end of sprint?
Licenses - I will send a note back to Jan as I imagine they are configuring our org settings.
@jonwehausen I am going to fix my case statement in Domo. In its current state, it includes all event labels for failed-client
but is missing a pattern that ends in failed
only. I will send a note when this is ready for validation.
@nedierecel Happy to go back and re-validate when needing to add the failed-client
but the failed-submission
is working fine.
Sample test procedure, used for validating numbers of each form. See example for 1990, checked each of the forms in your bottom half here, looks like it's working great!
The "Activity Log," referenced here, disappeared after the contract started on 1/24/20.
@nedierecel is there a way to link the charts back to their source?
@jonwehausen The error rates have been updated and are ready for review. I updated the Product Name lookup table to account for the other version of the label.
Forms 1995, 5495, and 5490 - These forms each have 1 month that is slightly off from Google Analytics. For example, Form 1995 for December is reporting .522% in Domo but GA calculation is .526%. I need to see if it sampling.
@joanneesteban I believe there is a way with a table but not sure if it's supported with a line graph. Here is the Domo resource on adding links to a table
It doesn't look like there is. I wonder what work arounds are available, then - if there's a way to add it to the Tooltips (but then it wouldn't be dynamic based on date).
@amycesal To align the colors on the form submissions and errors, I used the color rules option to apply the same color to the same products. Here is where this stands:
@nedierecel & @amycesal to collab on how
Next steps for testing is in 5001
Survey for Users:
@amycesal Good start! Can we add some questions to get us closer to:
Ok, I want to keep some of the questions the same so we can use them between both facility locator and Education. But I've added:
I had @gmaciolek take a look, and she had some great suggestions.
Updated the MVP with changes from @gmaciolek.
@nedierecel or @jonwehausen the CSAT/Forsee chart still needs the same time scale as the others.
@joanneesteban @amycesal I would recommend we create Participant level permissions for all stakeholders that want to view content. This would require in advance we provision access for education stakeholders and other leadership.
Resource: Sharing Content in Domo
For long-term roadmap and its discovery, we can continue looking at SSO or embedding.
CC: @jonwehausen
@joanneesteban @amycesal Timescale for CSAT data has been updated to reflect Feb-April 2019. Since comment card was not released until May 2019, the data points are empty for earlier part of the year.
@nedierecel
Added ForeSee dataset to Domo. It's all URLS, so needs to be filtered by Education. Date range is 05/01/2019 - 01/31/2020, with dummy data added for Jan-Apr 2019.
@joanneesteban I updated the chart to reflect Education data through January 2020.
A few clean up items we can document in our Research folder:
Working with Kevin H to schedule a demo with Education stakeholders
Share on #vfs-analytics and #vfs-all-teams with the following (draft) message:
🎉 📈 We're pleased to announce a Product Success Tool proof of concept with the Education products. Here is a preview of the dashboard. To request access to the entire dashboard, please emoji react to this post with :domo:
With a comment:
Many folks should have received an invite from Domo to create an account! If you did, this isn't spam! Please create an account to see 📊! If you did not, please ping us to add an account for you!
Ideal Launch Date - end of day, Monday, 3/2
@nedierecel
For consistency, add dummy data to allow 13 month chart. Add annotations at the bottom to relay that those months are not rolling, but will need to wait until the end of month.
Annotations Also add annotations that the GA data is not complete until the end of the month. Provide the URLs that the education survey is coming from.
Discovery on how to add additional ForeSee data can be added monthly.
@joanneesteban Ping Kevin to set up Education stakeholder meeting.
@nedierecel & @joanneesteban
@nedierecel - started a list of user access for Domo here.
@joanneesteban
@nedierecel
I've added the rest of the VA stakeholders to the Google Sheet, so we should be good to create the CSV for bulk user add when we release.
@joanneesteban
@joanneesteban @amycesal - I updated the Education MVP (Working Copy) Dashboard to reflect weekly time periods using dates in lieu of week numbers. There is also a slightly different version of notes in the Appendix.
@amycesal The weekly visualization is a bit crowded. Can you update?
One piece of feedback in terms of usability so far has been that filtering by form is confusing. When they filter by form within the chart, it excludes that form instead of filters by it. This is probably an issue with how Domo does its charting, but if there's a way for us to workaround it or maybe put some sort of instructions?
User Story or Problem Statement
As an Education Product Owner, I want to test the Product Success Tool MVP within Domo so I can make decisions and iterate on my products.
Goal
Objectives or Key Results this is meant to further
O2: Customers use qualitative and quantitative user data to build and iterate on VFSs.
Acceptance Criteria
Acceptance Criteria for Data to be Included
How to configure this issue
product support
,analytics-insights
,operations
,triage
,tools-improvements
)backend
,frontend
,devops
,design
,research
,product
,ia
,qa
,analytics
,call center
,research
,accessibility
,content
)bug
,request
,discovery
,documentation
, etc.)