search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
282 stars 203 forks source link

Analytics KPI dashboard request for 10203 #13102

Closed joanneesteban closed 3 years ago

joanneesteban commented 4 years ago

Request a KPI Dashboard

What this form is for

Use this template to request a KPI dashboard on Domo for your product.

Instructions for the VFS team

VFS Information

KPIs

10203 KPIs

Please leave the following blank

VSP Analytics & Insights Acceptance Criteria

Issue 1 - Groom Analytics for [team]

Issue 2 - Draft PST Dashboard for [team]

Definition of Done

bmcgrady-ep commented 4 years ago

Here is the 10203 DOMO Dashboard (WIP) - https://va-gov.domo.com/page/558555603.

Also, per conversation with Amy Knox, we need the COE (certificate of eligibility) data from the team in Buffalo to complete the Valid Submission KPI card.

bmcgrady-ep commented 4 years ago

Per this slack conversation with Amy, the new ask is for 2 more goal funnels.

From Amy: A funnel with successful submissions of just eligible users (applicants don't see "initial confirm eligibility" page nor " STEM confirm eligibility" page)? A funnel with successful submissions of all ineligible users (users see "STEM confirm eligibility" page and submit anyway)?

bmcgrady-ep commented 4 years ago

@amyknox - The funnels have been added to the DOMO dashboard. Let me know what you think.

amyknox commented 4 years ago

@bmcgrady-ep Thank you - this is looking promising! Curious about a couple things...

  1. Confirming that 10203 - Eligible Applicants Goal Funnel is: A funnel with successful submissions of just eligible users (applicants don't see "initial confirm eligibility" page nor " STEM confirm eligibility" page)? 10203 - Ineligible Applicants Goal Funnel is: A funnel with successful submissions of all ineligible users (users see "STEM confirm eligibility" page and submit anyway)?

  2. Is it possible to include the description of the various funnels within the dashboard so the distinction between the 4 funnels is clear?

  3. What's the date range for the various funnels - and how can you tell?

  4. What does "Sum of step number" mean in the lower funnels?

  5. What would be the LOE of moving funnels and graphs around the dashboard? Based on what we have now, we may want to flip the whole thing... 10203 - Eligible Applicants Goal Funnel and 10203 - Ineligible Applicants Goal Funnel at the top. Form 10203 - Open Funnel and Form 10203 - Closed Funnel in the middle 10203 Successful Submissions vs 1995 Successful Submissions at the bottom. It's going to become less relevant as time passes so we may eventually want to eliminate it.

My goal is to share this internally with the BAH team tomorrow morning and with Education tomorrow afternoon. I'll let you know if they have any additional input or questions.

bmcgrady-ep commented 4 years ago

@amyknox To answer your questions:

  1. 10203 - Eligible Applicants Goal Funnel is: A funnel with successful submissions of just eligible users, however these users could see the "initial confirm eligibility" page. We can change that, but can't an eligible user see the "initial confirm eligibility" page and still be eligible as long as they don't see the "STEM confirm eligibility" page?
  2. Yes, that is the case for the second Goal Funnel.
  3. The Goal Funnels are starting from September 1st - Present. Let me know if you want a different date range. Also, I added the date range to the cards.
  4. I believe it was an irrelevant calculation in DOMO, but I removed it from the funnels.
  5. I went ahead and moved these around.
amyknox commented 4 years ago

@bmcgrady-ep Thanks for the quick turnaround on those Qs. Responses with a couple follow-ups:

  1. 10203 - Eligible Applicants Goal Funnel: The "Initial confirm eligibility" page is triggered when an applicant has selected a non-Ch33 benefit on the Education Benefit Page. So if they see "Initial confirm eligibility", they're ineligible. Those applicants will (should!!) also see the STEM Confirm Eligibility page as a 2nd notice that they're ineligible.

  2. Great!

  3. The form launched early Sept. so I'm ok with starting 9/1/20. Curious if the date range will be static for the dashboard or if it increments forward like in Google Analytics?

3b. What are the date parameters for the "10203 Successful Submissions vs 1995 Successful Submissions" chart? Will that be static - always starting 1/1/2020 or will that increment?

  1. Thank you.

  2. Excellent! I like it. Also like the placeholder for the 22-10203 valid submissions + approved applications API at the bottom of the page.

bmcgrady-ep commented 4 years ago

@amyknox

  1. This has been updated to remove the "initial confirm eligibility" page.
  2. The date range updates every morning with the previous day's data, so it will increment forward like in Google Analytics. 3b. The date parameters for the "10203 Successful Submissions vs 1995 Successful Submissions" chart are January 1 - Present. They also update each day and increment forward.
amyknox commented 4 years ago

This is so good! Just 2 more questions:

  1. Could we swap the top 2 funnels so we have Eligible Applicants funnel on the left & Ineligible on the right?

  2. In the 10203 - Ineligible applicants... triple checking these applicants may or may not see "Initial Confirm Eligibility" but definitely see " STEM Confirm eligibility"?

If it's only showing the applicants who see "Initial Confirm Eligibility", we are only seeing those who are ineligible due to their benefit and not those ineligible due to exhaustion of Ch33 &/or field of study. All ineligible users will see STEM Confirm eligibility so if that's how the data is being pulled, we're good.

bmcgrady-ep commented 4 years ago

Hi @amyknox -

  1. Just swapped the top 2 funnels.
  2. Yes, those are applicants who definitely see the "STEM Confirm eligibility" page.
bmcgrady-ep commented 4 years ago

Hi @amyknox - does the dashboard look good? Let me know if we need to change anything, thanks!

amyknox commented 4 years ago

@bmcgrady-ep the dashboard looks great. I'll confirm with stakeholders on Thursday that all is a-ok, then we can close this. Thank you for all your help with this effort.

joanneesteban commented 3 years ago

@amyknox let us know if you need anything else!

amyknox commented 3 years ago

@joanneesteban - @bmcgrady-ep and I were chatting about refining the 22-10203 funnel so we can track eligible submissions vs. all submissions and be able to look at those numbers for specific periods of time.

Eligible submissions are not the same as successful submissions.

LMK if that work should be done here or in a new story.

bmcgrady-ep commented 3 years ago

@amyknox - Thank you for your patience here. After looking into this, the goal funnels in DOMO are for 9/1 - Present (the day before current day). We are not able to filter for the date of the funnels due to funnels being for the entire time period. However, if you want a specific date range for a funnel, I can create a separate funnel for you to use for that date range.

Also, I created these 2 segments for you eligible 10203 submissions vs ineligible 10203 submissions in Google Analytics. Here are the segments:

Just copy and paste the links into your browser and then select which view you want to import the segments into. Let me know if you need anything else!

amyknox commented 3 years ago

Hi Brian - I can't get those segments to work for me. The eligible applicants segment is showing the same submission rate as all applicants and there are no ineligible applicants which I know is not the case.

Just a reminder; Eligible applicants do not see either of the eligibility alerts. Ineligible applicants see one or both of the eligibility alerts. If it makes the work on you end easier, if an applicant sees the 1st alert (Initial confirm eligibility), they will also always see the 2nd alert (STEM Confirm eligibility). So for the segment, if it makes it easier, we really only need to track the 2nd alert and if they see that and submit, they're an ineligible applicant.

image.png

bmcgrady-ep commented 3 years ago

Hi @amyknox - It should be working now, I had to update the URL structure to the segments. Let me know if you have any issues with getting the segments to work.

amyknox commented 3 years ago

Hey @bmcgrady-ep - those are still not working for me. I reloaded them and then applied the new ones to the goal URL report and still getting the same results as in the screenshot above. Suggestions?

bmcgrady-ep commented 3 years ago

@amyknox - I believe I needed to reshare the updated segments with you. Here they are:

amyknox commented 3 years ago

Hey @bmcgrady-ep - Somethings night right. These look too good. We used to have a 90% rejection rate due to ineligible applicants and now we have a 99.9% eligible rate, per Google Analytics.

Also, checking against DOMO from 9/1, we have 147 ineligible applicants but GA segments say 40 for the same time period (pic below). And then... when you look at just October, the number of ineligible applicants for that shorter period of time more than doubles to 84 (2nd pic). ??

image

image

bmcgrady-ep commented 3 years ago

@amyknox - Which view are you looking at the segments in? I am looking in the Education View and this is the result I get with the segments, which closely matches the numbers in DOMO. image

amyknox commented 3 years ago

I don't know what was happening Tuesday. My numbers look like yours today.

However... wanting to confirm this 1 more time: Ineligible applicants = applicants who see this page: https://www.va.gov/education/other-va-education-benefits/stem-scholarship/apply-for-scholarship-form-22-10203/benefits/confirm-eligibility

I know it seems redundant to keep asking this but it is the make-or-break for the 22-10203 and I want to be sure we're on the same page (literally)!

image

bmcgrady-ep commented 3 years ago

@amyknox - Yep! This is the page that I have filtered out on the segment - /education/other-va-education-benefits/stem-scholarship/apply-for-scholarship-form-22-10203/benefits/confirm-eligibility.

amyknox commented 3 years ago

That's the page you filtered out on the "Eligible 10203 Submissions" segment?

bmcgrady-ep commented 3 years ago

No, that's the page that I am filtering for ineligible 10203 submissions - a user who lands on that page and submits the 10203 form would be an ineligible submission.

amyknox commented 3 years ago

Whew! I was using "filtering" in a different way. Glad we're on the same page. I think we're all good and... can CLOSE THIS STORY!!

Thank you so much for working through this.

bmcgrady-ep commented 3 years ago

Happy to help! Feel free to reopen if you have any more questions!

amycesal commented 3 years ago

The colors are matched across funnels:

Screen Shot 2020-11-13 at 12 48 56 PM
joanneesteban commented 3 years ago

Thanks for the tunnel fix, @amycesal Let's also add the tunnel table to be able to see the percentage dropoff.

amyknox commented 3 years ago

@bmcgrady-ep The numbers in my GA reports aren't mapping correctly to the DOMO reports. I'm also seeing an issue with the number of submissions increasing when I make the timeframe shorter.

22-10203 Submissions – Sept. 1 – Nov. 16, 2020 https://analytics.google.com/analytics/web/?authuser=0#/report/conversions-goals-uri/a50123418w177519031p184334251/_u.date00=20200901&_u.date01=20201116&_.goalOption=14&_.useg=builtin1,userzXgjk_0KTD6wipY8EsCSZw,userYWEd4_fEQcqVU3Sk9jXzdw/ image.png

22-10203 Submissions – Sept. 10 – Nov. 16, 2020 (updated start date only – observe how number of all submissions is greater despite the shorter timeframe) https://analytics.google.com/analytics/web/?authuser=0#/report/conversions-goals-uri/a50123418w177519031p184334251/_u.date00=20200910&_u.date01=20201116&_.goalOption=14&_.useg=builtin1,userzXgjk_0KTD6wipY8EsCSZw,userYWEd4_fEQcqVU3Sk9jXzdw/

image.png

bmcgrady-ep commented 3 years ago

@amyknox - I confirmed that the Goal Conversion is working correctly. The difference in the numbers has something to do with data sampling in Google Analytics.

bmcgrady-ep commented 3 years ago

@amyknox - If you remove the All Users segment from your analysis, this should give you normal numbers (when you narrow the date range the goal conversions go down).

Here is a screenshot of the results that I got when removing the All Users segment:

Date Range: September 1 - November 16 image

Date Range: September 10 - November 16 image

amycesal commented 3 years ago
Screen Shot 2020-11-17 at 3 00 10 PM

@bmcgrady-ep The data for the closed funnel doesn't include the percentage change. I can't include it in the table if it's not in the data. Let me know when it's fixed. Thanks!

joanneesteban commented 3 years ago

To Do, @bmcgrady-ep

bmcgrady-ep commented 3 years ago

@amycesal - I updated the data to include Percent Change. I had to create a new dataset, so can you update the table colors?

amycesal commented 3 years ago

All the funnels!

Screen Shot 2020-11-24 at 4 48 20 PM
bmcgrady-ep commented 3 years ago

@amyknox - Is there anything else you need in regards to this ticket? If not, I will close it out.

amyknox commented 3 years ago

@bmcgrady-ep Thank you & the rest of the team for your great work on this. Looks good to me to close!

bmcgrady-ep commented 3 years ago

@amyknox - Glad we can help! Feel free to reopen if there is anything else you need!