Closed joanneesteban closed 4 years ago
For an explanation of the closed funnel vs open funnel:
The open funnel chart is flexible in that: the user can enter at any stage, and accounts for pages that can be skipped or not shown at all (i.e /confirm-eligibility). For example, in cases where a user doesn't reach
/confirm-eligibility
, that step is still counted towards funnel execution. The conversion rate is agnostic of whether someone reached the ineligibility alert or not.The closed funnel charts ensure the user was forced to reach every step of the form. In other words, the closed funnel charts ensures folks definitely reached
/confirm-eligibility
and carried on with completion. This funnel type may undercount conversions if users don't follow those steps precisely.Current limitation to both funnels to note: These charts enforce the user started and completed the form within the same visit. In other words, if someone started the form, saved, came back 3 days later and completed those would not be counted. We are working on follow-up queries to account for these scenarios.
@jonwehausen - thank you for putting this together!
A couple questions...
Could the closed funnel be customized to exclude the 1st eligibility alert from the funnel?
If people see the 2nd alert, by definition they would have seen the 1st one, too. So if we could exclude that, it'd really show us everyone whose responses indicated they were ineligible but applied anyway. The view with both alerts is showing us just the people who are ineligible due to the benefit they're using.
@jonwehausen
Another KPI identified in the 10203 product outline is "Successful 22-10203 STEM Scholarship Application submissions esp. compared to applications via 22-1995"
This is reliant on reporting on COEs issued by the team in Buffalo - would there be a way to incorporate their external data into the DOMO dashboard?
Interestingly, there was a rule change last month that relaxes how eligibility is determined so the approval rate for STEM via 10203 will get an unexpected boost
@amyknox Hi! Can you elaborate on the COEs from Buffalo? Where does the data live, and can you let us know how the KPI is reliant on this?
COEs are Certificates of Eligibility which are issued to applicants who are approved for the STEM scholarship. The data would live on the system that the folks in Buffalo work with directly - not something that is GA trackable.
The KPI relies on this as these are individuals who complete a successful submission AND are found to be eligible for the scholarship. As you may recall, a key issue with the STEM scholarship application has been that the team in Buffalo has been dealing with an over-abundance of ineligible applicants.
@amyknox Thanks for explaining!
Also, would you say that your KPI is:
Successful 22-10203 STEM Scholarship Application submissions
= Eligible applicants
/ Successful VA.gov 10203 submissions
Is this different from the KPI, Valid 22-10203 Application submissions, # and %
I'm looking into COE's. Each individual who gets approved for education benefits gets a COE. I'm not optimistic there's an API to track that but will let you know what I find out!
For valid vs. successful submissions, we should find an agreed on vernacular. Here's my thinking... "Successful submissions" are those we can track solely in Google Analytics. An applicant enters the funnel, completes the application and submits the form. TA-DAH - success!
"Valid submissions" are those where the application is submitted AND the applicant is actually eligible for the scholarship. This will be a lower number and require input from the Team in Buffalo - or pulling from that COE API, if it exists.
Let me know if this makes sense to you!
10203 KPI | Metric | Source | Go/No Go |
---|---|---|---|
Successful submissions | 10203 successful submission total events vs 1995 successful submission total events | BigQuery | ✔️ Go |
Valid submissions | Total 10203 successful events + COE | BigQuery | 🔶 Future Go |
Note: We will need the COE data either from the team in Buffalo or through an API for the Valid Submissions metric.
Thanks for your work on this, @bmcgrady-ep ! @amycesal please review the current dashboard.
@amyknox let us know if you have any luck with the API!
@bmcgrady-ep Could you advise what the next steps are here towards closure is here? Thanks
@amyknox - I am closing out this issue. Feel free to reopen this issue, if you get access to the API, thanks!
Issue Description
Groom team KPIs for technical feasibility.
Tasks
Acceptance Criteria
KPIs in Epic.