Closed amyknox closed 4 years ago
Thank you @amyknox confirming receipt! Scheduled meeting for 2pm EST 7/8 to discuss.
edu-10203
<>
is used around a value below, this is to indicate dynamic value population and should change according to the appropriate value interacted with|
is used around a value below, this is to indicate any one (or logic) appropriate valuesStep in Form Process | Interaction | dataLayer Event |
---|---|---|
Form Starts | Form Start - Starting application without logging in | no-login-start-form |
Form Starts | Form Start - Call-to-action to Sign In to start form | login-link-clicked-cta-form |
Form Starts | Form Start - Begin form button after successful sign in | login-successful-start-form |
Form Saves and Submissions | Form Saves - Event is triggered for auto form saves | edu-10203-sip-form-saved |
Form Saves and Submissions | Form Prefill Failures - Tracks when form prefill cannot be retrieved | edu-10203-sip-form-prefill-failed |
Form Saves and Submissions | Form Submission - Measure the form submission; Success and failures are measured distinctly | edu-10203--submission |
Form Saves and Submissions | Form Submission Successful - Triggered when the form has been successfully submitted | edu-10203--submission-successful |
Form Saves and Submissions | Form Failures - Triggered when the form has failed client or server-side | edu-10203--submission-failed |
--submission-successful
and --submission-failed
events appropriatelydataLayer Variable | Description of Form Submission Variable | Screenshot |
---|---|---|
'edu-benefits-currently-used' |
Populates with a concatenated string seperated by a space for each of the checkboxes checked from /eligibility page |
|
'edu-currently-enrolled' |
Populates with 'Yes' or 'No' based on answer to "Are you enrolled in a science, technology, engineering, or math (STEM) undergraduate degree?" | |
'edu-benefits-remaining' |
Populates with answer to "About how much of your education benefit do you have left?" | |
'edu-name-of-degree' |
Populates with answer to "Whats the name of your stem degree" | |
'edu-stem-scholarship-school' |
Populates with answer to "Whats the name of the school where you plan on using the Rogers STEM Scholarship?" | |
'edu-school-city' |
Populates with answer to school city | |
'edu-school-state' |
Populates with answer to school state | |
'active-duty' |
Populates with answer to "Are you currently on active duty or will you be going on active duty while receiving the Rogers STEM Scholarship?" | |
'contact-method-preference' |
Populates with answer to "How should we contact you if we have questions about your application?" | |
'direct-deposit-account-type' |
Populates with answer to "Account type" |
Description of Form Interaction | Screenshot | dataLayer Event |
---|---|---|
Click to check multi-select checkbox | 'event': 'edu-form-change', 'edu-form-field': <question_text>', 'edu-form-value': <question_response_label> edu-form-action: 'checked'\|'unchecked'\|'clicked' |
Updated 7/30/2020 Please see comment here for revised DL spec for ineligibility alert page
@bsmartin-ep / @bmcgrady-ep
When one of you has capacity -- could you take a look at the drafted spec I have for 10203 above and let me know your thoughts.
I'll be starting on the 1995 adjustments here soon.
@amyknox / @dneel-bah
For 10203 -- We're working thru peer review of the dataLayer specification draft above. We'll let you know when we've got the draft finalized and ready for dev resources to start on implementation.
For 1995, couple prospective action items we would like to get your thoughts on:
1) We will want to remove those DL variables from the --submission-successful
and --submission-failed
events that no longer correlate with a question in the form (questions that are being migrated to 22-10203). We could use your assistance in verifying precisely which questions those are... From what I can see, some examples include: edu-stemApplicant
, edu-undergradStem
, edu-pursueTeaching
, edu-exhaustedAllBenefits
. Here's a screenshot sample of current --submission-successful
events for visibility:
2) We also could adopt / replace some of the new tracking paradigms we're proposing above in the 10203 spec in 1995, otherwise we could just remove the unnecessary variables with no other changes to 1995.
Hey @jonwehausen , It looks like you identified the correct DL variables. We are good with removing unnecessary variables or adding new tracking paradigm. This seems like a great time to add the new tracking, since we will be in there, but we're good either way.
@jonwehausen - The DL spec looks good to me. All of the correct variables look to be in the spec.
@shawkey-daniel / @amyknox Thank you guys for your patience on this. Per Daniel's latest comment, we would like to propose some adjustments to 1995 given the coming changes to questions with the introduction of 10203.
edu-1995--submission-successful
or edu-1995--submission-failed
|
indicate 1 of possible valuesundefined
for the given variableYes
or No
dataLayer Variable | Description of Form Submission Variable | Screenshot | Example Value |
---|---|---|---|
'benefits-used-recently' |
Value for "Which benefit are you currently using or have you used most recently?" | 'Yes'\|'No' |
|
'new-service-periods-to-record' |
Value for "Do you have any new periods of service to record since you last applied for education benefits?" | 'Yes'\|'No' |
|
'service-details' |
Array containing the details of service containing each branch name and pair of service start dates, and end dates given | 'service-details': [ {'service-branch': 'Air Force', 'service-start-date': '02/02/2020', 'service-end-date': '07/03/2020'}, {'service-branch': 'Navy', 'service-start-date': '05/14/1999', 'service-end-date': '05/14/2001'}] |
|
'service-before-1978' |
Value for "Do you have any periods of service that began before 1978? | 'Yes'\|'No' |
|
'edu-desired-facility-name' |
Value for "Name of school, university, or training facility" | 'harvard' |
|
'edu-desired-type-of-education' |
Value selected from dropdown for "Type of education or training" | 'Correspondence' |
|
'edu-desired-facility-state' |
Value selected from dropdown for state | 'Massachusetts' |
|
'edu-desired-facility-city' |
Value inputted for desired school city | 'Cambridge' |
|
'edu-prior-facility-name' |
Value inputted on /old-school for "Name of school, university, or training facility" |
'Yale' |
|
'edu-prior-facility-state' |
Value selected from dropdown for old school state | 'Connecticut' |
|
'edu-prior-facility-city' |
Value inputted for old school city | 'New Haven' |
|
'edu-prior-facility-end-date' |
Value inputted for date stopped taking classes | '04/09/2020' |
|
'preferred-contact-method' |
Value for "how would you like to be contacted..." | 'email' |
|
'married' |
Value for "Are you currently married" | 'Yes'\|'No' |
|
'dependent-children' |
Value for "Do you have children in the following categories...." | 'Yes'\|'No' |
|
'dependent-parent' |
Value for "Do you have a parent who is dependent for financial support?" | 'Yes'\|'No' |
|
'direct-deposit-method' |
Value for "Benefit payment method" | 'Unchanged'\|'Updated'\|'Stopped' |
|
'direct-deposit-account-type' |
Value for "Checking or Savings" | 'Checking'\|'Savings'\|undefined Please use undefined if user chose not to update |
@amyknox / @desireeturner / @shawkey-daniel / @dneel-bah
Please see the two comments above for the dataLayer spec for each form:
Please let us know if you have any questions. Happy to split these out into to separate Github issues as well.
@jonwehausen - We had to make some UI updates for 22-10203 so we need to update that tracking. I'll share notes with you.
@shawkey-daniel Is going to double check 22-1995 and get back with you. I think he had a question on sample values.
Our team typically creates our own stories off of your data layer requirements so no need to break this up.
@jonwehausen - Regarding 1995, 'benefits-used-recently' is set as 'Yes'|'No'. Shouldn't it be the selected value?
HI @jonwehausen - thanks for putting this all together! Here are notes on the 10203 Datalayer Specification you provided above
There's a conditional question that appears if the applicant says they're not pursuing a STEM degree currently. We'll want to track this. The question asks if they have a STEM degree and are pursuing a teaching certification. Answer is Y/N radio.
We had to rework the Alert / Explore other education benefits. It turns out you can't monkey with the Back\Continue buttons (we had changed "Continue" to "Exit application"). So now, we have a green "Exit application" button within the page and we've gone back to the standard Back | Continue form controls.
Tracking on what we initially designed
The update (and what the team will build) We'll need to retrofit the tracking outlined above to track this UI
For this, in addition to tracking whether this displays, is there a way to track whether individuals who see this alert abandon the form?
Hi @amyknox,
Thank you for this! Really appreciate you providing this additional context for the changes and areas I didn't cover, very helpful! Based on your notes here's what we would propose:
--submission-successful
and --submission-failed
eventsdataLayer Variable | Description of Form Submission Variable | Screenshot | Example Value |
---|---|---|---|
'edu-pursuing-teaching-certifcation' |
Value for "Are you pursuing a teaching certification" | 'Yes'\|'No' |
'event': 'edu-stem-scholarship-ineligibility-alert'
along with true|false
for each criteria met/unmetedu-stem-scholarship-ineligibility-alert
event should populate true
or false
according to the green checks (true
) and gray x's (false
)event |
Description of Interaction | Screenshot |
---|---|---|
'event': 'edu-stem-scholarship-ineligibility-alert', 'edu-eligibility-criteria-post911-met': true\|false, 'edu-eligibility-criteria-stem-or-teaching-met': true\|false, 'edu-eligibility-criteria-used-all-benefits-met': ':true\|false, 'edu-eligibilty-criteria-months-remaining-for-use': |
Display of the ineligibility alert | |
'event': 'cta-primary-button-click |
Click on "Exit Application" button |
@amyknox Please let us know if this all make sense and/or any other questions you may have. With tracking on the ineligibility alert display separate from the button, we'll be able to configure a funnel to understand form abandonment after receiving the alert, whether using the "Exit Application" button or not.
Hi @amyknox ,
Thank you for providing each of the ineligibility permutation combinations.
I updated my previous comment above with the latest spec that captures the criteria that was met/unmet at time of receiving the ineligibility alert.
@jonwehausen
The emails are accessible in GH Applicant confirmation: https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/education-careers/stem/stem-standalone-10203/content/stem-confirmation-email.md
Track link: "What happens after you apply"
SCO email (cc to applicant): https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/education-careers/stem/stem-standalone-10203/content/stem-SCO-email-20200804.md
Track links to
Dev Notes
edu-stem-scholarship-school
according to the school nametrue
or false
for cc-of-sco
if the email was or was not the email cc'ed to the applicant of the sco emailDescription of Interaction | DL Specification |
---|---|
All successful email sends upon successful form submission | 'event': 'edu-10203-email-confirmation-successful' , email-recipient: 'sco', //or 'applicant' 'edu-stem-scholarship-school': '<school_name>', 'cc-of-sco': true //or false |
All failed email sends upon successful form submission | 'event': 'edu-10203-email-confirmation-failed' , email-recipient: 'sco', //or 'applicant' 'edu-stem-scholarship-school': '<school_name>', 'cc-of-sco': true //or false |
href
link for the emailEmail link | Full UTM URL |
---|---|
Link for "What happens after you apply" | https://www.va.gov/education/after-you-apply/?utm_source=confirmation_email&utm_medium=email&utm_campaign=what_happens_after_you_apply |
Link for "Programs approved for STEM Scholarship" | https://www.va.gov/education/after-you-apply/?utm_source=confirmation_email&utm_medium=email&utm_campaign=stem_approved_programs |
@jonwehausen Approved Programs.. that isn't part of the "What happens after you apply" content.
Also, we still the syntax for the Privacy Policy... Should it be Link for "Privacy Policy" | https://www.va.gov/education/other-benefits/rogers-STEM-scholarship/?utm_source=confirmation_email&utm_medium=email&utm_campaign=privacy-policy |
---|
@jonwehausen The changes captured in the following comment are now available on staging: https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-664424693
Hi @amyknox ,
Thank you for the clarification there, that makes sense. The syntax UTM convention/syntax will remain similar for the newly introduced scholarship page. We would suggest simply modifying the utm_campaign
parameter (along with the new URL of course) when the new page URL has been set.
Your example for privacy policy looks good as well.
@jonwehausen - So we're crystal clear, can you confirm that in the SCO email, we should use:
Link for "Programs approved for STEM Scholarship" | https://benefits.va.gov/gibill/docs/fgib/STEM_Program_List.pdf?utm_source=confirmation_email&utm_medium=email&utm_campaign=stem_approved_programs |
---|
Confirmed with @mnorthuis that the Approved STEM Programs PDF is not migrating to VA.gov - 8/11
Updated Privacy policy link with new & improved & final STEM Application URL. Ideally we'll add an anchor to this link so users land directly at the Privacy Policy which is at the bottom of the page.
Link for "Privacy Policy" | https://www.va.gov/education/other-benefits/rogers-STEM-scholarship/?utm_source=confirmation_email&utm_medium=email&utm_campaign=privacy-policy |
---|
@jonwehausen The updates covered in the following comments have been made and are now available for review on staging: a. https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-662429290 b. https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-665338349
@bsmartin-ep Would you mind finishing the GTM implementation of 1995 and staging the implementation for 10203 according to the spec? Let me know of any questions. Thanks!
cc: @desireeturner @amyknox @joanneesteban @bsmartin-ep @bmcgrady-ep
Will post results of GTM testing in separate comment
Form Step / Interaction | dataLayer Screenshot | Test Status |
---|---|---|
Starting application without logging in | ✔️ PASS | |
Form Submission (fires on success and failure) | ✔️ PASS | |
Form Successful Submission | ✔️ PASS | |
Form Start - Begin form button after successful sign in | ✔️ PASS | |
Form Loaded | ✔️ PASS | |
Form Saved | ✔️ PASS | |
Form started new application | ✔️ PASS |
Description of Test Scenario | Test Case Screenshot | DL Screenshot | Test Status |
---|---|---|---|
Selection of Post-9/11 GI Bill (Chapter 33) | ✔️ PASS | ||
Selection of "Yes" to "Do you have any new periods of service to record since you last applied for education benefits?" | ✔️ PASS | ||
Adding services periods (mixed casing intentional for testing) | ✔️ PASS | ||
Selection of Yes to "Do you have any periods of service that began before 1978" | ✔️ PASS | ||
Adding details to "school, university, or training facility you want to attend" | ✔️ PASS | ||
Adding details to "school, university, or training facility you last attended" (mixed casing intentional for test purposes) | ✔️ PASS | ||
Answer to "how would you like to be contacted if we have questions about your application" | ✔️ PASS | ||
Answers to personal information questions | ✔️ PASS | ||
Direct deposit method details | ✔️ PASS |
cc: @desireeturner @joanneesteban @amyknox @bsmartin-ep @bmcgrady-ep will post results of GTM testing in seperate post
Description of Test Scenario | Test Case Screenshot | DL Screenshot | Test Status |
---|---|---|---|
Selections to "Which benefit have you used or are you currently using?" | ✔️ PASS | ||
Answers to Rogers Stem Scholarship eligibility questions | PASS >> will need to correct small "certification" typo | ||
Receiving ineligibility message | ✔️ PASS | ||
Answers to STEM degree and school details | ✔️ PASS | ||
Answer to "Are you currently on active duty or will you be going on active duty while receiving the Rogers STEM scholarship?" | ✔️ PASS | ||
Entering of direct deposit information | ✔️ PASS |
Description of Test Scenario | Test Case Screenshot | DL Screenshot | Test Status |
---|---|---|---|
Selecting of responses to "Which benefit have you used or are you currently using?" and clicking continue to next page | ✔️ PASS |
cc: @bsmartin-ep and @bmcgrady-ep for visibility
edu-eligibility-criteria-months-remaining-for-use
Typoedu-eligibility-criteria-months-remaining-for-use
cta-primary-button-click
on each of the "Exit Application" buttonscta-primary-button-click
on click for each of the "Exit Application" buttons on /education/other-va-education-benefits/stem-scholarship/apply-for-scholarship-form-22-10203/benefits/initial-confirm-eligibility
and /education/other-va-education-benefits/stem-scholarship/apply-for-scholarship-form-22-10203/benefits/confirm-eligibility
cta-primary-button-click
firing inadvertentlyThere looks to be a one-off cta-primary-button-click
event firing occassionally upon continuing from benefits/stem-eligibility
to /benefits/confirm-eligibility
and /benefits/eligibility
to /benefits/initial-confirm-eligibility
we will want to remove. The edu-stem-scholarship-ineligibility-alert
and edu-form-change
events are perfect here and we will be able to determine those who continue / move back using page URLs.
Screenshot 1 | Screenshot 2 |
---|---|
Would like to put explicit ineligibility alert tracking on the initial confirm page as well (/benefits/initial-confirm-eligibility
). Please use the following event
'event': edu-stem-scholarship-ineligibility-alert
@jonwehausen For tracking purposes, the new issues raised in comment https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-678442138 have been been put in a new story #12733 . This item can be closed if all other issues have been addressed.
fysa @bsmartin-ep @bmcgrady-ep ^
@jonwehausen @bsmartin-ep @bmcgrady-ep The typo in the following comment has been addressed and is available for review: https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-676864241
✔️ Pass
DataLayer Variable | DataLayer Test Value | GA Custom Dimension - Index & Name | GTM Form Submit Tag Config Screenshot |
---|---|---|---|
'preferred-contact-method' |
53: "Global - Preferred Contact Method" | ||
'benefits-used-recently' |
54: Global - Benefits Used Recently | ||
'edu-desired-facility-name' |
81: "Education - School Name" | ||
'edu-desired-facility-state' |
82: "Education - School State" | ||
'edu-desired-facility-city' |
83: "Education - School City" | ||
'edu-desired-type-of-education' |
59: "Education - Type of Education" | ||
'edu-prior-facility-name' |
94: "Education - Historical School Details" | ||
'edu-prior-facility-city' |
94: "Education - Historical School Details" | ||
'edu-prior-facility-state' |
94: "Education - Historical School Details" | ||
'edu-prior-facility-end-date' |
94: "Education - Historical School Details" | ||
'dependent-children' |
95: "Global - Dependent Children" | ||
'dependent-parent' |
96: "Global - Dependent Parent" | ||
'married' |
97: "Global - Married" | ||
'direct-deposit-method' |
98: "Global - Direct Deposit Change Method" | ||
'direct-deposit-account-type' |
99: "Global - Direct Deposit Account Type" | ||
'service-before-1978' |
100: "Education - Service Prior to 1978" | ||
'service-branch' |
101: "Global - Service Branch(s) | ||
'service-start-date' |
102: "Global - Service Start Dates" | ||
'service-end-date' |
103: "Global - Service End Dates" | ||
'new-service-periods-to-record' |
n/a | no need for CD...data from new service records are contained within CD 94 |
✔️ PASS
DataLayer Variable | DataLayer Test Value | GA Custom Dimension - Index & Name | GTM Form Submit Tag Config Screenshot |
---|---|---|---|
'active-duty' |
42: "Global - Currently Active Duty" | ||
'edu-benefits-currently-used' |
54: "Global - Benefits Used Recently" | ||
'edu-currently-enrolled' |
55: "Education - Enrolled in Undergrad STEM" | ||
'edu-pursuing-teaching-certification' |
56: "Education - Pursuing Teaching Certification" | ||
'edu-benefits-remaining' |
105: "Education - Benefits Remaining" | ||
'edu-name-of-degree' |
104: "Education - Degree Name / Field of Study" | ||
'edu-stem-scholarship-school' |
81: "Education - School Name" | ||
'edu-school-state' |
82: "Education - School State" | ||
'edu-school-city' |
83: "Education - School City" | ||
'contact-method-preference' |
53: "Global - Preferred Contact Method" | ||
'direct-deposit-account-type' |
99: "Global - Direct Deposit Account Type" |
Radio Button Screenshot | DataLayer Screenshot | GA Custom Metric - Index & Name | GTM Form Submit Tag Config Screenshot |
---|---|---|---|
- 28 = STEM Scholarship Criteria Met - Post 911 - 29 = STEM Scholarship Criteria Met - STEM or Teaching - 30 = STEM Scholarship Criteria Met - Benefits Remaining |
Interaction Screenshot | DataLayer Screenshot | GTM Form Submit Tag Config Screenshot |
---|---|---|
cc: @desireeturner @amyknox @dneel-bah @joanneesteban @bsmartin-ep @bmcgrady-ep
@jonwehausen The comments included above (https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-678442138) have now been addressed in #12733 and are ready for your review.
Description of Interaction | DL Screenshot | Tag Screenshot | Test Status |
---|---|---|---|
Removal of cta-primary-button-click on benefits/stem-eligibility to /benefits/confirm-eligibility |
N/A | N/A | ✔️ PASS |
Initial ineligibility alert received | ✔️ PASS | ||
Click to 'Exit application' on initial ineligibility alert | ✔️ PASS | ||
Receive 2nd ineligibility alert | ✔️ PASS | ||
Click on 'Exit Application' upon receiving the 2nd ineligibility alert | ✔️ PASS |
@amyknox @bsmartin-ep @desireeturner
Closing issue. Charts on Education Rollout mockup have been updated to include the initial ineligibility steps
Google Analytics Implementation
What this form is for
For more information, please check out the Google Tag Manager resources.
An example of a completed request template can be found here. A fully completed issue can help speed up the implementation process.
Description
Google Analytics
Who is this request for?
Provide any relevant contact information and link this issue to the insights team on the DSVA Slack Channel:
#VFS-Platform-Support
.What team/project is this work for?
Provide the Domain where Google Analytics needs to be implemented i.e. www.staging.va.gov
22-1995 on Production https://www.va.gov/education/apply-for-education-benefits/application/1995/introduction 22-10203 on Staging: https://staging.va.gov/education/apply-for-education-benefits/application/10203/introduction
Please provide any upcoming deadlines or important milestones ex: launch
UPDATED Production Roll-out: 9/10
Typical Website Interactions to Track
Conversions - Understanding the User Flow: Provide URLs that are part of the application from introduction page to successful form completion; conditional pages that are visible depending on an applicant's answer should be marker accordingly.
22-10203 inVision Prototype (WIP) https://bahdigital.invisionapp.com/share/BQIAE8O42M8
[URLs here]
Other User Interactions: Provide details on other interactions and metrics, like click to help text links, that are helpful in understanding user behavior.
Where are applicants coming from (email, social, va.gov/apply?
Authenticated vs. unauthenticated applicants
For authenticated applicants, are they authenticated when they arrive at the intro page or do they authenticate there
Successful application submissions
Which ineligibility alerts do applicants see?
What actions do they take at the ineligibility alert: click "explore other benefits" button; click "no" they don't want to continue application; click "yes" they want to continue the application?
Sample Users when authentication is required do not put staging credentials in your
va.gov-team
ticket; store or reference them in a .md file in theva.gov-team-sensitive
repositoryTest Users to replicate the event do not put staging credentials in your
va.gov-team
ticket; store or reference them in a .md file in theva.gov-team-sensitive
repositoryCertain conditions/responses that need to be reviewed
Additional Comments
Acceptance Criteria
Definition of Done