department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
282 stars 203 forks source link

Analytics Implementation for 22-10203 / 22-1995 #10780

Closed amyknox closed 4 years ago

amyknox commented 4 years ago

Google Analytics Implementation

What this form is for

Please see the following information for implementation timeline: All other teams: Implementation will usually be completed in 2 sprints

  • Sprint 1: Discovery and FE Implementation
  • Sprint 2: Tagging Implementation and QA

For more information, please check out the Google Tag Manager resources.

An example of a completed request template can be found here. A fully completed issue can help speed up the implementation process.

Description

Google Analytics

22-1995 on Production https://www.va.gov/education/apply-for-education-benefits/application/1995/introduction 22-10203 on Staging: https://staging.va.gov/education/apply-for-education-benefits/application/10203/introduction

_Identify if PII is passed in URL or title tag_

- No PII is passed in URL or title tags

Please leave the following blank

Acceptance Criteria

Definition of Done

jonwehausen commented 4 years ago

Thank you @amyknox confirming receipt! Scheduled meeting for 2pm EST 7/8 to discuss.

jonwehausen commented 4 years ago

Datalayer Specification - 10203

General Developer Notes:

Standard Platform Form Tracking

Step in Form Process Interaction dataLayer Event
Form Starts Form Start - Starting application without logging in no-login-start-form
Form Starts Form Start - Call-to-action to Sign In to start form login-link-clicked-cta-form
Form Starts Form Start - Begin form button after successful sign in login-successful-start-form
Form Saves and Submissions Form Saves - Event is triggered for auto form saves edu-10203-sip-form-saved
Form Saves and Submissions Form Prefill Failures - Tracks when form prefill cannot be retrieved edu-10203-sip-form-prefill-failed
Form Saves and Submissions Form Submission - Measure the form submission; Success and failures are measured distinctly edu-10203--submission
Form Saves and Submissions Form Submission Successful - Triggered when the form has been successfully submitted edu-10203--submission-successful
Form Saves and Submissions Form Failures - Triggered when the form has failed client or server-side edu-10203--submission-failed

Successful / Failed Submission Additional Context

dataLayer Variable Description of Form Submission Variable Screenshot
'edu-benefits-currently-used' Populates with a concatenated string seperated by a space for each of the checkboxes checked from /eligibility page image
'edu-currently-enrolled' Populates with 'Yes' or 'No' based on answer to "Are you enrolled in a science, technology, engineering, or math (STEM) undergraduate degree?" image
'edu-benefits-remaining' Populates with answer to "About how much of your education benefit do you have left?" image
'edu-name-of-degree' Populates with answer to "Whats the name of your stem degree" image
'edu-stem-scholarship-school' Populates with answer to "Whats the name of the school where you plan on using the Rogers STEM Scholarship?" image
'edu-school-city' Populates with answer to school city image
'edu-school-state' Populates with answer to school state image
'active-duty' Populates with answer to "Are you currently on active duty or will you be going on active duty while receiving the Rogers STEM Scholarship?" image
'contact-method-preference' Populates with answer to "How should we contact you if we have questions about your application?" image
'direct-deposit-account-type' Populates with answer to "Account type" image

Form Component Interaction

Description of Form Interaction Screenshot dataLayer Event
Click to check multi-select checkbox image 'event': 'edu-form-change',
'edu-form-field': <question_text>',
'edu-form-value': <question_response_label>
edu-form-action: 'checked'\|'unchecked'\|'clicked'

Updated 7/30/2020 Please see comment here for revised DL spec for ineligibility alert page

jonwehausen commented 4 years ago

@bsmartin-ep / @bmcgrady-ep

When one of you has capacity -- could you take a look at the drafted spec I have for 10203 above and let me know your thoughts.

I'll be starting on the 1995 adjustments here soon.

jonwehausen commented 4 years ago

@amyknox / @dneel-bah

For 10203 -- We're working thru peer review of the dataLayer specification draft above. We'll let you know when we've got the draft finalized and ready for dev resources to start on implementation.

For 1995, couple prospective action items we would like to get your thoughts on: 1) We will want to remove those DL variables from the --submission-successful and --submission-failed events that no longer correlate with a question in the form (questions that are being migrated to 22-10203). We could use your assistance in verifying precisely which questions those are... From what I can see, some examples include: edu-stemApplicant, edu-undergradStem, edu-pursueTeaching, edu-exhaustedAllBenefits. Here's a screenshot sample of current --submission-successful events for visibility: image

2) We also could adopt / replace some of the new tracking paradigms we're proposing above in the 10203 spec in 1995, otherwise we could just remove the unnecessary variables with no other changes to 1995.

shawkey-daniel commented 4 years ago

Hey @jonwehausen , It looks like you identified the correct DL variables. We are good with removing unnecessary variables or adding new tracking paradigm. This seems like a great time to add the new tracking, since we will be in there, but we're good either way.

bmcgrady-ep commented 4 years ago

@jonwehausen - The DL spec looks good to me. All of the correct variables look to be in the spec.

jonwehausen commented 4 years ago

Datalayer Specification - 1995 - Adjustments

@shawkey-daniel / @amyknox Thank you guys for your patience on this. Per Daniel's latest comment, we would like to propose some adjustments to 1995 given the coming changes to questions with the introduction of 10203.

Developer Notes

dataLayer Variable Description of Form Submission Variable Screenshot Example Value
'benefits-used-recently' Value for "Which benefit are you currently using or have you used most recently?" image 'Yes'\|'No'
'new-service-periods-to-record' Value for "Do you have any new periods of service to record since you last applied for education benefits?" image 'Yes'\|'No'
'service-details' Array containing the details of service containing each branch name and pair of service start dates, and end dates given image 'service-details': [
{'service-branch': 'Air Force', 'service-start-date': '02/02/2020', 'service-end-date': '07/03/2020'},
{'service-branch': 'Navy', 'service-start-date': '05/14/1999', 'service-end-date': '05/14/2001'}]
'service-before-1978' Value for "Do you have any periods of service that began before 1978? image 'Yes'\|'No'
'edu-desired-facility-name' Value for "Name of school, university, or training facility" image 'harvard'
'edu-desired-type-of-education' Value selected from dropdown for "Type of education or training" image 'Correspondence'
'edu-desired-facility-state' Value selected from dropdown for state image 'Massachusetts'
'edu-desired-facility-city' Value inputted for desired school city image 'Cambridge'
'edu-prior-facility-name' Value inputted on /old-school for "Name of school, university, or training facility" image 'Yale'
'edu-prior-facility-state' Value selected from dropdown for old school state image 'Connecticut'
'edu-prior-facility-city' Value inputted for old school city image 'New Haven'
'edu-prior-facility-end-date' Value inputted for date stopped taking classes image '04/09/2020'
'preferred-contact-method' Value for "how would you like to be contacted..." image 'email'
'married' Value for "Are you currently married" image 'Yes'\|'No'
'dependent-children' Value for "Do you have children in the following categories...." image 'Yes'\|'No'
'dependent-parent' Value for "Do you have a parent who is dependent for financial support?" image 'Yes'\|'No'
'direct-deposit-method' Value for "Benefit payment method" image 'Unchanged'\|'Updated'\|'Stopped'
'direct-deposit-account-type' Value for "Checking or Savings" image 'Checking'\|'Savings'\|undefined

Please use undefined if user chose not to update
jonwehausen commented 4 years ago

@amyknox / @desireeturner / @shawkey-daniel / @dneel-bah

Please see the two comments above for the dataLayer spec for each form:

Please let us know if you have any questions. Happy to split these out into to separate Github issues as well.

amyknox commented 4 years ago

@jonwehausen - We had to make some UI updates for 22-10203 so we need to update that tracking. I'll share notes with you.

@shawkey-daniel Is going to double check 22-1995 and get back with you. I think he had a question on sample values.

Our team typically creates our own stories off of your data layer requirements so no need to break this up.

shawkey-daniel commented 4 years ago

@jonwehausen - Regarding 1995, 'benefits-used-recently' is set as 'Yes'|'No'. Shouldn't it be the selected value?

amyknox commented 4 years ago

HI @jonwehausen - thanks for putting this all together! Here are notes on the 10203 Datalayer Specification you provided above

Pursuing a teaching certification

There's a conditional question that appears if the applicant says they're not pursuing a STEM degree currently. We'll want to track this. The question asks if they have a STEM degree and are pursuing a teaching certification. Answer is Y/N radio.

image

Explore other education benefits

We had to rework the Alert / Explore other education benefits. It turns out you can't monkey with the Back\Continue buttons (we had changed "Continue" to "Exit application"). So now, we have a green "Exit application" button within the page and we've gone back to the standard Back | Continue form controls.

Tracking on what we initially designed image

The update (and what the team will build) We'll need to retrofit the tracking outlined above to track this UI image

Ineligibility alert

For this, in addition to tracking whether this displays, is there a way to track whether individuals who see this alert abandon the form?

jonwehausen commented 4 years ago

Hi @amyknox,

Thank you for this! Really appreciate you providing this additional context for the changes and areas I didn't cover, very helpful! Based on your notes here's what we would propose:

Pursuing a teaching certification

dataLayer Variable Description of Form Submission Variable Screenshot Example Value
'edu-pursuing-teaching-certifcation' Value for "Are you pursuing a teaching certification" image 'Yes'\|'No'

Ineligibility & Exit Application Tracking - Updated 7/30/2020

event Description of Interaction Screenshot
'event': 'edu-stem-scholarship-ineligibility-alert',
'edu-eligibility-criteria-post911-met': true\|false,
'edu-eligibility-criteria-stem-or-teaching-met': true\|false,
'edu-eligibility-criteria-used-all-benefits-met': ':true\|false,
'edu-eligibilty-criteria-months-remaining-for-use':
Display of the ineligibility alert image
'event': 'cta-primary-button-click Click on "Exit Application" button image

@amyknox Please let us know if this all make sense and/or any other questions you may have. With tracking on the ineligibility alert display separate from the button, we'll be able to configure a funnel to understand form abandonment after receiving the alert, whether using the "Exit Application" button or not.

jonwehausen commented 4 years ago

Hi @amyknox ,

Thank you for providing each of the ineligibility permutation combinations.

I updated my previous comment above with the latest spec that captures the criteria that was met/unmet at time of receiving the ineligibility alert.

jonwehausen commented 4 years ago

Next steps

amyknox commented 4 years ago

@jonwehausen

The emails are accessible in GH Applicant confirmation: https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/education-careers/stem/stem-standalone-10203/content/stem-confirmation-email.md

Track link: "What happens after you apply"

SCO email (cc to applicant): https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/education-careers/stem/stem-standalone-10203/content/stem-SCO-email-20200804.md

Track links to

jonwehausen commented 4 years ago

Datalayer Specification for Emailing Confirmation

Dev Notes

Description of Interaction DL Specification
All successful email sends upon successful form submission 'event': 'edu-10203-email-confirmation-successful',
email-recipient: 'sco', //or 'applicant'
'edu-stem-scholarship-school': '<school_name>',
'cc-of-sco': true //or false
All failed email sends upon successful form submission 'event': 'edu-10203-email-confirmation-failed',
email-recipient: 'sco', //or 'applicant'
'edu-stem-scholarship-school': '<school_name>',
'cc-of-sco': true //or false
jonwehausen commented 4 years ago

Tracking for Inbound Clicks within Emails

UTMs in a nutshell

Email link Full UTM URL
Link for "What happens after you apply" https://www.va.gov/education/after-you-apply/?utm_source=confirmation_email&utm_medium=email&utm_campaign=what_happens_after_you_apply
Link for "Programs approved for STEM Scholarship" https://www.va.gov/education/after-you-apply/?utm_source=confirmation_email&utm_medium=email&utm_campaign=stem_approved_programs
amyknox commented 4 years ago

@jonwehausen Approved Programs.. that isn't part of the "What happens after you apply" content.

Also, we still the syntax for the Privacy Policy... Should it be Link for "Privacy Policy" https://www.va.gov/education/other-benefits/rogers-STEM-scholarship/?utm_source=confirmation_email&utm_medium=email&utm_campaign=privacy-policy
desireeturner commented 4 years ago

@jonwehausen The changes captured in the following comment are now available on staging: https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-664424693

jonwehausen commented 4 years ago

Hi @amyknox ,

Thank you for the clarification there, that makes sense. The syntax UTM convention/syntax will remain similar for the newly introduced scholarship page. We would suggest simply modifying the utm_campaign parameter (along with the new URL of course) when the new page URL has been set.

Your example for privacy policy looks good as well.

amyknox commented 4 years ago

@jonwehausen - So we're crystal clear, can you confirm that in the SCO email, we should use:

Link for "Programs approved for STEM Scholarship" https://benefits.va.gov/gibill/docs/fgib/STEM_Program_List.pdf?utm_source=confirmation_email&utm_medium=email&utm_campaign=stem_approved_programs

Confirmed with @mnorthuis that the Approved STEM Programs PDF is not migrating to VA.gov - 8/11

amyknox commented 4 years ago

Updated Privacy policy link with new & improved & final STEM Application URL. Ideally we'll add an anchor to this link so users land directly at the Privacy Policy which is at the bottom of the page.

Link for "Privacy Policy" https://www.va.gov/education/other-benefits/rogers-STEM-scholarship/?utm_source=confirmation_email&utm_medium=email&utm_campaign=privacy-policy
desireeturner commented 4 years ago

@jonwehausen The updates covered in the following comments have been made and are now available for review on staging: a. https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-662429290 b. https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-665338349

jonwehausen commented 4 years ago

@bsmartin-ep Would you mind finishing the GTM implementation of 1995 and staging the implementation for 10203 according to the spec? Let me know of any questions. Thanks!

jonwehausen commented 4 years ago

QA Testing of 1995

cc: @desireeturner @amyknox @joanneesteban @bsmartin-ep @bmcgrady-ep

Will post results of GTM testing in separate comment

Standard Form Events

Form Step / Interaction dataLayer Screenshot Test Status
Starting application without logging in image ✔️ PASS
Form Submission (fires on success and failure) image ✔️ PASS
Form Successful Submission image ✔️ PASS
Form Start - Begin form button after successful sign in image ✔️ PASS
Form Loaded image ✔️ PASS
Form Saved image ✔️ PASS
Form started new application image ✔️ PASS

Test Scenarios of Datalayer Metadata

Description of Test Scenario Test Case Screenshot DL Screenshot Test Status
Selection of Post-9/11 GI Bill (Chapter 33) image image ✔️ PASS
Selection of "Yes" to "Do you have any new periods of service to record since you last applied for education benefits?" image image ✔️ PASS
Adding services periods (mixed casing intentional for testing) image image ✔️ PASS
Selection of Yes to "Do you have any periods of service that began before 1978" image image ✔️ PASS
Adding details to "school, university, or training facility you want to attend" image image ✔️ PASS
Adding details to "school, university, or training facility you last attended" (mixed casing intentional for test purposes) image image ✔️ PASS
Answer to "how would you like to be contacted if we have questions about your application" image image ✔️ PASS
Answers to personal information questions image image ✔️ PASS
Direct deposit method details image image ✔️ PASS
jonwehausen commented 4 years ago

QA Testing of 10203

cc: @desireeturner @joanneesteban @amyknox @bsmartin-ep @bmcgrady-ep will post results of GTM testing in seperate post

Testing of Datalayer Metadata

Description of Test Scenario Test Case Screenshot DL Screenshot Test Status
Selections to "Which benefit have you used or are you currently using?" image image ✔️ PASS
Answers to Rogers Stem Scholarship eligibility questions image image PASS >> will need to correct small "certification" typo
Receiving ineligibility message image image ✔️ PASS
Answers to STEM degree and school details image image ✔️ PASS
Answer to "Are you currently on active duty or will you be going on active duty while receiving the Rogers STEM scholarship?" image image ✔️ PASS
Entering of direct deposit information image image ✔️ PASS

Form Changes

Description of Test Scenario Test Case Screenshot DL Screenshot Test Status
Selecting of responses to "Which benefit have you used or are you currently using?" and clicking continue to next page image image image image ✔️ PASS
jonwehausen commented 4 years ago

Follow Up Items to Correct Following Secondary QA

cc: @bsmartin-ep and @bmcgrady-ep for visibility


1) edu-eligibility-criteria-months-remaining-for-use Typo

Typo (missing second "i" for edu-eligibility-criteria-months-remaining-for-use image

2) Addition of cta-primary-button-click on each of the "Exit Application" buttons

The addition of event cta-primary-button-click on click for each of the "Exit Application" buttons on /education/other-va-education-benefits/stem-scholarship/apply-for-scholarship-form-22-10203/benefits/initial-confirm-eligibility and /education/other-va-education-benefits/stem-scholarship/apply-for-scholarship-form-22-10203/benefits/confirm-eligibility image

3) Removal of cta-primary-button-click firing inadvertently

There looks to be a one-off cta-primary-button-click event firing occassionally upon continuing from benefits/stem-eligibility to /benefits/confirm-eligibility and /benefits/eligibility to /benefits/initial-confirm-eligibility we will want to remove. The edu-stem-scholarship-ineligibility-alert and edu-form-change events are perfect here and we will be able to determine those who continue / move back using page URLs.

Screenshot 1 Screenshot 2
image image

4) Ineligibility Alert Tracking On Initial Confirm

Would like to put explicit ineligibility alert tracking on the initial confirm page as well (/benefits/initial-confirm-eligibility). Please use the following event

   'event': edu-stem-scholarship-ineligibility-alert
dneel-bah commented 4 years ago

@jonwehausen For tracking purposes, the new issues raised in comment https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-678442138 have been been put in a new story #12733 . This item can be closed if all other issues have been addressed.

joanneesteban commented 4 years ago

fysa @bsmartin-ep @bmcgrady-ep ^

desireeturner commented 4 years ago

@jonwehausen @bsmartin-ep @bmcgrady-ep The typo in the following comment has been addressed and is available for review: https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-676864241

jonwehausen commented 4 years ago

GTM Tag Management QA 1995

✔️ Pass

Testing Notes

Custom Dimension Values (1995 customization)

DataLayer Variable DataLayer Test Value GA Custom Dimension - Index & Name GTM Form Submit Tag Config Screenshot
'preferred-contact-method' image 53: "Global - Preferred Contact Method" image
'benefits-used-recently' image 54: Global - Benefits Used Recently image
'edu-desired-facility-name' image 81: "Education - School Name" image
'edu-desired-facility-state' image 82: "Education - School State" image
'edu-desired-facility-city' image 83: "Education - School City" image
'edu-desired-type-of-education' image 59: "Education - Type of Education" image
'edu-prior-facility-name' image 94: "Education - Historical School Details" image
'edu-prior-facility-city' image 94: "Education - Historical School Details" image
'edu-prior-facility-state' image 94: "Education - Historical School Details" image
'edu-prior-facility-end-date' image 94: "Education - Historical School Details" image
'dependent-children' image 95: "Global - Dependent Children" image
'dependent-parent' image 96: "Global - Dependent Parent" image
'married' image 97: "Global - Married" image
'direct-deposit-method' image 98: "Global - Direct Deposit Change Method" image
'direct-deposit-account-type' image 99: "Global - Direct Deposit Account Type" image
'service-before-1978' image 100: "Education - Service Prior to 1978" image
'service-branch' image 101: "Global - Service Branch(s) image
'service-start-date' image 102: "Global - Service Start Dates" image
'service-end-date' image 103: "Global - Service End Dates" image
'new-service-periods-to-record' n/a no need for CD...data from new service records are contained within CD 94
jonwehausen commented 4 years ago

GTM Tag Management QA 10203

✔️ PASS

Custom Dimension Value (10203 customization)

DataLayer Variable DataLayer Test Value GA Custom Dimension - Index & Name GTM Form Submit Tag Config Screenshot
'active-duty' image 42: "Global - Currently Active Duty" image
'edu-benefits-currently-used' image 54: "Global - Benefits Used Recently" image
'edu-currently-enrolled' image 55: "Education - Enrolled in Undergrad STEM" image
'edu-pursuing-teaching-certification' image 56: "Education - Pursuing Teaching Certification" image
'edu-benefits-remaining' image 105: "Education - Benefits Remaining" image
'edu-name-of-degree' image 104: "Education - Degree Name / Field of Study" image
'edu-stem-scholarship-school' image 81: "Education - School Name" image
'edu-school-state' image 82: "Education - School State" image
'edu-school-city' image 83: "Education - School City" image
'contact-method-preference' image 53: "Global - Preferred Contact Method" image
'direct-deposit-account-type' image 99: "Global - Direct Deposit Account Type" image

Product Ineligibility Alerts & Criteria

Radio Button Screenshot DataLayer Screenshot GA Custom Metric - Index & Name GTM Form Submit Tag Config Screenshot
image image - 28 = STEM Scholarship Criteria Met - Post 911

- 29 = STEM Scholarship Criteria Met - STEM or Teaching

- 30 = STEM Scholarship Criteria Met - Benefits Remaining
image

Education Form Change

Interaction Screenshot DataLayer Screenshot GTM Form Submit Tag Config Screenshot
image image image


image
jonwehausen commented 4 years ago

Update

cc: @desireeturner @amyknox @dneel-bah @joanneesteban @bsmartin-ep @bmcgrady-ep

Next Steps

desireeturner commented 4 years ago

@jonwehausen The comments included above (https://github.com/department-of-veterans-affairs/va.gov-team/issues/10780#issuecomment-678442138) have now been addressed in #12733 and are ready for your review.

jonwehausen commented 4 years ago

QA Testing

Description of Interaction DL Screenshot Tag Screenshot Test Status
Removal of cta-primary-button-click on benefits/stem-eligibility to /benefits/confirm-eligibility N/A N/A ✔️ PASS
Initial ineligibility alert received image image

image
✔️ PASS
Click to 'Exit application' on initial ineligibility alert image image

image
✔️ PASS
Receive 2nd ineligibility alert image image ✔️ PASS
Click on 'Exit Application' upon receiving the 2nd ineligibility alert image image

image
✔️ PASS
jonwehausen commented 4 years ago

@amyknox @bsmartin-ep @desireeturner

Closing issue. Charts on Education Rollout mockup have been updated to include the initial ineligibility steps