Analytics Implementation - STEM Automated Denial

[amyknox]

Google Analytics Implementation 22-10203: Rogers STEM Scholarship - Automated Denial

Description

• Is this request for Google Analytics implementation or QA?

  • Google Analytics Implementation

• Who is this request for?

  • Booz Allen Team: @amyknox @dneel-bah

• What team/project is this work for?

  • Booz Allen / VA Education Service, 22-10203 STEM Automated Decision

• Provide the Domain where Google Analytics needs to be implemented

  • www.staging.va.gov -> www.va.gov

  • No PII is passed in URL or title tag

• Please provide any upcoming deadlines or important milestones

  • Limited production rollout to begin 3/19/21

• Typical Website Interactions to Track

  • Conversions - Understanding the User Flow:

  • Denied applicant is sent an email with URL directing back to Claims Status Tool: https://staging.va.gov/track-claims/your-stem-claims/8468/status (Image 1)

  • Claims tool has card with Denied status & View details button (Image 2)

  • Status details page has information on reason for denial and various links for more info and support (Image 3)

Image 1

Image 2

Image 3

• Other User Interactions: Details on other interactions and metrics, like click to help text links, that are helpful in understanding user behavior.

  • How many emails are delivered
  • How many / % that click through & arrive on the Claims Tool landing page
  • How many authenticate
  • How many click through on View details button
  • Clicks on Appeal/Review forms, More resources, Right sidebar

• Sample Users when authentication is required do not put staging credentials in your va.gov-team ticket; store or reference them in a .md file in the va.gov-team-sensitive repository

  • Test user table here. Test user 220 has several denied applications

• Test Users to replicate the event do not put staging credentials in your va.gov-team ticket; store or reference them in a .md file in the va.gov-team-sensitive repository

  • Same as above

• Certain conditions/responses that need to be reviewed

  • [Provide conditions/responses here]

• Additional Comments

Please leave the following blank

Acceptance Criteria

• [ ] QA has been completed
• [ ] Events have been added to Product and Benefit Hub Content Groups
• [ ] VFS team completes brief VSP Collaboration Cycle Feedback survey

Definition of Done

• [ ] All appropriate issue tagging is completed
• [ ] All AC completed
• [ ] VSP: Platform Collaboration Point Tracker is updated

[jonwehausen]

Hi @amyknox ,

Thank you very much for your request, we appreciate your thoroughly completing our request template and providing screenshots.

For initial context, I wasn't able to access any existing denied application claims using user+220 so the items you see below were based mostly on your screenshots. I did also try a few other test users with no luck, if you could provide a couple more than allows us to see the full experience we would greatly appreciate it.

Please find the data layer specification for the items you identified below:

Data Layer Specification

Description of Interaction + Tracking Component	Data Layer Specification	Dev Notes
Clicks on right "Need Help" sidebar links	'event': 'nav-ask-va-questions-link-click', 'ask-va-questions-header': 'Need help'
Appeal / Higher Level Review form downloads	N/A	• Assuming these are PDF downloads, we have this globally tracked and no data layer implementation needed
Clicks on links within "More Resources about VA benefits" links	'event': nav-linkslist' 'links-list-section-header': 'More resources about VA benefits' //dynamically populate with value of links list section 'links-list-header': //dynamically populate with text of header clicked, i.e 'Edith Nurse Rogers STEM Scholarship', 'Find a VA Form', 'GI Bill Comparison Tool'
Click throughs on View details action link	'event': 'cta-action-link-click', 'action-link-type': 'primary', 'secondary', 'reverse' //name of action link button used from css class 'action-link-click-label': //click label for action link 'action-link-icon-color': //name of color from class, i.e 'white'

Remaining Tracking Components

Description of Metric	Questions/Recommendation/Notes
How many emails are delivered	• How soon after application completion are claims status emails typically sent? • Is it completely up to govdelivery for when that email goes out?
How many / % that click through & arrive on the Claims Tool landing page	To capture this most optimally, we would recommend to embed UTM parameters within the URL within the emails linking back to the claims tool landing page -- a full URL would look like: https://www.va.gov/track-claims/your-stem-claims/8468/status?utm_source=govdelivery&utm_medium=email&utm_campaign=stem_scholarship_decision You could also condense that entire UTM-contained URL in a URL shortener if desired

[jonwehausen]

Shifting this ticket to Stand By while data layer is implemented

Next Steps for me

• Add event taxonomy for nav-sidebar-right-rail-click in GTM

cc: @laurensimpson14 @bsmartin-ep @joanneesteban

[amyknox]

Jon – double check user 220. There are multiple denied STEM claims for that user.

Link to claims status tool in staging: https://staging.va.gov/track-claims/your-stem-claims/8512/status

You can find log-in info for user 220 here: https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/blob/master/Administrative/vagov-users/mvi-staging-users.csv

Amy Knox

[amyknox]

Just verified 220 has denied STEM applications. Yep - 4 of them.

[jonwehausen]

Hi @amyknox ,

Thanks for your confirmation here. I can confirm I see user+200 in fact has denied claims for testing.

I can also confirm our global tracking on PDF download is in-fact capturing here. You can see examples of these events within our non-prod property here.

We will be on standby for the remainder of the DL to be implemented.

[dneel-bah]

@jonwehausen Were you also able to confirm that the global button listener picked up the items mentioned for "Click throughs on View details button"? Or will we need to add that tracking after all?

[amyknox]

@jonwehausen - double checking the requirements. The right sidebar contents may be moved to the bottom of the page in the not too distant future. My understanding is there's a motion afoot to deprecate the right sidebar across the entire site. Curious to get your thoughts on how to best code those links so any shifts in usage can be tracked.

Does it make sense to maintain the 'nav-sidebar-right-rail-click' implementation or should we use something more generic?

[jonwehausen]

Update

@dneel-bah

• Currently unable to validate our globalized button tracking with backend appearing to be down

That said, based on what I see here it looks as though these are action links which would not be captured by our button analytics. Were these in fact converted to action links over buttons?

---

@amyknox ,

Thanks for bringing this to my attention -- and yes I agree we should make the taxonomy for generic with that information.

Since the sidebar may get moved to the bottom of the page, and from what I can see on the vets-website side, this looks to be an extension of <AskVAQuestions -- let's refactor the event taxonomy to be

  'event': 'nav-ask-va-questions-link-click', 
  'ask-va-questions-header': 'Need help'

[dneel-bah]

Update

@dneel-bah

• Currently unable to validate our globalized button tracking with backend appearing to be down

That said, based on what I see here it looks as though these are action links which would not be captured by our button analytics. Were these in fact converted to action links over buttons?

@amyknox ,

Thanks for bringing this to my attention -- and yes I agree we should make the taxonomy for generic with that information.

Since the sidebar may get moved to the bottom of the page, and from what I can see on the vets-website side, this looks to be an extension of <AskVAQuestions -- let's refactor the event taxonomy to be

  'event': 'nav-ask-va-questions-link-click', 
  'ask-va-questions-header': 'Need help'

@jonwehausen

Yes, the buttons were converted to action links based on feedback from one of the DEPO checkpoints. Since the links aren't handled by the global handler, should we use the event posted above, or are tweaks necessary?

[jonwehausen]

Hi @dneel-bah ,

Thanks so much for this update.

Per the change from buttons to action links, we would like to pivot to the following data layer implementation:

'event': 'cta-action-link-click', 
'action-link-type': 'primary', // or 'secondary', 'reverse' --- i.e click class of 'vads-c-action-link--blue' = 'secondary' ... based on name of action link button used from css class
'action-link-click-label': //click label for action link
'action-link-icon-color': //name of color from class, i.e 'white'

I have updated both of our data layer taxonomy adjustment within the original DL specification comment as well

cc: @bsmartin-ep As he is starting the work to standardize action links over the coming days

[jonwehausen]

@dneel-bah

Per my last note -- your team can hold off implementing this specification as @bsmartin-ep is going to do the work to catch these site-wide, so as not to risk double firing of events, your team can hold off and we can validate when Brian has completed the global listener work this sprint.

[sonntag-adam]

@jonwehausen @joanneesteban this is in staging ready for review.

[jonwehausen]

QA Testing

@sonntag-adam Please see test results below

Description of Interaction	Data Layer	Data Layer Test Status	Test Notes
Click on a Need Help? link on the right rail nav		✔️ PASS	N/A
Clicks within "More resources about VA benefits		⚠️ DOES NOT PASS	@sonntag-adam We will want to correct links-list-section-header to be 'More resources about VA benefits'
Click on "Download VA Form 20-0995"	N/A	✔️ PASS	• Confirmed universal PDF tag is firing here with no datalayer changes necessary

[jonwehausen]

@sonntag-adam One other update here (and I apologize in advance for the coordination on my part here). Unfortunately our team was only able to deploy our global action link tracking to the static pages at this time.

That said if we could have the cta-action-link-click specification inserted here, that would be very much appreciated.

If there's additional metadata you all would like to pass with these events: such as the title, status of the claim, and/or submitted on date, please see this full specification:

'event': 'cta-action-link-click', 
'action-link-type': 'primary', // or 'secondary', 'reverse' --- i.e click class of 'vads-c-action-link--blue' = 'secondary' ... based on name of action link button used from css class
'action-link-click-label': //click label for action link, 'View details'
'action-link-icon-color': //name of color from class, i.e 'blue'
'claim-type': //'Dependency', 'STEM Scholarship'
'claim-last-updated-date': //date claim was last updated on
'claim-submitted-date': //date claim was submitted on
'claim-status': //i.e 'Denied', 'Claim received'

Thanks and please feel free to reach out with any questions you may have.

[jonwehausen]

@bsmartin-ep @laurensimpson14 Please see this GTM workspace as this is now ready for QA once @sonntag-adam's changes have been made: https://tagmanager.google.com/#/container/accounts/144676348/containers/2214120/workspaces/1000514

[laurensimpson14]

@sonntag-adam Is this ready to be retested?

[joanneesteban]

@sonntag-adam @amyknox Hello! The team is checking in on this to see if it's ready to be tested.

[dneel-bah]

@sonntag-adam @amyknox Hello! The team is checking in on this to see if it's ready to be tested.

@joanneesteban The failure and additional request fell through the cracks. We have captured the failure as a defect and the new request as a story. They are both in our backlog and will be worked as we have availability.

[joanneesteban]

@dneel-bah thanks for the response! Let us know if/when this is ready for Analytics support.

[sonntag-adam]

@joanneesteban this is now in staging ready for review.

[laurensimpson14]

@sonntag-adam, @dneel-bah, & @amyknox,

It looks like there are two items remaining on this ticket to be QA'd (please correct me if I'm wrong!). I've QA'd them below.

Description of Interaction	Screenshot of Interaction	Data Layer	Data Layer Test Status	Test Notes
Clicks within More resources about VA benefits			🟡 Passing, although please see the notes to confirm this is tracking correctly	I see that earlier in this ticket Jon wanted to collect these clicks as More resources about VA benefits rather than links-list-section-header. However, I don't think that this change is necessary. links-list-section-header is one of our standard event naming conventions. As long as there isn't a reason that I'm unaware of to change this, then this is tracking correctly and good to go.
Click on View Details			🟡 Passing, although please see the notes to confirm this is tracking correctly	I've clicked on claims with different statuses using test user 220. These statuses include Claim received, Denied, Preparation for notification, Initial review, Evidence gathering, review, and decision, and Closed. The Action Link CTA Click only fires for the claims with a status of Closed. This is correctly tracking the additional dimensions of claim type, updated date, submitted date, and status (see screenshot). If the claim does not have a status of Closed, then clicking View Details does not fire the new Action Link CTA Click tag. Instead, it fires a Claims View Details tag that does not track the additional dimensions of claim type, updated date, submitted date, and status. Is this what was intended? If so, then this is tracking correctly.

[dneel-bah]

@laurensimpson14

Regarding the first item, "Clicks within More resources about VA benefits", the tracker wasn't wired up correctly initially which resulted in Johns feedback. If you have passed that we can consider it completed.

Regarding the second item, "Click on View Details", I believe that is expected behavior as well. The STEM automated decision cards displayed will always have a status of closed at this point in time. If you have passed that we can consider it completed as well.

Based on this understanding, can we close out this ticket?

[laurensimpson14]

Hi @dneel-bah Thank you for the additional background! Yes, based on that I'd say we are good to close out this ticket.