Analytics Feasibility Review: OKRs Accredited Rep Team

johnny-jesensky-adhoc commented 1 month ago

Description

We met with the Find-A-Rep Team on Wed, Sept 18, 2024 to review their OKRs. We need to review their OKRs, and the metrics they wish to track, and determine whether they are feasible. If they are, we should look to highlight the right metrics, dimensions, or custom work that could be leveraged.

Mural

Find A Rep OKRs

Document notes in either the Mural board, or some other document that could be shared with the Find-A-Rep Team.

Tasks

[x] Review OKRs.
[x] Confirm that the interactions they want to measure make sense.
[x] Identify which metrics, dimension are possible, and/or recommended to achieve these.
[ ] Identify what might require new tracking, versus existing tracking.
[x] Possible Task -- have brainstorming meeting internally to talk through these.

Acceptance Criteria

[x] OKRs have been reviewed
[x] Our analysis and findings have been shared.
[x] Share findings with Lindsay and the Find-A-Rep Team

johnny-jesensky-adhoc commented 1 month ago

Schedule brainstorming meeting to consolidate our thoughts on this.

jestutt commented 1 month ago

As I mentioned during Wednesday's meeting, I referred to the guidelines suggested by the Digital.gov team as I was reviewing the OKR mural board. Between those guidelines and a refresher on OKRs, it quickly becomes apparent that we are missing the Initiatives or CTAs, the set tasks that visitors must complete to achieve the goals and objectives. Without those tasks, it's hard to know the metrics and what tracking needs reviewed, added or updated.

It should be noted that the OKR framework is generally used for the tracking of goals status towards achieving an organizations strategy, which is different from the performance measurement (and KPIs) we normally see our web reporting. Think of it as OKRs are a high-level insight generated from one or more KPIs but not all KPIs feed into an OKR. A visual breakdown.

Here's most of what I touched on in that meeting.

Content Pages (Landing and FAQ)

I would argue that the content pages are support pages for the search and don't qualify for their own OKR or even KPI. Instead, we can look at the individual pages for interactions that, while not part of regular reports, could feed into future analyses.

Landing page

What content area gets the most attention?

On the page and accordion clicks
CTA, more info and content link clicks

FAQ

What question gets the most attention?

It's best for measurement of FAQs if they are all accessed thru an accordion or have an On the page menu so you have a trackable interaction that you can connect to the individual question to show usage/demand

Find a Representative

The listed objectives are sentiment focused with no clear definition of what it means to "successfully navigate", "find value in engaging" or "remain healthy". None of these point to tasks that can be measured in web or backend metrics.

Expedite the deprecation of the legacy eBenefits platform by delivering replacement services

This is one of the Goals Lindsay shared on Slack and I think it should have it's own objective. Something along the lines of a clear or maybe % based decline in usage over the next 6- or 12-months. The task would be visits to those other tools that I believe we have in GA.

Increase the findablilty of the Find a Representative tool on VA.gov

In our meeting with the the team, someone said that the end goal for the "successfully navigate" objective was to get the tool linked to in the header. The ability to navigate to/find content on a website is standardly measured by user testing. The only task interaction we'd find in GA (and backend) to show that specific content is hard to find would site-wide searches. There are a metrics and angles this task feed into but the simplest is defining what portion of the internal traffic to the FAR search pages comes from the side-wide search. We can look at other tools with and without a header link to define a threshold percentage that you can measure against and use in the objective.

Vets are successfully finding a representative

If success = connecting with representatives -- we cannot measure this. The connection happens offline and there's no way to connect a search to an offline interactions. At least not right now.
if success = results returned for a search -- Does the backend search data include the number of results a search returned? We should be able to get searches with no result in GA (I don't think we're getting this now). -- This could also feed into the quality of the rep data's coverage along with those outdated info submissions.

Search is available/has a XX% up time

Except for the one labeled future, the "healthy" key results feed into the up time of the tool. These are all backend (non-GA) data points, correct?

Widget

Other than connecting logins to the widgets, I don't think there's are any web-based tasks that we can measure for the non-health related objectives. We do need to look a the widget to confirm the existing state of tracking for it (and other widgets) so we have an idea of what might need updating when we have a better idea of the final tracking needs.

johnny-jesensky-adhoc commented 1 month ago

Results have been shared with the Find-A-Rep Team. I'm coordinating to find a time to schedule a follow-up meeting with them.

Moving this to Done.

FYI @Blivaditis, @jestutt

oddball-lindsay commented 3 weeks ago

Notes from our call today with @jestutt @Blivaditis @johnny-jesensky-adhoc @jfinnerty13 @holdenhinkle, where we focused on Find a Rep OKRs:

Objective: Users are able to successfully navigate to Find a Representative on VA.gov

We can look at internal traffic and segment it by source (search, entry point, etc.) -- this is ready to track

Objective: Expedite the deprecation of the legacy services (eBenefits and OGC's accreditation search) by delivering replacement services

Recommendation is to make this it's own objective, not looped into the objective above.
The ARM team is concerned that the tracking of eBenefits and OGC websites are not accurate in GA4 (cc @jfinnerty13)
- ❗ we are unsure if we can track this

Objective: Users are engaging with Find a Representative

We can compare engagement with a baseline, which would be the Find a VA Location engagement -- this is ready to track
We can compare % of users who perform a search with a baseline, which would be the % of users who perform a search on the Find a VA Location -- this is ready to track

Objective: Users are able to successfully find a representative on VA.gov

We can look at average number of searches per user -- this is ready to track
We can look at the percent of searches that return no search results (this is an empty state that the FE knows about)
- ❗ there may be additional ARM work to track this
We can look at the percent of searches that return an error (this is an error state that the FE knows about)
- ❗ there may be additional ARM work to track this

General action: meet again (ideally for more than 30min) to:

Review what we can track for Find a Rep
Widget OKR(s)
Tracking for our content pages

@johnny-jesensky-adhoc will you be scheduling this or shall I?

jfinnerty13 commented 3 weeks ago

Adding context for eBenefits.

One of the links we'd like to track is: https://www.ebenefits.va.gov/ebenefits/vso-search
When I attempt to obtain data in GA4 it looks like the attached, but as you can see I can only see 1 user, and I can't even get June to load.
As compared to UA3 (attached PDF) you can see roughly 11k pageviews in June. 2024_06_eBenefits_VSOSearch.pdf

Hopefully it's just user error, but let me know if there's anything else I can provide.

johnny-jesensky-adhoc commented 3 weeks ago

@oddball-lindsay -- Thanks for sharing your notes. I'll look at calendars, and try to get something on the books between our two teams, (will aim for 45 min - 1 hour).

@jfinnerty13 -- thanks for sharing. We'll take a closer look at what ebenefits.va.gov tracking we have set up on our GA4 Property.

FYI @jestutt, @Blivaditis

jestutt commented 3 weeks ago

@jfinnerty13 Okay, this is not good. Ebenefits.va.gov is not part of our GA/GTM implementation. I see an unknown GTM container (GTM-TDS4ZQ) on the ebenefits.va.gov site and what looks like tracking events for the VA.gov's GA UA properties but no tracking events for the VA.gov's GA4 properties.

Interestingly, the search itself is iframed into the ebenefits page and the iframe source -- https://www.ebenefits.va.gov/wssweb/wss-common-webparts/mvc/vsosearch -- is the only content I see to have the DAP tracking snippet on it. This means that we should be able to get the information you need from the DAP Veterans Affairs GA4 property. If you haven't done so yet, here's how to get access to DAP.

jfinnerty13 commented 3 weeks ago

Pinging @oddball-lindsay @jenniferbertsch for awareness, please see Jamies above comment.

@jestutt double checking can we view eBenefits in GA4 or do we also need to request DAP access? Sorry you mention both so I wasn't sure. Thanks!

jestutt commented 3 weeks ago

@jfinnerty13 You'll need to request access to the DAP GA4 property for Veterans Affairs and then you can see GA data for the VSO Search only.

oddball-lindsay commented 3 weeks ago

@jestutt my end goal for these OKRs is to have them in a Domo dashboard for our VA enablement to easily visualize. Would tracking in DAP be able to be routed to Domo, or will this piece have to live on it's own?

cc @Blivaditis in case this is more of a Domo question!

oddball-lindsay commented 2 weeks ago

Confirmed that we will not be able to visualize DAP in Domo (Domo is only for GA4) so we will need DAP access and will have to track eBenefits traffic there.

department-of-veterans-affairs / va.gov-team