Analytics edge case KPI dashboard request for Facilities Team / Facility Locator

[mmiddaugh]

Request an Edge Case KPI Dashboard

What this form is for

Use this ticket template to request a new edge case KPI dashboard on Domo for your product. Edge case dashboards are for products that fall outside of the scope of our standardized KPI dashboards (Forms, Content, and Search).

If you are unsure what type of dashboard your product qualifies for, please check the Eligibility Dashboard on Domo.

Expected delivery timelines

The Analytics team will notify you once this ticket has been pulled into a sprint. Please note that these dashboards will require additional data and design discovery and will take additional time to deliver. You can also reach out for updates on this ticket.

Instructions for the VFS team

In order to pull the correct data for your product, we need your help to define what pages and events to include, in addition to basic product info. For edge cases, we also ask for product outlines or other resources that will help us understand success for your product and define metrics:

• [ ] Make a copy of this Google Sheet (or download Excel version).
  • [ ] Fill out product information basics.
  • [ ] If a part of your product involves any sort of linear process/funnel for which you want to be able to measure conversions, please include it in the "Forms" tab of the template.
  • [ ] All other pages associated with your product for which you'd want to track visits and engagement should be added to the "Content" tab.
• [ ] Link to your completed spreadsheet here:____
• [ ] Check your sharing settings to make sure the Analytics team will be able to access it.
• [ ] Answer the questions in the VFS-provided information section below.
• [ ] Add links to the VFS-provided information section below.
• [ ] Add your team label to this ticket to follow its progress.

VFS-provided information

Product questions

Help us understand your product better by answering the following questions:

1. What does your team consider a completed task?
   • Primary tasks:
   • Firing a search
   • Clicking on a search results to learn more about the location
   • Secondary tasks
   • Interacting with other page elements such as the "Try using our other tools to search" link below the map or the "Learn about your in-network urgent care benefits" within community care urgent care search results.
2. Who is your product's main user base?
   • Veterans and beneficiaries
3. Why does your product require an edge case?
   • The Facility Locator uses a combination of several search parameters to initiate a search which references the Lighthouse Facilities API, rather than the global search function
4. Does this product currently have a Domo dashboard? If yes please add link.
   • not currently

Product resources

Add links to product outlines, feature information, or any other resources you think would help us better understand your product and OKRs below. Feel free to also type background information into this section directly. The more context you're able to provide, the fewer questions we'll need to ask you before getting started!

Product URL: https://www.va.gov/find-locations/

Most recent product brief: Initiative Brief - Facility Locator Search Improvements

Valuable metrics

• Users
• new users
• referral sources
• Device distribution
• Facility Detail pageviews, such as https://www.va.gov/find-locations/facility/vba_346b
• Modernized facility pageviews, such as https://www.va.gov/pittsburgh-health-care/ or https://www.va.gov/reno-vet-center/
• bounce rate - defined as abandonment before firing a search (because if the Veteran only needed an address or phone number, the task may be satisfied within the search results, making clicking through to more information unnecessary
• use of "escape hatch" -> "Try using our other tools to search" link below the map
• scroll depth within search results list - or paginated search results (i.e. Veteran viewed through 3 pages of results)
• search term distribution
• search refinements by type
• CSAT/Foresee aggregates and by facility type, if possible

Additional info

• Please use the KPI Framework for more information on the metrics used in the Standard KPI dashboards.
• For additional info, read through the How to Request a KPI Dashboard Documentation.
• If you do not have Domo access, please request it here.

[michelle-dooley]

Thank @mmiddaugh - we will review and get back with timing.

[michelle-dooley]

Hi @mmiddaugh - been doing some planning for this and have some questions...

• For the Facility Detail pageviews we should include all urls that start with "https://www.va.gov/find-locations/facility/" correct?

• For Modernized facility pageviews the links (https://www.va.gov/pittsburgh-health-care/) do not have a substring within the url (like "find-location/facility") so I will need you provide a list of all urls to include.

• What do you mean by "referral source"?

• Can you clarify how you want bounce rate defined? Are you looking metrics on users who use the "search" on the facilities page? Or are you looking for users that click through to other pages (and what are those pages?). Could you provide me an example?

• For escape hatch that link click is not being tracked however can provide metrics on pageviews from location pages to https://www.va.gov/directory/guide/home.asp - will that work?

• Scroll depth within search results are you looking for scroll depth metrics on these pages if not can you tell me what search page url you are referring to?

[michelle-dooley]

Hey @mmiddaugh - just following up on this, had some questions above we need in order to begin the work.

[mmiddaugh]

Apologies @michelle-dooley ! I didn't get a notification that you had questions - I just popped into to check on the status.

For the Facility Detail pageviews we should include all urls that start with "https://www.va.gov/find-locations/facility/" correct?

• yes

For Modernized facility pageviews the links (https://www.va.gov/pittsburgh-health-care/) do not have a substring within the url (like "find-location/facility") so I will need you provide a list of all urls to include.

• the list would be thousands of URLs - can you use the same Regex or other logic as on the Content DOMO?
• Vet Centers are all https://www.va.gov/`(name)`-vet-center/ -> https://www.va.gov/dundalk-vet-center/
• VAMCs are https://www.va.gov/`health care system name-health-care/locations/facility name`-va-clinic/ -> https://www.va.gov/maryland-health-care/locations/baltimore-va-clinic/
• or https://www.va.gov/`health care system name`-health-care/ -> https://www.va.gov/maryland-health-care/

What do you mean by "referral source"?

• We're interested in knowing how the user arrived on the Facility Locator - was it from a link on an internal VA.gov page? (and if so, which one?) or organic search, for example. Would it be easier to provide the previous page url?

Can you clarify how you want bounce rate defined?

• Bounce rate = someone who performed a search and did not click through to a facility detail page or modernized page

For escape hatch that link click is not being tracked however can provide metrics on pageviews from location pages to https://www.va.gov/directory/guide/home.asp - will that work?

• Can you provide Facility Locator pages to https://www.va.gov/directory/guide/home.asp?
• Please let me know if the team needs to add an event to make this possible

Scroll depth within search results are you looking for scroll depth metrics on these pages if not can you tell me what search page url you are referring to?

• When the user does a search, we'd like to know how deep into the results he/she goes. For example, something like this... on average, the second listing on the first page is most likely to be clicked but most clicks are within the top 5.

[michelle-dooley]

Thanks @mmiddaugh for the clarifications. We will review and work on a mock up of what we can provide based on this for your review.

[michelle-dooley]

@mmiddaugh as we are working on this mock up I am wondering why the Content Dashboard filtered to a specific location url won't give what you need for the facility specific metrics you are looking for?

Private Zenhub Image

[mmiddaugh]

@michelle-dooley When looking for analytics related to the interactions on Facility pages, we do use the Content DOMO, filtered to the VAMC or Vet Center product, applying page URLs as needed, depending on the desired metrics. But this issue is for metrics related to searches performed on the Facility Locator at https://www.va.gov/find-locations, which is a Search product, not a Content product.

As indicated in the initial issue description from July, we requested this edge case dashboard because the Facility Locator can't be addressed by one of the standardized KPI dashboards because it uses a combination of several search parameters to initiate a search which references the Lighthouse Facilities API, rather than the global search function.

[michelle-dooley]

Thanks @mmiddaugh - we will continue this work in the sprint starting tomorrow. We will schedule some time to review mock up before we build.

[michelle-dooley]

Hi @mmiddaugh - want to get some time on our calendars to review mock up. Do any of these time slots work for you.

Friday 10-12:30 or 1-2 EST Monday 2-4 EST

[mmiddaugh]

Hi @michelle-dooley - I am apparently not getting notifications for comments on this issue - truly apologize. Can we find a time to review please?

[michelle-dooley]

Hey @mmiddaugh we have created the template in Domo and are currently QA'ng the data. How about we get some time after holiday to review, should be complete by then.

[mmiddaugh]

yes please! My calendar is up to date. This link should let you see my availability

[mmiddaugh]

hi Basil and @ChristineYoungZA - thank you for your good work on the Facility Locator Dashboard.

Hope the following will be helpful - please let me know if you need additional info (kindly send me a Slack heads up so I don't keep you waiting)

Here's a sample GA page where I look at search refinements (screenshot below). I'm honestly not sure if it is counting refinements via interaction with the map or only changes to the search fields.

---

Related to searches by facility type, we'd like to know the percentage of searches for the following, at a minimum (as available in the Facility type drop down screenshot)

• VA health (all)
• Urgent care (all)
• Emergency care (all)
• Community providers in VA's network
• Community pharmacies in VA's network
• VA benefits
• VA cemeteries

• Vet Centers

  Some facility types have subcategories (using the Service type field). This data would be nice to have, not required. It's available in GA as Site search category. Service type applies to the following facility types:

  • VA health (service type dropdown)
  • Urgent care (service type dropdown)
  • Emergency care (service type dropdown)
  • VA benefits (service type dropdown)

  • Community providers in VA's network (typeahead service match corresponding to a code such as 122300000X)

    Click through to facility pages is possible for VA facility searches only - via linked facility names in search results. (The VA.gov DOMO dashboard has metrics for Facility details viewed that should work the same way)

• VA health pages have a URL root like va.gov/[system name]-health-care/locations/[facility name]/, such as Washington VA Medical Center - these locations are included in Urgent care and Emergency care search results when All in-network or VA urgent/emergency care is selected as the service type
• Vet Center pages have a URL root like va.gov/[name]-vet-center/ or va.gov/[name]-vet-center/locations, such as Reno Vet Center
• VA benefits currently point to legacy page with a URL root like va.gov/find-locations/facility/vba_[facility id] such as National Capital Region VA Regional Benefits Office but are starting a MVP modernization roll-out which will follow the VA health and Vet Center pattern
• VA cemeteries currently point to legacy page with a URL root like va.gov/find-locations/facility/nca_[facility id] such as Alexandria National Cemetery
• There are no corresponding pages for Community providers, Community pharmacies or In network urgent/emergency care search results

---

Screenshots

GA search refinements

Facility type dropdown

Example service type subcategory dropdown

GA site search category

[johnny-jesensky-adhoc]

Hi @mmiddaugh, I've taken over as the PM for the Analytics Team from Michelle Dooley, and wanted to provide a quick update on this! Our team has performed some QC with data from last quarter (10.01.23 - 12.31.23) , and can confirm the reports are working as expected, and ready for your team's review - with a few exceptions:

• We are currently unable to track how often there were no results found with the current event structure - we've moved that to chart to the bottom of the dashboard for now, but can remove to avoid any confusion.

• We need to do some additional discovery, related to searches by facility type. I just noticed that from your most recent comment that was a request, and while that isn't something that is in there now, is something we can look to add, will just need some additional time.

We also had a couple of quick questions:

• We currently have a a couple of quarters of data loaded, (July 1, 2023 - Dec 31, 2023) - how far back would you like the data for this dashboard?

Please let me know if you have any questions, or if anything looks off or was missed. Once we get the timeline of how far back we need to load data, we're hoping this can start being used by your team as a Version 1, then continue to work on iterating and refining this over time.

I'm hoping nothing was missed with this changing hands from Michelle D. to myself, but happy to hop on call to talk through anything that seems amiss.

FYI @Blivaditis, @ChristineYoungZA, @dorseypa1

[mmiddaugh]

Hi @johnny-jesensky-adhoc - thanks for the ping! I'm looking forward to the availability of v1 It would be nice to have data for the full year of 2023 but July 1, 2023 is a decent place to start. Is anything needed from the Facilities team to track searches with no results found? Please let me know if you have additional questions about searches by facility type.

[johnny-jesensky-adhoc]

Hi @mmiddaugh, excellent - we will look to load data for all of 2023, as well as set up a recurring data load schedule so your team can start to leverage the dashboard with current metrics. Once that it set up, it should be good to use in a V1 state - we will let you know.

Our team will also look further into the other two items,

• Tracking for when no results are found.
• Percentage of searches by facility type.

FYI @ChristineYoungZA, @Blivaditis, @dorseypa1

[johnny-jesensky-adhoc]

Hi @mmiddaugh, just wanted to provide you with another update on this. We have performed the following updates on the Facility Locator Edge Case dashboard this week.

• Data has been loaded from January 1st, 2023 - Sunday, February 18th, 2024.
• The dashboard is set to update weekly with the latest data populating every Sunday.

The dashboard should be good to start being used as a Version 1. We will plan to look into the other items in a separate sprint, being tracked on this ticket.

Please let us know if you have any questions on this. Thanks!

FYI @ChristineYoungZA, @dorseypa1, @Blivaditis

[mmiddaugh]

@johnny-jesensky-adhoc I'm sorry to report that the dashboard is missing expected data in some areas and appears to report incorrect data in others. I don't think it is usable in the current state.

The graph appearing under the What type of facilities are people searching for?` heading actually represents the number of times a user has clicked through from a search result to the corresponding facility page for more information, not what people are actually searching for. The expected data is the percentage of searches for each type of facility

• VA health (all)
• Urgent care (all)
• Emergency care (all)
• Community providers in VA's network
• Community pharmacies in VA's network
• VA benefits
• VA cemeteries
• Vet Centers

(I included a link to a similar GA search in an earlier comment)

---

Click through is itself a desirable metric but the graph seems to only capture the click throughs to modernized VAMC and Vet Center pages and does not include data for two facility types, which both follow the pattern of https://www.va.gov/find-locations/facility/ facility id

• VA benefit facilities have a facility id which begins with vba, such as VA Regional Benefit Satellite Office at Thomas S. Foley U.S. Courthouse
• VA cemeteries have a facility id which begins with nca, such as Cheltenham Veterans Cemetery

---

I do not think the data presented in the Search refinement block is correctly defined. It indicates 27 searches were refined and 5,679,171 searches were not. (DOMO filtered to show data for the month to date)

By comparison, this GA page shows the refinement rate varying from one facility type to another, across 341,751 searches and an average of 5.99% for the same time period.

screenshot from DOMO

GA table

Presenting the average rate of refinement would be more useful than the number of refined searches compared to searches without a refinement. Most useful would be the refinement rate per facility type, as shown in the GA table above.

[This Github page] contains data tracked all of last year if it might be useful in evaluating how the data is being defined.

[johnny-jesensky-adhoc]

Hi @mmiddaugh - thank you for flagging this, and I'm sorry to hear some of the cards in that report are showing inaccurate data - I'll have my team take look at this early next week, and perhaps we can schedule a follow-up call with you to make sure we have everything fully validated on this front. We want this to be a report your team can fully rely on and trust, so we will work to correct.

I will be in touch.

FYI @Blivaditis, @ChristineYoungZA