Analytics KPI dashboard request for Virtual Agent Chatbot

[ianpsant]

Request a Standard KPI Dashboard

What this form is for

Use this template to request a KPI dashboard on Domo for your product.

We are currently only supporting standardized dashboards for Forms, Content, & Search products. If you have another VA.gov product, please reach out to the Platform Analytics & Insights team on #vfs-platform-support. Please note that these dashboards will require additional data and design discovery, and will take additional time to deliver.

Additional Info

• Please use the KPI Framework for more information on the metrics used in the Standard KPI dashboards.

• For additional info, read through the How to Request a KPI Dashboard Documentation.

Expected delivery timelines

Your KPI dashboard request will be added to the Platform Analytics & Insights team's customer support queue, and we expect tickets at the top of the queue to be pulled in every Platform Sprint Planning & Refinement (every Wednesday). We appreciate your patience.

• Once your ticket has been pulled into the Platform sprint, we expect to deliver your completed dashboard within the next 4 business days. Please use this ticket for any follow up questions.

Instructions for the VFS team

• [X] Please fill out this template (attached Excel sheet if you do not have Google access) & link it below to help define your product/feature's information. The Analytics team will not be able to pull this ticket into the queue until this is provided & linked.
  • https://docs.google.com/spreadsheets/d/1kUq3PAFQ__lXfK424jIyIiSZILwj-JgxHANmxY6uH5w/edit?usp=sharing
• [X] Add your team label to follow progress
• [X] If you do not have Domo access, please request access here.

VFS-provided information

• We are piloting a Virtual Agent Chatbot on VA.gov gathering quantitiative and qualitative feedback and proving out the Microsoft technology it is built on.
• The Chatbot will be launched on February 28th. Twenty-five percent (25%) of users on https://www.va.gov/contact-us/ will be presented an alert that will contain a link to the page that the chatbot is located on.
• The February 28th launch will be provide an unauthenticated experience to users. That is, they won't be required to log in to use the Chatbot.
• Early April, functionality will be released that will allow users to view their claims and appeals, and if they ask about these topics, they will be asked to authenticate (log in).
• In May, if approved, we will increase the percentage of those who can see the Chatbot link to fifty percent (50%).
• The Chatbot page they will be directed to is https://staging.va.gov/virtual-agent-study/. We are awaiting the production URL from the IA team in the next few days, at which point we will update this ticket.
• Seeking a Domo KPI dashboard for this staging and production Chatbot page in order to capture typical product metrics for those who land / exit this page.
• Once users begin interacting with the Chatbot on this page, we have other tools (Power BI) that we will be using to measure virtual agent performance.

Please leave the following blank

---

VSP Analytics & Insights Acceptance Criteria

1. Build (See internal process document & how to)

• [x] Add the VSP Analytics & Insights Group as the dashboard owner
• [x] Structure funnel data for ETL
• [x] Pipe available metrics to Domo
• [x] Document any blocked/unavailable metrics in the Analytics team space
• [x] Update the dashboard header, including Google Analytics links
• [ ] If any, add implementation issue for additional necessary implementation

2. Q&A

• [x] Please see & complete the checklist for full KPI dashboarding QA checklist

3. Share

• [ ] Add VFS group & default group to the dashboard

  For new VFS customers only:

• [ ] Schedule onboarding meeting

• [ ] Host the meeting to explain how the dashboard works

Definition of Done

• [x] Acceptance Criteria has been completed
  • [x] KPI Dashboard full QA checklist completed
• [x] Dashboard has been shared back with the VFS team

[michelle-dooley]

@ianpsant - thank you for submitting this request. We will review and get back to you with any questions we may have.

[chloedotbrown]

Hey @ianpsant, can you ping us on this ticket when you have a production URL ready to go? Once we have that, our team can start adding Virtual Agent to the Content dashboard!

[ianpsant]

Hi @chloedotbrown, as soon as we find out the production URL, you'll be one of the first to know! We're hoping very soon since we're launching 2/28. Incidentally this request to the IA team is being tracked here https://github.com/department-of-veterans-affairs/va.gov-team/issues/33905, FYI

[ianpsant]

Hi @chloedotbrown, speak of the devil, a decision on the production URL came in today as documented by https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/platform/information-architecture/ia-reviews/chatbot.md

Not so surprising, the URL will be: http://www.va.gov/contact-us/virtual-agent/

Let us know if you need anything else before our launch.

[chloedotbrown]

Fantastic - thank you so much, @ianpsant!

@jonathan-epstein13, this should be ready for you to start the process of adding to the Content dashboard.

[michelle-dooley]

Hi @ianpsant this dashboard has been set up. You will not see it out there yet b/c you haven't launched and there is no data to display. Please let us know as soon as you launch and we will complete the data QA piece. Thanks much!!

[michelle-dooley]

Moving to internal QA status pending product launch.

[michelle-dooley]

Hi @ianpsant - just a reminder to please let us know when you have completed your launch and data has begun rolling through for us to QA, Excited to get this dashboard completed for you!!

[ianpsant]

Hi @michelle-dooley! It's been a hectic few days, but we went live Wednesday afternoon (3/2)!. We've seen data in GA. Please let us know if you need anything from us to help in your QA.

[jennaouellet]

QA:

Highlights

Chart title	Assets	Pass / Fail
Usage trends	Custom segment	:green_circle: PASS
Unique users	Custom segment	:green_circle: PASS
Total clicks by device	Custom report: report	:green_circle: PASS
Unique pageviews	Filtered report: report	:green_circle: PASS

Click interactions

Chart title	Assets	Pass / Fail
User interaction rate	(BigQuery only metric)	:green_circle: PASS
Total interaction clicks	Custom report tab 3: report	:green_circle: PASS
Avg. clicks per user	Custom report tab 3: report / User report	:green_circle: PASS
Interactions with most clicks	Custom report tab 2: report	:green_circle: PASS

Depth of engagement

Chart title	Assets	Pass / Fail
Scroll depth distribution	n/a	:green_circle: N/A
Avg. minutes on page	n/a	:green_circle: N/A
Avg. scroll depth	n/a	:green_circle: N/A

Finding the page

Chart title	Assets	Pass / Fail
Top pages previously viewed	Custom report: report	:green_circle: PASS
Devices used to access the page	Custom report tab 2: report	:green_circle: PASS
Top browsers used to access the page	Custom report tab 3: report	:green_circle: PASS

[jennaouellet]

@ianpsant QA has been completed and Virtual Agent content is now on the Content KPI Dashboard! You can view your data by filtering the product at the top of the dashboard to be Virtual Agent Chatbot pages.

I'll close this ticket, though feel free to follow up if you have any questions.