UX monitoring of TE Release

[evelynhilbrichdavis]

Issue Description

We need to implement UX monitoring for the upcoming release of Toxic Exposure to ensure that the user experience meets our quality standards and to identify any potential issues early. This will involve both quantitative data collection and qualitative feedback to provide a comprehensive view of user interactions and satisfaction.

Note: This monitoring will run the course of the 1.0 roll-out, from 5% to 100%. After that point, we will continue to review GA4 metrics and shift the Medallia focus to research study planning.

Monitoring tools:

1. GA4 to track form metrics (especially opt-in, form exit, and abandonment)
2. Medallia survey data-- qualitative and quantitative feedback
   • Satisfaction (Feedback survey)
   • Overall experience (Intercept survey)
   • Understandability (Intercept survey)
   • Trust (Intercept survey)

Related:

1. Review Medallia survey data to prepare for TE launch#88714

---

Tasks

• [X] Set up weekly data requests for Medallia from Contact Center (Plan developed, see comement)
• [ ] Set up weekly cadence with Product (Jared) to discuss GA4 data
• [x] Generate weekly Github reports that highlight any significant findings
• [x] Escalate problems to OCTO and engineering team

Tasks below will be moved into tickets divided by release, not one single big ticket. No survey/CC feedback has been reviewed to date

• [ ] Analyze GA4 data and user feedback from Medallia surveys
• [ ] Compare Medallia insights with quantitative data from GA4

Acceptance Criteria

• [x] Reports are complete and shared with OCTO and engineering team 5% has been shared; future will be captured in broken down tickets

• [ ] Findings inform research questions for upcoming Toxic Exposure usability study. This will be included in future ticketing-- dividing up so it isn't one big lump sum ticket

  How to configure this issue

• [ ] Attached to a Milestone (when will this be completed?)
• [ ] Attached to an Epic (what body of work is this a part of?)
• [ ] Labeled with Team (product support, analytics-insights, operations, service-design, Console-Services, tools-fe)
• [ ] Labeled with Practice Area (backend, frontend, devops, design, research, product, ia, qa, analytics, contact center, research, accessibility, content)
• [ ] Labeled with Type (bug, request, discovery, documentation, etc.)

[shannonkford]

@evelynhilbrichdavis a couple of comments/questions:

• I don't think this should be timeboxed to the first 2 weeks of launch, since during that time we'll be at 50% or under (according to current plans, we'd be at 50% for 2 days only), assuming all goes to plan. Having a week at 100% would be useful.

• You might want to include looking at the GA4 data in the tasks - will this also be on a weekly basis?

[evelynhilbrichdavis]

@shannonkford back at you--

1. I will adjust the ticket to include combined Medallia + GA4 reporting through 100%. My initial rationale on the timebox was that we still don't know how useful Medallia will be for monitoring, so I wanted to give us a chance to phase it out and change gears. But agreed that 100% could be useful.

2. Adding GA4 to the tasks to be more clear-- the weekly reporting will share out insight from combined GA4 data and Medallia review. Since there will be a delay on these reports (requesting Medallia data and processing it), Jared will also be keeping OCTO up to date on KPIs (I believe those check-points are already on our release cal).

[evelynhilbrichdavis]

We added Contact Center data to our plan for reviewing Medallia survey data. Optimized time for data pull is on Fridays, so those dates will be: 1.0 @ 50% on 9/13 1.0 @ 100% / 1.1 at 5% on 9/20 1.1 @ 75% on 10/4

[evelynhilbrichdavis]

This monitoring work is now divided up between the following tickets so we have a better sense of my capacity. Tickets are split up based on rollout, not sprint-- and each contains one of the more time-consuming qualitative data pulls. @mengA6 @pacerwow

1. 92321 (1.0 10-50%)

2. 92326 (1.0 75-1.1 10%

3. 92327 (1.1 25-100%)