Establish broken link CMS notification measurements, baseline objectives, and KPIs

[EWashb]

Background

After an MVP audience is decided via #9552, a pilot program for this audience will be determined. The pilot program framework should be created with specific objectives and KPIs for success.

Things to Consider

• Time from broken link detected > human on help desk side becomes aware / waiting on an editor to do something
• There are lots of variables we can't control: vacation/leave
• We want to focus on the time between detection and editor receiving info
• We want to move toward teaching editors to fix over fixing it for them

Acceptance Criteria

• [ ] Determine if there are ways that we can measure help desk LOE to see if notifications are helpful in the long term
• [ ] Explore how to measure "time to resolve broken links" - work with help desk (define how our approach fits (teach vs do for them) with how we're measuring).
• [ ] Establish HD labels/ processes for tracking
• [ ] Establish a way to differentiate tickets that are about borken links themselves vs tickets about the notifications feature
• [ ] Create plan/framework for data review
• [ ] PO review

[EWashb]

@stefaniefgray would love your help on these metrics if you're able to jump in with me next sprint!

[EWashb]

meeting scheduled 7/21

[EWashb]

Current options for measurement:

1. number of slackbot notices
2. Jira- broken links tickets over time report

Process to think through:

1. templating the process/responses?
2. Should we lump together broken link reports instead of individual tickets?
3. Need more definition of Tier 1/2/3

[stefaniefgray]

Jane's broken link review: https://drive.google.com/file/d/13YGVri1YeE9KIZLJ5aSLgsVGaoQIbcEi/view

[stefaniefgray]

Need to figure out whole process and what's going on with Outlook links

[stefaniefgray]

Could we automate broken link reports?

Current method -- VA Boston - 7 broken links VA Kansas City - 5 broken links VA Bay Pines - 11 broken links

Comes up as 3 tickets instead of 23 broken links

This obscures the data about which sections are truly the biggest broken link culprits

Can this be automated?

[EWashb]

After two meetings talking these things through, we have found that this ticket is substantial and convoluted. Measurements/metrics coming from the Help Desk would not be a good baseline metric because there is a lot to consider about time to resolve. This is a very manual process for them, so there is no real way to track time to resolve that will be accurate enough to measure change over time- editors do not input tickets, help deskers do and they are not 1:1 tickets. Currently, one slackbot notification can encompass one geographic area which can include any number of broken links.

Next steps:

• Future state blue sky = a less manual/new process to remove the time-consuming tasks of help desk responding to broken links Questions for this step: Can engineers do a query to measure the CMS vs the help desk?
• Possible templated CMS notification responses: Questions for this step: Can we figure out broken link buckets for auto-responses? Is there a way once the flag is created to autogenerate an email to the editor?

Additional tickets should be created for this.

[EWashb]

also updated the estimate because this has taken several days to get to this point in the process

[EWashb]

We should consider the HD process and how to create a template to have them correct these errors themselves. Current state has been very hands-on and is not scalable. If we change the process how will this impact the veteran?

[EWashb]

create a HD ticket that will block this

[EWashb]

Moved to stretch column (should have sooner after the initial meeting with PO). We need to look at helpdesk process and how to move editors into a more "self-service" model.

[stefaniefgray]

@TroyA6 Here's the other one about automating broken link notifications that go out to Editors

cc: @EWashb ^^ Troy and I had a great conversation on Zoom about this today, so that's why you see me resurrecting these :)

[EWashb]

removed from framework epic because this is not critical functionality to the MVP framework, but another type of notification we could pursue