Launch new QA standard re endpoint monitoring (and modified SR requirements and required artifacts)

[shiragoodman]

User Story

As a Governance team member, I want to launch and announce the changes to the QA Standards re: endpoint monitoring so that VFS teams can be prepared for upcoming Staging Reviews.

As a Governance team member, I want to launch and announce the changes to the SR requirements and required artifacts so that VFS teams can be prepared for scheduling an upcoming Staging Review.

Assignee: @briandeconinck Peer Reviewer: @it-harrison

Description

Note: This ticket includes the launch of the new endpoint monitoring QA standard as per the initiative, as well as the launch of the modified SR requirements (per ticket #88107) and the new SR required artifacts (per ticket #88140). The latter tickets were handled as tech-debt and are not affiliated with a larger initiative.

Since all changes impact the SR code blocks and the open Collab Cycle Request tickets, it makes it easier for Governance to launch and announce the changes together .

• We can launch endpoint monitoring QA standard by Sep 10, but when should teams be held accountable to meet it? Possibly end of September? Starting October 1? Tag Matt.
  • Update 8/30: due to the escalated priority of eliminating all silent failures (aka where the endpoints fail and the Veteran forms are NOT submitted to the appropriate systems), we will begin enforcing the new QA Standard for endpoint monitoring immediately, meaning September 10.

Impacted Artifacts

Live pages

• QA Collab Cycle Findings for Staging (gh issue template)
• QA Standards (Platform Website)
• Staging Review touchpoint page (Platform Website)
• Change history (Platform Website)
• How to complete an IA review for Staging review touchpoint (Confluence operational)
• How to complete a Content Review for the Staging Review Touchpoint (Confluence operational)
• How to document findings for Staging Review (Confluence operational)
• How to complete a QA review for the Staging review touchpoint (Confluence operational)
• Collaboration Cycle Kickoff - Code Blocks Staging Review section (Confluence VFS-facing)
• Staging Review Calendly event (Calendly)
• All open Collaboration Cycle Request tickets (gh issue tickets)

Drafted pages

• Folder with draft pages

Tasks

• [x] Draft change history post for Platform Website
  • Include 1 bullet for the new endpoint monitoring QA standard, and 1 bullet (or 2) for the SR requirements changes and the required artifact changes
  • Draft change history posts
• [x] Draft change announcement following Platform guidelines
  • Draft change announcements
• [x] Add notes from Matt to QA Standards PW page
• [x] Ensure Last Updated date on all PW pages is updated to current date
• [x] Run script to update open Collab Cycle Request tickets
• [x] Publish all changes to Platform Website
  • [x] Staging Review touchpoint page (Platform Website)
  • [x] QA Standards (Platform Website)
  • [x] Change history (Platform Website)
• [x] Publish all changes to operational docs in Confluence
  • [x] How to complete an IA review for Staging review touchpoint (Confluence operational)
    • Changes already published in Erin's previous work
  • [x] How to complete a Content Review for the Staging Review Touchpoint (Confluence operational)
    • Changes already published in Erin's previous work
  • [x] How to document findings for Staging Review (Confluence operational)
    • Changes already published in Erin's previous work
  • [x] How to complete a QA review for the Staging review touchpoint (Confluence operational)
  • [x] Collaboration Cycle Kickoff - Code Blocks Staging Review section (Confluence VFS-facing)
• [x] Squash and merge the pull request #86108
• [x] Edit Calendly SR sign up page and save changes
• [x] Once all of the above is complete, post change announcement to #vfs-change-announcements channel. Cross share post on any other relevant channels in Slack.
• [x] Notify @erinrwhite of status of changes so they can communicate in the 9/10 Team of Teams meeting.

Peer Review

To be completed by peer reviewer

• [x] User story and acceptance criteria are met

Acceptance Criteria

• [x] We have published all guidance related to the new endpoint monitoring QA standard
• [x] We have published all guidance related to the modified SR requirements and modified SR required artifacts.
• [x] VFS teams have been made aware of these changes
• [x] VFS teams with open Collab Cycle Request initiatives have the updated guidance in their tickets
• [x] VFS teams signing up for Staging Review can see the updated requirements
• [x] We have notified VFS teams of the new standard at the 9/3 and 9/10 Team of Teams meetings

Team Notification

• [x] Slack post on #platform-governance-team
• [ ] Presentation at Weekly Initiative Meeting
• [ ] No notification needed

How to prepare this issue

Refinement

• [x] Ticket has user story, description, tasks, and acceptance criteria
• [x] Ticket has been sized
• [x] Once all above items are checked, add the 'Ready' label.

  Planning

  If this ticket is picked up from the Backlog mid-sprint, connect with Shira to ensure the below items are completed correctly

• [x] Ticket has been discussed as a team at Planning
• [x] Ticket has been assigned to appropriate initiative or quarterly epic
• [x] Ticket has been assigned to a Sprint
• [x] Ticket has been moved to the Planned pipeline in the Governance team board
• [x] Ticket as an assignee and peer reviewer

[shiragoodman]

@humancompanion-usds We plan to launch the new QA standard for endpoint monitoring, and the SR requirements changes and required artifacts changes by end of sprint - Sep 10. However, we think it makes sense to not begin enforcing the new QA standard until Oct 1. Are you ok with that?

[shiragoodman]

@briandeconinck @it-harrison in my 1:1 with Matt on 8/30, we learned that there is an escalated priority of eliminating silent failures. This means that we will not be giving VFS teams any grace period at all for adhering to the new QA standard. When the new QA standard for endpoint monitoring is launched on September 10, VFS teams will be expected to meet standards right away. Meaning ay staging Review scheduled post 9/10 will need to meet ALL QA standards. I have updated the Description above with my 8/30 update.

This ticket is a priority! Please ensure everything is in order and on track to launch by September 10. If any issues come up at all, please notify @humancompanion-usds immediately. ty!

[shiragoodman]

also, @erinrwhite will be previewing these launch changes at the September 3 Team of Teams meeting and will be communicating the launch date of September 10. Please keep them informed as well if anything changes.

[erinrwhite]

@it-harrison has updated collab cycle tickets with this info as part of #91315 👍🏻

[humancompanion-usds]

@briandeconinck - Some content Steve and I would like to add to the QA Standard for endpoint monitoring:

Minimum requirements to meet this standard

1. End point monitoring must cover the three fundamental types of problems: A. Unexpectedly high errors B. Unexpectedly low traffic C. Silent failures (silent to the end user) of any kind
2. Don't ever include PII or PHI in logs!
3. Errors that appear in DataDog should be routed to a Slack channel. (You should name this channel something like #(team name)-notifications
4. The Slack channel should be monitored daily by the VFS team.
5. The team should establish a process for acknowledging and assigning errors to a team member to investigate and resolve. (Playbook on this forthcoming)

Behaviors to avoid

• Do not silence errors if there are too many. Instead we expect you to tune monitoring thresholds to be useful.
• Do not assign monitoring to one person.

[briandeconinck]

@it-harrison I have made edits to the draft QA Standards page to include the content @humancompanion-usds provided above -- mostly copy/paste, some slight changes to wording. Let me know if this all seems good to you. Not sure if you need to make any changes to your operational guidance based on Matt's notes.

Change history posts and change announcements are also ready to go. If the updates to the QA Standards page look good to you I'll begin moving things over to publish on Platform website.

[it-harrison]

@briandeconinck changes look awesome! thank you!

[briandeconinck]

Cool, thank you. I'll start getting things queued up for Platform website!

[briandeconinck]

Friday afternoon update:

• All tasks are complete except for announcing to #vfs-change-announcements
• Waiting on updates to be published to Platform Website.

[briandeconinck]

@it-harrison I believe all tasks on this are completed. Could you review just to make sure I didn't miss anything?

[erinrwhite]

Ben Brasso had a great clarifying suggestion to the entry point guidance on Slack, so I have added a sub-bullet of information to...

• The entry point part of the Staging Review page that will publish tomorrow. (Page is waiting to publish on confluence)
• the Kickoff code blocks page
• I don't think we should worry about older tickets. This is just a clarifying edit.

Change text is...

If the primary entry point is not a page on VA.gov, include information about how to view it. Reach out to @platform-governance-team-members on Slack with any questions.

[it-harrison]

@briandeconinck this is good to close!