Initial Set of Core 3 monitors for Tier-1 Services

[zachgoldfine44]

Identify 3 Core monitors - error rate, Anamoly and Job retries monitor for all the tier-1 services.

Definition of Done:

• [x] 2-3 Core monitors for initial set of Tier-1 services are linked in spreadsheet
• [x] Linked core monitors show up in Datadog Watchtower

[zachgoldfine44]

Steve: Debt resolution & financial report: created sidekiq monitor and dashboard. There is at least one monitor.

Steve: Direct deposit: will follow up with Denise & Samara.

Steve: Education rows: just got a response with the right contact, will follow up and hoping for progress next week.

[zachgoldfine44]

Steve:

• Debt resolution: will follow up with team, should have something by Wednesday.
• Financial report: will follow up with team, should have something by Wednesday.
• Direct Deposit: need to follow up with Denise and Samara
• Education features: Reached out today, haven't yet heard back. If no response by Wednesday, follow up perhaps with Charles reaching out to someone.

[zachgoldfine44]

Update:

• Debt resolution monitors added❤️
• Financial report monitors added❤️

Next steps @va-albers :

• Reach out to Denise and Samara for Direct Deposit
• Reach back out to Riley Ross.

(Note: monitors showed issues from most recent release wtih financial report. So, they're aware of the issues.)

For education:

• Reached out to Bobby Trulong and Darla Von Neukirk

Charles: Riley Ross is OIT lead on education work. "We have code yellow priority. Haven't heard back from Darla. What do you rec as next step?"

[va-albers]

• Slack thread on Education family of forms
• Haven't heard back from Darla (who is out today), but Bobby accepted a invite for a call early next week.

[mattpointzxer0]

Alternative plan is for us to just get the names of the related controllers and do this ourselves (waiting until after Monday, when a meetings is on the books).

[mattpointzxer0]

Include Ask VA: https://github.com/department-of-veterans-affairs/va.gov-team/issues/68420

[zachgoldfine44]

Steve:

• Education: pinged Charles with timeline of edu comms. Had a meeting with Darla and Bobby today. Asked for official confirmation from Riley Ross to get prioritization to help. Steve connecting with Riley to answer any q's he might have and confirm need for prioritization.
• Direct Deposit: Meeting with Tom Harrison on Thursday (referred by Samara)

[va-albers]

Follow-up. Met with Team leads on Monday, getting prioritization from Riley (I sent a meeting invite for tomorrow to review the ask).

Tom Harrison meeting on Thursday

[mattpointzxer0]

Steve: Riley (EDU IT) has a resource constraint, but they will set up a meeting for next week with Steve.

Zach: The feature flag is missing some of the core monitors.... @BillChapmanUSDS offers himself as tribute. ETA: Friday.

[mattpointzxer0]

Bill: We need a careful QA of all of the alerts for each of the spreadsheet features before finalizing the Code Yellow.

[zachgoldfine44]

Direct deposit monitors set up and in spreadsheet.

Meeting with Bobby from EDU Tuesday 11/7 4:30 pm ET. (Adding Patrick V to this meeting).

Steve also setting up a follow-up meeting with Darla from EDU.

[zachgoldfine44]

Halfway-through setting up monitors for known education features. Darla needs prioritization from Riley. Steve getting that for Darla.

Remaining:

• Debt resolution anomaly monitor: @va-albers
• Evidence upload failed job with no more retries: @va-albers and @mchelen-gov
• Update / complete Profile monitors: @AparnaNittalaUSDS

[zachgoldfine44]

Education monitors still needed. Check in with Steve Wednesday 11/22.

[zachgoldfine44]

Update: Chris reached out to Rob Orifici, no response. Reached out to another group.

[zachgoldfine44]

Mike and Steve to talk about making sure monitors are tagged appropriately

[mheadd]

@va-albers I think we can call this done. The 8 additional Tier 1 services are covered in this ticket.