Come up with a way to notify other members of the team when a new IC or OL is needed

[jimleroyer]

Description

As an incident command or ops lead, I want a second line of defense on support, So that I can ping extra help if I become unavailable or needs an extra help if support 🔥 is ravaging.

WHY are we building?

More reliability on support. We are running a 24/7 service and needs to ensure we are always able to have our IC and OL roles fulfilled.

WHAT are we building?

Potentially a second team in opsgenie that would be inserted into the escalation policies, notified after the main support team.

VALUE created by our solution

Reliability and resilience in our support.

Acceptance Criteria

• [ ] Raise the discussion within the team first, for which we will decide acceptance criteria for this new line of defense.

QA Steps

• [x] Present the new procedure to the team.
• [x] Have the opsgenie (and/or other means) reviewed by a team member. Present to the team as well.

[jimleroyer]

Waiting on team lead to organize polling and maybe another meeting with the SRE/Core team to have their takes.

[jimleroyer]

We had a meeting today and following conclusions came, along with a few action items:

• We do not expect folks to carry their laptop or mobile if they are not compensated.
• Meeting with Core team would be valuable to know what are their thoughts on their topic and if they want to be part of the second line of defense.
• It might be possible to get the second line of defense compensated if we raise it to management. Our acting director will bring it up.
• Hence two potential solutions can be drawn from this meeting:
  • Second line of defense is not compensated for waiting time and a best effort method. A third line of defense with all the Notify members will follow.
  • Second line of defense is compensated for waiting time. More team members were interested in this option, but even then, not all would want to participate. This could be a voluntary participation.

[jimleroyer]

Next action is waiting on team leader for chiming in with Platform Core team, and director to speak with management on compensation.

[ben851]

Jimmy met with Ioana, we might be blocked on this due to lack of funding approval...

[jimleroyer]

Jimmy to ping Mohamed and Ioana about this. Must likely this specific card will get rescoped to a mechanism to reach out to others.

[sastels]

Jimmy asking today!

[sastels]

Jimmy will document process.

[jimleroyer]

I didn't work on this today but I ask our favorite AI to generate documentation on this. I will review tomorrow, correct if necessary and put this in a Google document.

Sending Alerts through Opsgenie Web Interface:

1. Login to Opsgenie:

• Open your web browser and navigate to the Opsgenie login page.
• Enter your credentials and log in to your Opsgenie account.

2. Navigate to Alerts:

• Once logged in, you'll typically find an "Alerts" section in the main menu. Click on it.

3. Create New Alert:

• Look for a button or option to create a new alert. This is usually prominently displayed on the Alerts page.

4. Fill in Alert Details:

• Provide essential information for the alert:
  • Message: Briefly describe the nature of the alert.
  • Teams: Select the relevant team for the alert (e.g., "MyTeam").

5. Select Recipients:

• Depending on your Opsgenie configuration, you may have a field for selecting specific team members.
  • If you want to alert specific team members, look for a "Recipients" field and choose the individuals from the list.

6. Send to Whole Team:

• If you want to send the alert to the entire team:
  • Ensure you've selected the appropriate team in the "Teams" field.
  • Opsgenie will automatically notify all members of the selected team.

7. Configure Additional Settings:

• Opsgenie often provides additional settings for alerts such as priority, tags, and attachments. Adjust these as needed.

8. Review and Send:

• Take a moment to review the alert details.
• Look for a "Create" or "Send" button, and click it to initiate the alert.

9. Monitor Alert Status:

   • After sending the alert, you can monitor its status in the Opsgenie interface.
   • Opsgenie typically provides real-time updates on the status of alerts, acknowledgments, and resolutions.

10. Handle Acknowledgments and Resolutions:

• Opsgenie allows users to acknowledge and resolve alerts. If your team members are responsible for these actions, guide them through the interface to perform these tasks.

[jimleroyer]

Well the AI description needed some tweaking but it wasn't far off surprisingly.

The new documentation ready to be reviewed: https://docs.google.com/document/d/1FsGzDwWdZ51B6AHdCK2W4NA-vHAGmi6f7xmv5I2f0Lc/

[jimleroyer]

Ben, Yael and Jumana gave their approval on the document after review. Closing the ticket as it has been through review.