Update Sentry alerting for new OC rotation

[cynthiaxyin]

The new schedule. See Slack thread for pointers from Pedro. DB sources will probably be making similar updates too.

We should make the necessary changes and verify everything works as expected before the split on Dec 13.

[evantahler]

Doc: https://docs.google.com/document/d/1HweO9GZ0mOXdNDJOu50m63o5BfPKWDmZPXRHe-DaYhY/edit Repo: https://github.com/airbytehq/workflow-actions/blob/b35c4f7ea31e0fb3624edb777cb7a4ad5f5df8a1/README.md

notes on testing:

• we might need to make re-open some old issues to make sure that issue get routed to the right place

[cynthiaxyin]

Summary of changes made for posterity (essentially duplicated everything that currently exists for DB/DW but just for destinations):

• new schedule tied to new escalation policy + Airbyte Global escalation policy
• new service linked to the PD integration in Sentry here
• duplicated the 2 DB/DW alert rules here and added failure_origin == destination || normalization + tied to new service

Testing:

• hubot picks up the new rotation automatically
• Sentry test notification correctly pages me + posts in #oncall-bots
• unfortunately reopening an old Sentry event that previously executed a page doesn't do anything (neither the old nor new rotation gets paged)

TBD if we can properly test all the new alert rules. It would be nice to form enough confidence to deprecate the old schedule by the end of my rotation.

[cynthiaxyin]

Sentry paged API connectors instead of destinations :( Wrote up a doc: https://docs.google.com/document/d/1MZ_2BS9jvYCHuV304nnRz6PjvEZyhs_ylG7LAODvylw/edit

[evantahler]

re-assigning to @edgao while @cynthiaxyin is out.

[edgao]

this works now! from slack thread:

• @pedroslopez updated sentry ownership rules to route pages to the right team
• then we muted the old dw/dw sentry alerts, so that only the new destinations alerts (+ sources alerts) are still active
• then we triggered a page and verified that I got paged, and nobody else got paged