The current PagerDuty Notifier is broken, _get_alerts is using an undocumented api to get alerts for a service. When this API is used even when a limit param is passed it just hangs. The longest time I had this hanging in a request was 2 hours, after that I stoped the request.
This has been an issue for months and our implementation does not page us, instead the travis build does not notify PD due to the hanging request.
Requirements
Move away from the undocumented api
Fix the gathering of alerts so auditree can page.
Approach
Instead of using the undocumented https://api.pagerduty.com/alerts api and instead use a combination of the https://api.pagerduty.com/incidents and https://api.pagerduty.com/incidents/{id}/alerts apis to get the alerts.
We would first get all incidents for a service and loop over each one to get all alerts for the returned incidents.
Overview
The current PagerDuty Notifier is broken, _get_alerts is using an undocumented api to get alerts for a service. When this API is used even when a limit param is passed it just hangs. The longest time I had this hanging in a request was 2 hours, after that I stoped the request.
This has been an issue for months and our implementation does not page us, instead the travis build does not notify PD due to the hanging request.
Requirements
Approach
Instead of using the undocumented
https://api.pagerduty.com/alertsapi and instead use a combination of thehttps://api.pagerduty.com/incidentsandhttps://api.pagerduty.com/incidents/{id}/alertsapis to get the alerts.We would first get all incidents for a service and loop over each one to get all alerts for the returned incidents.
Documentation for these APIS:
Security and Privacy
N/A
Test Plan
Will be tested localy with our compliance checks both before and after the change to confirm that the change now pages us.